Flink博客记录

Flink

一、Flink 记录

1.1 内存管理

1.1.1 Flink内存管理

https://www.cnblogs.com/lighten/p/13053828.html

1.1.2 Flink_Flink TaskManager 内存模型 &＃xff0c;以及相关内存配置参数

https://blog.csdn.net/u010003835/article/details/106294342

1.2 Flink 记录

–>> 一文搞懂 Flink 的 Exactly Once 和 At Least Once

https://mp.weixin.qq.com/s/QcL33ghaM-riceOr218oXg

1.2.1 Kafka偏移量提交

https://stackoverflow.com/questions/54585740/commit-kafka-offsets-manually-in-flink

1.2.2 Flink savepoint

https://zhuanlan.zhihu.com/p/94594551

https://zhuanlan.zhihu.com/p/79526638

1.2.3 checkpoint

1.2.3.1 checkpoint 并行度改变后&＃xff0c;状态重分配

https://blog.csdn.net/lvwenyuan_1/article/details/98511963

1.2.3.2 checkpoint 恢复

https://mp.weixin.qq.com/s/rncSvPKdFrT2tFMs6TZpcA

1.2.4 反压

http://wuchong.me/blog/2016/04/26/flink-internals-how-to-handle-backpressure/

1.2.5 序列化

http://wuchong.me/blog/2016/04/29/flink-internals-memory-manage/

https://developer.aliyun.com/article/721997

1.2.6 CPU L1 L2 L3 是什么&＃xff0c;作用是什么&＃xff1f;

https://blog.csdn.net/u010632165/article/details/106795478

1.2.7 Flink on yarn 指定节点

https://blog.csdn.net/weixin_47235286/article/details/121883864

yarn-session.sh -n 2 -jm 1024 -tm 1024 -d -s 2-n 2 表示指定两个容器
-jm 1024 表示jobmanager 1024M内存
-tm 1024表示taskmanager 1024M内存
-d --detached 任务后台运行
-s 指定每一个taskmanager分配多少个slots(处理进程)。建议设置为每个机器的CPU核数。一般情况下&＃xff0c;vcore的数量等于处理的slot&＃xff08;-s&＃xff09;的数量
-nm,--name YARN上为一个自定义的应用设置一个名字
-q,--query 显示yarn中可用的资源 (内存, cpu核数)
-qu,--queue <arg> 指定YARN队列.
-z,--zookeeperNamespace <arg> 针对HA模式在zookeeper上创建NameSpace


Flink JobListener

Flink 维表更新处理办法

https://zhuanlan.zhihu.com/p/490835750

Flink trigger

https://blog.csdn.net/qq_31866793/article/details/102842570
https://www.jianshu.com/p/e32e04a1d6c4
—>>> https://blog.csdn.net/weixin_43495317/article/details/124241897

Flink 多个source watermark

https://my.oschina.net/qiangzigege/blog/2993330?p&＃61;{{currentPage&＃43;1}}

异步 Asyc

https://blog.csdn.net/qq_46045632/article/details/122588486?utm_medium&＃61;distribute.pc_aggpage_search_result.none-task-blog-2_{aggregatepage}first_rank_ecpm_v1~rank_v31_ecpm-1-122588486.pc_agg_new_rank&utm_term&＃61;flink&＃43;%E5%AE%9E%E6%97%B6%E6%9F%A5%E8%AF%A2&＃43;%E7%BB%B4%E8%A1%A8&spm&＃61;1000.2123.3001.4430

Flink 流批一体的实践与探索

https://blog.csdn.net/zhaodedong/article/details/105424236

监控

Flink 任务实时监控最佳实践

https://mp.weixin.qq.com/s?__biz&＃61;Mzg3MDE0MjUzMA&＃61;&＃61;&mid&＃61;2247484144&idx&＃61;1&sn&＃61;12462bf23cbb4e318857dd53375666b5&chksm&＃61;ce9305ddf9e48ccb64e5712ee399b2a10fe1b1fcb7081211109af90da961d6c8a59135860bc0&mpshare&＃61;1&scene&＃61;23&srcid&＃61;0808f76eX4ygFWId3wSbDzbT&sharer_sharetime&＃61;1628421639852&sharer_shareid&＃61;d73403798b935b929a6ed550a277d0ee#rd

二、FlinkSQL

CDH FlinkSQL 集成 hive

https://www.jianshu.com/p/e964ba3fb8f8

FlinkSQL平台化

Flink SQL on Zeppelin - 打造自己的可视化Flink SQL开发平台

https://mp.weixin.qq.com/s/QbBCO4ul6-hofl-AErld9g

大数据开发之Flink SQL建设实时数仓实践

https://blog.csdn.net/m0_58371965/article/details/122577670

基于 Flink &＃43; Hive 构建流批一体准实时数仓

https://mp.weixin.qq.com/s?__biz&＃61;MzU3Mzg4OTMyNQ&＃61;&＃61;&mid&＃61;2247489376&idx&＃61;1&sn&＃61;000fda2c695c88a6542f5cebb3207b64&chksm&＃61;fd3b9922ca4c1034b8bc0bff6d091c31c13b41443c3a71f5420007d91bc24b60bb102d6a9a94&xtrack&＃61;1&scene&＃61;90&subscene&＃61;93&sessionid&＃61;1601205791&clicktime&＃61;1601205953&enterid&＃61;1601205953&ascene&＃61;56&devicetype&＃61;android-26&version&＃61;3.0.31.2998&nettype&＃61;cmnet&abtest_COOKIE&＃61;AAACAA%3D%3D&lang&＃61;zh_CN&exportkey&＃61;AeaDMVfzSL04VBtRUGcfL2Q%3D&pass_ticket&＃61;%2BVHG5JT6TAPRJ0%2BEgPbDiMJd0tzudJm5cBMmrmzu1D1RF1aDyY2nyEIuVQLSQS23&wx_header&＃61;1&platform&＃61;win

自定义函数

https://www.alibabacloud.com/help/zh/doc-detail/69462.html

三、FlinkCDC

–>> Flink1.11中的CDC Connectors操作实践

https://mp.weixin.qq.com/s/h0RvUH8upF8Cqn7PKfo6Qw

Flink CDC 和 kafka 进行多源合并和下游同步方案

https://blog.csdn.net/qq_24505127/article/details/122761938

FlinkCDC &＃43; hudi &＃43; hive

https://www.csdn.net/tags/NtjacgzsNTgwODEtYmxvZwO0O0OO0O0O.html

—>>> hudi

https://blog.csdn.net/m0_66705151/category_11623872.html

监控

flink自定义metrics监控kafka消费

https://blog.csdn.net/zyy_2018/article/details/119537150

http://t.zoukankan.com/qiu-hua-p-13910809.html

Calcite记录

Apache Calcite 处理流程详解

http://matt33.com/2019/03/07/apache-calcite-process-flow/

flink sql 知其所以然&＃xff08;六&＃xff09;| flink sql 约会 calcite&＃xff08;看这篇就够了&＃xff09;

https://mp.weixin.qq.com/s/SxRKp368mYSKVmuduPoXFg

Calcite应用指南

https://blog.csdn.net/hxcaifly/article/details/88661252

基于Calcite自定义SQL解析器

https://blog.csdn.net/weixin_38381496/article/details/117418643

Calcite中定制自已SQL解析器

https://www.jianshu.com/p/e4f883a3b969
https://zhuanlan.zhihu.com/p/65345335

Apache Calcite学习资料

https://zhuanlan.zhihu.com/p/76510053

Calcite系列博客

https://blog.csdn.net/jyh2005/category_9706931.html

https://blog.csdn.net/qxc1281/category_8479654.html

https://www.jianshu.com/nb/35041702

–>https://blog.csdn.net/qq_31183071/category_9427896.html

–>https://blog.csdn.net/weixin_44112790/category_10882344.html

优化

Flink 单并行度内使用多线程&＃xff0c;提高写入性能10倍

https://blog.csdn.net/qq_23160237/article/details/103821970

源码分析

–>https://blog.jrwang.me/tags/flink/
https://www.cnblogs.com/ljygz/tag/%E5%AE%9E%E6%97%B6%E8%AE%A1%E7%AE%97/default.html?page&＃61;1

实践

Flink 容器化在唯品会的实践

https://mp.weixin.qq.com/s/Bl8HlJ_59ulbwqPNpi2CiA

部署

Flink on K8S

https://cloud.tencent.com/developer/article/1763130

Flink sql 消费kafka的顺序是怎么样的? 解答为什么第二次运行sql的结果和第一次不同

https://blog.51cto.com/u_15278282/4895929

–>> Flink on Yarn 远程 debug 应用提交流程及源码分析

https://zhuanlan.zhihu.com/p/444605072

Flink on yarn 源码分析

https://blog.csdn.net/jhkkhtd/article/details/118029480?spm&＃61;1001.2014.3001.5502

Flink on yarn

https://blog.csdn.net/xc_zhou/article/details/124371043?utm_term&＃61;flink%E9%A1%B5%E9%9D%A2%E6%8F%90%E4%BA%A4job&utm_medium&＃61;distribute.pc_aggpage_search_result.none-task-blog-2_allsobaiduweb~default-3-124371043-null-null&spm&＃61;3001.4430

Flink 系列博客

王知无

https://blog.csdn.net/u013411339/category_8631091.html

大数据之flink

https://www.cnblogs.com/huanghanyu/category/1758541.html

浪尖

https://blog.csdn.net/rlnlo2pnefx9c/category_10872751.html

Flink 开源项目

zhisheng

https://github.com/zhisheng17/flink-learning

flink-recommandSystem-demo

https://github.com/CheckChe0803/flink-recommandSystem-demo

Flink问题记录汇总

https://developer.aliyun.com/ask/288158?spm&＃61;a2c6h.13066369.0.0.14375f93zjXkcc
https://developer.aliyun.com/ask/258515