大数据实时计算Flink-SQL架构介绍课件.pptx

1、大数据实时计算 Flink SQL 架构介绍技术创新 变革未来目目 录录1Background2Flink SQL 基本概念3Flink SQL 核心功能4Flink SQL 优化5阿里云流计算产品BackgroundAlibaba Blink阿里巴巴阿里巴巴Blink团队有团队有 20+ flinkcontributor，6 名名 committer，向社区贡向社区贡献了数百个献了数百个Commit+=Apache FlinkAlibabas ImprovementsBlinkAlibaba BlinkBlink Runtime+ Flink SQL=团队工作团队工作主导制定主导制定 Fli

2、nk SQL 语义语义 DynamicTable 2016-2017 Retraction 2016-2017完完善善 Flink SQL 功能功能 Agg, Join, Window 2017 跑通全部 TPCH Query 2018性能提升性能提升 大量的查询优化 2017-2018资源配置自动化资源配置自动化 2018贡献社区贡献社区贡献社区贡献社区部分贡献社区部分贡献社区Flink SQLWhy SQL?DeclarativeOne Query, One ResultOptimizedUnderstandableStableUnifySQL 不是为流设计的不是为流设计的没有Retrac

3、tion批计算查询返回一个结果并结束数据是有限的数据是有限的批处理批处理流数据是无穷的流数据是无穷的流上的查询不断产生结果且不会结束有对历史数据的修改(Retraction)流处理流处理动态表动态表（Dynamic Table）动态表（Dynamic Table）:数据会随着时间变化的表动态表动态表 + 连续查询连续查询连续查询（Continuous Query）：持续运行的查询StreamStreamStreamStream连连续续查查询询StreamStreamStreamStream连连续续查查询询连续查询连续查询StreamStream流计算流计算 Retraction流计算流计算 R

4、etraction世界上不需要所谓的世界上不需要所谓的 Stream SQL标准的标准的 ANSI SQL 就可以用来定义流计算就可以用来定义流计算Flink SQL 核心功能核心功能DDL & DMLUDF/UDTF/UDAFWindow AggJoinGroup AggOver AggLoading Data- 定义数据源表定义数据源表CREATE TABLE clicks (VARCHAR, TIMESTAMP, VARCHARuser cTime url) WITH (type = kafka,topic = click_topic,);SELECT * FROM clicksuser

5、cTimeurlMary12:00:00./homeBob12:00:00./cartMary12:00:05./prod?id=1Saving Data- 定义数据结果表定义数据结果表CREATE TABLE last_clicks (user cTime urlVARCHAR, TIMESTAMP, VARCHAR,PRIMARY KEY (user) WITH (type = mysql,);INSERT INTO last_clicksSELECT * FROM clicksMulti OutputCREATE VIEW taobao_clicks ASSELECT * FROM cl

6、icks WHERE url LIKE http:/%INSERT INTO mysql_result SELECT * FROM taobao_clicksINSERT INTO hbase_resultSELECT * FROM taobao_clicksCREATE TABLE mysql_clicks (user cTime urlVARCHAR, TIMESTAMP, VARCHAR,PRIMARY KEY (user) WITH (type = mysql,);CREATE TABLE hbase_clicks (user cTime urlVARCHAR, TIMESTAMP,

7、VARCHAR,PRIMARY KEY (user) WITH (type = hbase,);Group AggregateMary1Mary2resultusercntMary3Bob1SELECTuser,COUNT(url) as cnt FROM clicksGROUP BY userclicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:00:05./prod?id=1Mary12:01:45./prod?id=7从历史到现在每个用户点击的次数Window Aggregate每小时每个用户点击的次数resultuser

8、endTcntMary13:00:003Bob13:00:001Bob14:00:001Liz14:00:002Bob13:01:00./prod?id=4Liz13:30:00./cartLiz13:59:00./homeSELECTuser, TUMBLE_END(cTime,INTERVAL 1 HOURS)AS endT, COUNT(url) AS cntFROM clicksGROUP BYuser, TUMBLE(cTime,INTERVAL 1 HOURS)clicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:0

9、2:00./prod?id=2Mary12:55:00./home双双流流 JOIN：支持 INNER, LEFT, RIGHT, FULL, SEMI, ANTISELECTo.orderId, o.productId, o.orderTime, s.shipTim FROM Order JOIN ShipmON o.orderOrdersorderIdproductIdorderTime53010:17:0061010:17:0591011:02:00121011:24:11ShipmentsorderIdshipTime510:55:00610:20:00911:58:001211:44

10、:00es AS oents AS sId = s.orderIdresultorderIdproductIdorderTimeshipTime53010:17:0010:55:0061010:17:0510:20:0091011:02:0011:58:00121011:24:1111:44:00维维表表 JOIN：支持 INNER, LEFTCREATE TABLE Products ( productId VARCHAR, productName VARCHAR, price DECIMAL,PRIMARY KEY (productId), PERIOD FOR SYSTEM_TIME)

11、WITH (type = hbase);SELECT o.*, p.* FROM Orders AS oJOIN Products FOR SYSTEM_TIME AS OF PROCTIME() AS pON o.productId = p.productId聊几个优聊几个优化化awabwbDataBaseReduced ThroughputWait for ResponseadDataBaseabcbcdSend RequestReceive RequestWaitConcurrent ProcessingIncreased ThroughputSync. IOAsync. IO异步维表

12、JOIN异步维表 JOINCREATE TABLE Products ( productId VARCHAR, productName VARCHAR, price DECIMAL,PRIMARY KEY (productId), PERIOD FOR SYSTEM_TIME) WITH (type = hbase,async = true);SELECT o.*, p.* FROM Orders AS oJOIN Products FOR SYSTEM_TIME AS OF PROCTIME()AS pON o.productId = p.productId一行配置的改动AggAggMapMapMap如何处理数据倾斜Data-Skew如何处理数据倾斜Data-SkewAggAggMapMapMapHot!反压反压反压如何处理数据倾斜Data-SkewLocal-Global Aggregation 优化优化如何处理数据倾斜Data-SkewLocal AggLocalAggLocalAggGlobal AggGlobal AggAggAggMapMapMapMapMapMapLocal-Global AggregationSimple AggregationLocal-Global带来带来 20X的性能提升的性能提升优化前优化后阿里云流计算产品阿里云流计算产品谢谢 谢！谢！