Spark某两个节点数据分析速度慢hbase数据删除(分裂)元信息未删除导致Thereisanoverlapintheregionchain.

基于Hbase2.0&＃xff0c;Spark2.2

问题描述

执行Spark处理Hbase数据时&＃xff0c;遇到某两个Excutor处理速度特别慢&＃xff0c;如图

正常速度10多分钟 左右处理完成&＃xff0c; 一个多小时有另外一个处理完成&＃xff0c;还有一个在处理中。

分析原因

1.查看hbase数据分布状况&＃xff0c;看看是否存在数据倾斜问题&＃xff0c;可以看出平均为6.8G左右&＃xff0c;但有部分数据是1.71GB和3.4GB左右&＃xff0c;但之前任务跑spark是80多个Executor不会导致时间差距这么大。

其实并非倾斜&＃xff0c;下图1.71GB是6.8GB刚刚分裂的Region&＃xff0c;分裂成了四个1.71*4 &＃61; 6.84GB&＃xff0c;大小刚刚是平均大小&＃xff0c;

startrowkey &＃xff1a;bc27300~bf2a300 大小3.41&＃43;1.71.&＃43;1.71 &＃61; 6.83 都是正常的&＃xff08;数据存储前rowkey做了处理&＃xff09;

2.查看Spark日志看看有什么问题

 发现日志,可以看到startRow&＃61;0000&＃xff0c;stopRow&＃61;fffg &＃xff0c;这个Spark扫描Hbase设置的条件

19/04/15 09:29:41 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: zh_ams_ns:wechat_article, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"0000","stopRow":"fffg","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"fn":["article_type","download_type"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: 7fef300_18b81d0e3245848ddfaf6a97ea868de1, end row: c02b300_52789a8b10e2d896e9be1272d9169af1, region location: hostname1, encoded region name: 8943fc0bd38fd292d9acb1c6bb4b7a6)
19/04/15 09:29:41 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: zh_ams_ns:wechat_article, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"0000","stopRow":"fffg","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"fn":["article_type","download_type"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: 6405300_0030bf135d4c2c20e9915c1941bd8cc3, end row: 67ff300_beefe04a96f547403165d057642c7a6e, region location: hostname1, encoded region name: 96c9963bdd044ebdf2bd883435735d5)
19/04/15 09:29:41 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0

这个节点的具体处理数据信息为

start row: 7fef300_18b81d0e3245848ddfaf6a97ea868de5, end row: c02b300_52789a8b10e2d896e9be1272d9169afd, region location: hostname1
start row: 6405300_0030bf135d4c2c20e9915c1941bd8cc3, end row: 67ff300_beefe04a96f547403165d057642c7a6e, region location: hostname1

明显看出7fef300~c02b300&＃xff0c;范围远大于6405300~67ff300&＃xff0c;查看其他Executor&＃xff0c;也发现跨度不会这么大&＃xff0c;查看Hbase表的数据状态发现7fef300~c02b300的region没有数据&＃xff0c;所有信息都为零。

可以初步判断出region分裂后&＃xff0c;region的元信息没有进行删除&＃xff0c; 导致spark读取时把rowkey 7fef300~c02b300的数据重新读取一遍&＃xff0c;其中包含多个region导致Executor处理速度过慢。

为什么发现很多个region信息分裂后&＃xff0c;元信息都没有删除&＃xff0c;但是只有两个是非常慢的&＃xff0c;可以从hbase数据中发现只有两个元信息跨度比较大&＃xff0c;其他的rowkey范围宽度较小&＃xff0c;所以没有明显感觉出来&＃xff0c;如下图只跨了两个region

判断完成后&＃xff0c;我们通过hbase hbck查看一下表的状态&＃xff0c;截取报异常的一段&＃xff0c;提示tableName,,1539422767596.259a2349c4f07b7d625142d323bede56.   region与其他region有重复区域&＃xff0c;验证了我们刚才的想法

ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,0ffd300_74fdb5eebd54ee32e19abb31bccec158,1542774878705.a860b3230759d601187e9dcf6a4bdad0.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,10fd300_7a83ffc34c076654e69d558f1c9f1038,1550065067486.62a2f5707f49a536755d8a357d30bfd0.) There is an overlap in the region chain.
ERROR: (regions tableName,0ffd300_74fdb5eebd54ee32e19abb31bccec158,1542774878705.a860b3230759d601187e9dcf6a4bdad0. and tableName,10fd300_7a83ffc34c076654e69d558f1c9f1038,1550065067486.62a2f5707f49a536755d8a357d30bfd0.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,11fe300_2aa404cbc291ffd913bab13d9339085c,1543086502643.f4c52419535c5574f7b79c05f82a3bd2.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,12fe300_3f46699293dbeaa042e6856597d26dca,1543086502643.dcfff4e27445fd7ef7d54e931fc3fee1.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,13fd300_c223d551c26e191ffacb9981c5fb2cfd,1542770662897.f0c75de535b324f02267b2e2d0020c75.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,1800300_0c10469f3ca178e9d5d81b507cc1f684,1548215404844.8546360f0bb0b7d0dcbd2c07772127ab.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,1c00300_b0e2e6ce62897dbd1e033f88bcf22336,1548215404844.b12cda6b9cb167c54ed47f62cc0aa7a2.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2002300_cb4302be610bfedebaedce4a53cd0e4e,1549815723592.9b62e6522d5c687a088d95fbd1ca0c1b.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2404300_806ce32f00569ec34eb1fdc008c574cc,1549815723592.dada7d00670779d1aaf6c0d47ebb50c7.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2805300_3063b4db3bc49e0cd2d2e2430b8f92e8,1551035005714.2a6ad9c50e10527e2f08cad1d5189e4a.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,2c04300_c3f674b39aeb8f289313f87d8d21bfe7,1551035005714.df8ba688e677572214f75b270dedcd1b.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3005300_be2766fa12b179733c9a55801367d56f,1551240890445.e479bcacf255f34bd0a98540e6e02ac5.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3406300_ec238a180804b26ea232125ff11fcf68,1551240890445.21a2461c5242d970cc46486b14817e12.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3806300_886cf9e6502abd928a8c063ca517f54e,1548178323143.01d1784953e10b53ed9a41953302b3bc.) There is an overlap in the region chain.
ERROR: (regions tableName,,1539422767596.259a2349c4f07b7d625142d323bede56. and tableName,3c05300_7ea6294dd2cd312d63d846bf5de5cd35,1548178323143.f861eb4119dc4dc3f7c5e73555824552.) There is an overlap in the region chain.
19/04/16 13:28:43 WARN util.HBaseFsck: reached end of problem group: 4003300_c126f5cc6cef33e84de0cafc9e52d44f

重大问题&＃xff1a;统计完成时发现&＃xff0c;会导致数据重复读取&＃xff0c;正常数据量为142825979条  &＃xff0c;结果查询出来224847552条。

解决方法

1.运行过程中&＃xff0c;不但慢后来因为Executor处理数据过大&＃xff0c;内存不足导致异常&＃xff0c;解决方式

https://blog.csdn.net/zhangshenghang/article/details/89313245

2.删除分裂前的region

根据region name进行删除&＃xff0c;元信息在hbase:meta表中&＃xff0c;查看状态

get &＃39;hbase:meta&＃39;,&＃39;tableName,7fef300_18331d0e3245848ddfa16a97ea868de5,1539422555091.dee970bfd95a946d9a9268db01f7ec77.&＃39;

删除

deleteall &＃39;hbase:meta&＃39;,&＃39;tableName,7fef300_18b81d0e3245848ddfaf6a97ea868de5,1539422555091.dee970bfd95a946d9a9268db01f7ec77.&＃39;

删除后再次跑spark即可过滤掉这个region&＃xff0c;解决该问题。

 但是存在一个隐藏问题&＃xff0c;将表enable之后&＃xff0c;删除的region又出现了&＃xff0c;查看zookeeper也没有改配置&＃xff0c;如何恢复的呢&＃xff1f;后来经查阅相关资料发现master是有缓存存储这些信息的&＃xff0c;删除元信息&＃xff08;目录 hdfs   /hbase/MasterProcWALs &＃xff09;后重启hbase Master集群元信息就可以了。

问题&＃xff1a;

千万不要删除有数据或者正常的region&＃xff0c;删除并重启后&＃xff0c;表数据异常无法读取数据&＃xff0c;提示如下

hbase(main):062:0> count &＃39;tableName&＃39;ERROR: Unknown table tableName!

使用 hbase hbck查看状态&＃xff0c;提示region rowkey 范围开始应该为 &＃39;&＃39; 空字符串&＃xff0c;如果删除正常的region会导致异常

ERROR: (region tableName_201615,201604097053999999_38599d53219f66c8605abbe144b33844,1494968976496.08e0b835edffba9c7f47c68d800f297a.) First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole.
ERROR: Found inconsistency in table tableName_201615

正常 如下图所示