hadoop修复hdfs中文件块错误和文件修复Error:java.io.IOException:CannotobtainblocklengthforLocatedBlock

一&＃xff1a;问题 Error: java.io.IOException: Cannot obtain block length for LocatedBlock

Error: java.io.IOException: Cannot obtain block length for LocatedBlock{BP-1241298414-127.0.0.1-1523601065028:blk_1073766720_25914; getBlockSize()&＃61;1077; corrupt&＃61;false; offset&＃61;0; locs&＃61;[DatanodeInfoWithStorage[172.16.144.58:50010,DS-69e6b461-59df-4e19-90ff-e766c0e3535c,DISK], DatanodeInfoWithStorage[172.16.112.83:50010,DS-3cef3847-e522-4b28-a84e-ff5ec9f2b02a,DISK]]}at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:444)at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:351)at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:282)at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:272)at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132)at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:914)at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:554)at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:792)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

二&＃xff1a;解决办法&＃xff1a;

1、找出出问题的文件

输入下面命令&＃xff1a;

hdfs fsck /logs/product/desktop-ecommerce/2018-05-31 -openforwrite


结果&＃xff1a;

Connecting to namenode via http://master:11070/fsck?ugi&＃61;root&openforwrite&＃61;1&path&＃61;%2Flogs%2Fproduct%2Fdesktop-ecommerce%2F2018-05-31FSCK started by root (auth:SIMPLE) from /172.16.199.39 for path /logs/product/desktop-ecommerce/2018-05-31 at Fri Jun 01 09:33:35 CST 2018../logs/product/desktop-ecommerce/2018-05-31/FlumeData.1527714183563 1077 bytes, 1 block(s), OPENFORWRITE: .Status: HEALTHYTotal size: 5336068 BTotal dirs: 1Total files: 4Total symlinks: 0Total blocks (validated): 4 (avg. block size 1334017 B)Minimally replicated blocks: 3 (75.0 %)Over-replicated blocks: 0 (0.0 %)Under-replicated blocks: 0 (0.0 %)Mis-replicated blocks: 0 (0.0 %)Default replication factor: 2Average block replication: 1.5Corrupt blocks: 0Missing replicas: 0 (0.0 %)Number of data-nodes: 2Number of racks: 1FSCK ended at Fri Jun 01 09:33:35 CST 2018 in 1 milliseconds


找到了问题文件&＃xff1a;/logs/product/desktop-ecommerce/2018-05-31/FlumeData.1527714183563

三&＃xff1a;对文件执行修复命令

hdfs debug recoverLease -path /logs/product/desktop-ecommerce/2018-05-31/FlumeData.1527714183563 -retries 3

四&＃xff1a;再次执行mapreduce程序 没有报错 完美解决问题&＃xff01;