3.ApacheHive查询报错FAILED:org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

版本

CDH 6.2.0
Hive 2.1.1-cdh6.2.0

执行查询视图的sql报错:FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

视图由多个表join生成。
sql如下

select * from view_test limit 10;


日志：

Task failed!
Task ID:
Stage-11
Logs:
/tmp/hdfs/hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
MapReduce Jobs Launched:
Stage-Stage-2: Map: 5 Reduce: 6 Cumulative CPU: 167.37 sec HDFS Read: 367721084 HDFS Write: 390996703 HDFS EC Read: 0 SUCCESS
Total MapReduce CPU Time Spent: 2 minutes 47 seconds 370 msec


/tmp/hdfs/hive.log

2021-01-29T09:28:31,911 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main-LocalTask-MAPREDLOCAL-stdout] mr.MapredLocalTask:
2021-01-29T09:28:33,576 ERROR [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] exec.Task: Execution failed with exit status: 3
2021-01-29T09:28:33,576 ERROR [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] exec.Task: Obtaining error information
2021-01-29T09:28:33,576 ERROR [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] exec.Task:
Task failed!
Task ID:
Stage-11
Logs:
2021-01-29T09:28:33,576 ERROR [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] exec.Task: /tmp/hdfs/hive.log
2021-01-29T09:28:33,576 ERROR [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] mr.MapredLocalTask: Execution failed with exit status: 3
2021-01-29T09:28:33,599 ERROR [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] ql.Driver: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
2021-01-29T09:28:33,599 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] ql.Driver: MapReduce Jobs Launched:
2021-01-29T09:28:33,611 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] ql.Driver: Stage-Stage-2: Map: 5 Reduce: 6 Cumulative CPU: 160.0 sec HDFS Read: 367720339 HDFS Write: 390996703 HDFS EC Read: 0 SUCCESS
2021-01-29T09:28:33,611 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] ql.Driver: Total MapReduce CPU Time Spent: 2 minutes 40 seconds 0 msec
2021-01-29T09:28:33,611 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] ql.Driver: Completed executing command(queryId=hdfs_20210129092522_3916c3bc-672e-4215-b31c-940c174692a6); Time taken: 188.998 seconds
2021-01-29T09:28:33,702 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] conf.HiveConf: Using the default value passed in for log id: 8b09e42b-81d2-4cce-80c7-d27a63772e5c
2021-01-29T09:28:33,702 INFO [8b09e42b-81d2-4cce-80c7-d27a63772e5c main] session.SessionState: Resetting thread name to main
二、定位

1 搜索引擎搜索

关键字：FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

https://blog.csdn.net/ddxygq/article/details/81676046/

博客里提到这个报错是因为开启了mapjoin导致的问题
小表数据太大，导致了内存溢出

2 查看源码

上面的注释也写了是mapjoin

1 关闭mapjoin

为什么选择关闭呢，因为已经给hive进程加过内存了，加内存只能缓解，每次小表数据量增加，都需要再次加内存，索性就直接关闭了

1.1 临时修改

su hdfs
hive
set hive.auto.convert.join = false;


修改后再次执行sql ，已经可以查询出结果

1.2 永久修改

这里使用的CDH集群

打开CDH页面 ——> 点击hive ——> 点击配置 ——>在搜索框输入mapjoin ——> 取消复选框 ——> 点击保存更改
重启集群

再次执行sql ，已经可以查询出结果