作者:记录生活传奇_909_874 | 来源:互联网 | 2023-05-17 16:17
mapreduce执行时一直处在runningjob一栏自己感觉是resourcemanager的问题,访问出问题了,因为nodemanager是启动了的INFOmapreduce.J
mapreduce 执行时一直处在running job 一栏
自己感觉是resourcemanager的问题,访问出问题了,因为nodemanager是启动了的
INFO mapreduce.Job: Running job: job_1440140752819_0001
日志文件为:
STARTUP_MSG: Starting NodeManager
STARTUP_MSG: host = itcast02/192.168.8.138
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.2.0
STARTUP_MSG: classpath = /itcast/hadoop-2.2.0/etc/hadoop:/itcast/hadoop-2.2.0/etc/hadoop:/itcast/hadoop-2.2.0/etc/hadoop:
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2014-06-25T14:34Z
STARTUP_MSG: java = 1.7.0_75
************************************************************/
2015-08-21 14:34:16,216 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
2015-08-21 14:34:17,297 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2015-08-21 14:34:17,297 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2015-08-21 14:34:17,298 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
2015-08-21 14:34:17,298 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2015-08-21 14:34:17,298 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2015-08-21 14:34:17,299 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2015-08-21 14:34:17,316 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2015-08-21 14:34:17,317 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.NodeManager
frederick丶kooky~! 2015/8/21 14:58:08
2015-08-21 14:34:17,360 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-08-21 14:34:17,412 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-08-21 14:34:17,412 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started
2015-08-21 14:34:17,424 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
2015-08-21 14:34:17,425 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: per directory file limit = 8192
2015-08-21 14:34:17,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file:/itcast/hadoop-2.2.0/tmp/nm-local-dir/usercache_DEL_1440130574099
2015-08-21 14:34:17,444 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file:/itcast/hadoop-2.2.0/tmp/nm-local-dir/usercache_DEL_1440138662662
另附配置文件
etc/hosts
127.0.0.1 localhost
192.168.8.88 itcast01 (主机)
192.168.8.138 itcast02 (slave)
192.168.8.139 itcast03 (slave)
core-site.xml
fs.default.name
hdfs://itcast01:9000
hadoop.tmp.dir
/itcast/hadoop-2.2.0/tmp
hdfs-site.xml
dfs.replication
1
yarn-site.xml
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resoucemanager.hostname
itcast01
mapred-site.xml
mapreduce.framework.name
yarn
slaves
itcast02
itcast03
vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=itcast01
8 个解决方案
我是这么做的 在执行任务的时候每一步都 写个日志文件 查看执行情况
你etc/hosts 配置文件中最好把 127.0.0.1 localhost 注释掉
#127.0.0.1 localhost
192.168.8.88 itcast01 (主机)
192.168.8.138 itcast02 (slave)
192.168.8.139 itcast03 (slave)
在你 192.168.8.88 itcast01 (主机) jps 看看 resourcemanager起了没,其他2个slave节点上看看 nodemanger起了没,datanode起了没
基本上就是namenode没有顺利启动,yarn配置缺失或者空间分配过小的问题,我倾向是第三种... 当初我也是这个问0题加大内存就解决了。