【client eclipse集群提交运行】:客户端 eclipse集群提交 mapreduce代码
1. 需求:在master:8088上,有客户端提交的job,则需要local-->yarn
需要改: 【mapred-site.xml】 mapreduce.framework.namelocal //改成yarnThe runtime framework for executing MapReduce jobs.Can be one of local, classic or yarn.
2.报错:
2018-06-28 00:23:18,573 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:· (62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2018-06-28 00:23:20,481 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032Retrying connect to server: 0.0.0.0/0.0.0.0:8032.//主机名默认是0.0.0.0,然后尝试连接连不上,需要指定指定主机名为master,去连接
解决:【yarn-site.xml】
The hostname of the RM.yarn.resourcemanager.hostnamemaster //默认值是:0.0.0.0
3. 报错:
Exit code: 1Exception message: /bin/bash: line 0: fg: no job control//没有job协议 Container exited with a non-zero exit code 1Failing this attempt. Failing the application.2018-06-28 00:40:54,407 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 0
解决: 【mapred-site.xml】
允许跨平台,使得Windows平台向集群提交job
If enabled, user can submit an application cross-platform //用户就可以提交应用跨平台i.e. submit an application from a Windows client to a Linux/Unix server orvice versa.mapreduce.app-submission.cross-platform //app应用允许跨平台true //默认false,应该改成true
4. 执行之后,无结果运行:
No job jar file set. User classes may not be found. See Job or Job#setJar(String). Container killed by the ApplicationMaster.Container killed on request. Exit code is 143Container exited with a non-zero exit code 143
原因:没有生成jar文件
解决:打jar包(将写的java文件和配置的文件都导出)。
放置到classpath下:右键jar包-->Bulid Path-->add to Bulid Path
5.报错:0.0.0./0.0.0.:10020,反复的报这个错
10020是历史服务器;
解决:把历史服务器开启来就好,开启history服务,
命令即可:
1) $>mr-jobhistory-daemon.sh start historyserver
2) 还要修改【mapred-site.xml】:
mapreduce.jobhistory.address = master:10020
mapred-site.xml中还有添加:
mapreduce.jobhistory.address0.0.0.0:10020 //改成master:10020MapReduce JobHistory Server IPC host:port