作者:mobiledu2502912277 | 来源:互联网 | 2023-07-30 15:51
(1)operationhistory:SPARK_HADOOP_VERSION2.2.0SPARK_YARNtruesbtsbtassembly按官方教程失败,但去掉红色
(1) operation history:
>SPARK_HADOOP_VERSION=2.2.0
SPARK_YARN=true sbt/sbt assembly
按官方教程失败,但去掉红色的部分,很容易编译成功,这条指令会带来什么?
修改为
SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true
sbt/sbt clean compile package assembly
正常通过
(2) 启动测试程序
export YARN_CONF_DIR=$HADOOP_HOME
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar ./spark-class org.apache.spark.deploy.yarn.Client --jar ./examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar --class org.apache.spark.examples.JavaSparkPi --args yarn-standalone --num-workers 3 --master-memory 2g --worker-memory 2g --worker-cores 1
(3) 结果
...
14/02/27 00:25:57 INFO yarn.Client: Application report from ASM:
application identifier: application_1393415612957_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: 192.168.0.105
appQueue: default
appMasterRpcPort: 0
appStartTime: 1393431898882
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: jiahuili-MBP.local:8088/proxy/application_1393415612957_0004/
appUser: admin
14/02/27 00:25:58 INFO yarn.Client: Application report from ASM:
application identifier: application_1393415612957_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: 192.168.0.105
appQueue: default
appMasterRpcPort: 0
appStartTime: 1393431898882
yarnAppState: FINISHED
distributedFinalState: SUCCEEDED
appTrackingUrl: jiahuili-MBP.local:8088/proxy/application_1393415612957_0004/A
appUser: admin
check http://localhost:8088/cluster/ for task details
Ref:
(1) http://spark.incubator.apache.org/docs/latest/running-on-yarn.html
(2) http://blog.csdn.net/macyang/article/details/17632645
-- about yarn
(3) http://dongxicheng.org/framework-on-yarn/build-spark-on-hadoop-2-yarn/
operation note for "spark with yarn",布布扣,bubuko.com