SparkCore（5）：SparkonStandalone配置和测试

1.实现功能

 Spark应用运行在Standalone资源管理框架系统上&＃xff0c;Standalone是spark自带的一种资源管理框架&＃xff0c;类似yarn&＃xff0c;分布式的。

2.Standalone的框架

        Worker&＃xff1a; 执行节点服务&＃xff0c;管理当前节点的资源及启动executor
        Master: 集群资源管理及申请

3.配置信息

&＃xff08;1&＃xff09;要求&＃xff1a;spark的local本地模式可以成功运行&＃xff0c;配置spark-env.sh

JAVA_HOME&＃61;/opt/jdk1.8.0_151
SCALA_HOME&＃61;/opt/modules/scala-2.11.8HADOOP_CONF_DIR&＃61;/opt/modules/apache/hadoop-2.7.3/etc/hadoop
SPARK_LOCAL_IP&＃61;bigdata.ibeifeng.com

&＃xff08;2&＃xff09;在spark-env.sh添加master和worker信息

&＃xff08;a&＃xff09;虚拟机

SPARK_MASTER_IP&＃61;bigdata.ibeifeng.com
SPARK_MASTER_PORT&＃61;7070
SPARK_MASTER_WEBUI_PORT&＃61;8080
SPARK_WORKER_CORES&＃61;2
SPARK_WORKER_MEMORY&＃61;2g
SPARK_WORKER_PORT&＃61;7071
SPARK_WORKER_WEBUI_PORT&＃61;8081
SPARK_WORKER_INSTANCES&＃61;2

&＃xff08;b&＃xff09;服务器配置

SPARK_MASTER_HOST&＃61;hadoop
SPARK_WORKER_CORES&＃61;2
SPARK_WORKER_MEMORY&＃61;2g
SPARK_WORKER_INSTANCES&＃61;1

(3)配置slaves文件

mv slaves.template slaves

添加

&＃xff08;a&＃xff09;虚拟机

# A Spark Worker will be started on each of the machines listed below.
bigdata.ibeifeng.com

&＃xff08;b&＃xff09;服务器

# A Spark Worker will be started on each of the machines listed below.
hadoop

&＃xff08;4&＃xff09;启动服务

sbin/start-all.sh

结果&＃xff1a;

&＃xff08;a&＃xff09;服务器

starting org.apache.spark.deploy.master.Master, logging to /opt/modules/spark-2.1.0-bin-2.6.0-cdh5.7.0/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop.out
hadoop: starting org.apache.spark.deploy.worker.Worker, logging to /opt/modules/spark-2.1.0-bin-2.6.0-cdh5.7.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop.out其中&＃xff0c;master和worker分别记录在/opt/modules/spark-2.1.0-bin-2.6.0-cdh5.7.0/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop.out和/opt/modules/spark-2.1.0-bin-2.6.0-cdh5.7.0/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop.out

4.测试

&＃xff08;1&＃xff09;启动spark-shell

&＃xff08;a&＃xff09;虚拟机

bin/spark-shell --master spark://bigdata.ibeifeng.com:7070

&＃xff08;b&＃xff09;服务器

bin/spark-shell --master spark://hadoop:7077
结果&＃xff1a;
Spark context available as &＃39;sc&＃39; (master &＃61; spark://hadoop:7077, app id &＃61; app-20190116000819-0001).
Spark session available as &＃39;spark&＃39;.
Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ &＃96;/ __/ &＃39;_//___/ .__/\_,_/_/ /_/\_\ version 2.1.0/_/Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_151)
Type in expressions to have them evaluated.
Type :help for more information.
scala>

&＃xff08;2&＃xff09;测试topN

val lines &＃61; sc.textFile("/README.md") #这个是HDFS上的路径
val words &＃61; lines.flatMap(line &＃61;> line.split(" "))
val words2 &＃61; words.map(word &＃61;> (word,1))
val wordCountRDD&＃61; words2.reduceByKey(_ &＃43; _)
wordCountRDD.sortBy(t &＃61;> -t._2).take(10)

&＃xff08;测试成功~&＃xff09;