centos7.2(linux)+spark2.1.0安装

介绍下spark在Linux上的安装.

操作系统是cent os, cent os其实是纯净版的Linux.

(1)版本

spark版本选2.x以上.2.1.0是去年中旬刚出来的.

centos7.2 64

(2)下载spark

下载地址,百度.

(3)安装

解压,放到/usr路径下,本人的是:

/usr/spark

#set spark path 
export SPARK_HOME=/usr/spark
export PATH=$SPARK_HOME/bin:$PATH

source ~/.bash_profile

./bin/spark-shell

:help
:quit

cp ./conf/spark-defaults.conf.template ./conf/spark-defaults.conf
vi ./conf/spark-defaults.conf

# Example:
# spark.master                     spark://master:7077
# spark.eventLog.enabled           true
# spark.eventLog.dir               hdfs://namenode:8021/directory
# spark.serializer                 org.apache.spark.serializer.KryoSerializer
# spark.driver.memory              5g
# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

# add min memory
#spark.executor.memory              471859200B
#add default memory
spark.executor.memory              512m

./bin/spark-shell

val textFile = sc.textFile("/testdata/words.txt")

textFile.count()