spark之hadoop2.6环境搭建笔记

一 spark、hadoop、yarn关系

spark &＃xff1a;计算
hadoop&＃xff1a;存储
yarn&＃xff1a; 资源管理

在这里主要配置hdfs和yarn
hdfs
yarn
mapreduce(计算框架&＃xff0c; spark&＃xff09;

yarn&＃xff1a; 主进程&＃xff1a;resourcemanager
yarn的开&＃xff1a; sbin/start-yarn.sh
yarn的关闭&＃xff1a;sbin/stop-yarn.sh
登录的url&＃xff1a;http://localhost:8088

hdfs:
namenode 进程
datanode 进程
dfs的开与关&＃xff1a; sbin/start-dfs.sh , sbin/stop-dfs.sh
登录url&＃xff1a;http://localhost:50070

二 hadoop2.6集群环境搭建

1. hadoop 下载&＃xff0c;解压

2. 设置环境变量
   2.1 HADOOP_HOME 设置
   2.2 hadoop_CONF_DIR 设置 $HADOOP_HOME/etc/hadoop

   2.3 YARN_CONF_DIR 设置 $HADOOP_HOME/etc/hadoop
   具体见&＃xff1a;

   vim ~/bashrc
export JAVA_HOME&＃61;/usr/lib/java/jdk1.8.0_45
export JRE_HOME&＃61;${JAVA_HOME}/jre
export HADOOP_HOME&＃61;/usr/local/hadoop/hadoop-2.6.0
export CLASS_PATH&＃61;.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH&＃61;${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
export HADOOP_CONF_DIR&＃61;${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR&＃61;${HADOOP_HOME}/etc/hadoop

3. core-site.xml 设置

<configuration><property><name>fs.default.namename><value>hdfs://localhost:9000value>property><property><name>hadoop.tmp.dirname><value>/usr/local/hadoop/hadoop-2.6.0/tmpvalue>property>configuration>

1. hdfs-site.xml 设置

<configuration><property><name>dfs.replicationname><value>1value>property><property><name>dfs.name.dirname><value>/usr/local/hadoop/hadoop-2.6.0/dfs/namevalue>property><property><name>dfs.data.dirname><value>/usr/local/hadoop/hadoop-2.6.0/dfs/datavalue>property>configuration>

1. mapred-site.xml 设置

<configuration><property><name>mapred.job.trackername><value>localhost:9001value>property>configuration>

1. hadoop-env.sh设置 $JAVA_HOME

# The java implementation to use.export JAVA_HOME&＃61;/usr/lib/java/jdk1.8.0_45

1. YARN-env.sh设置 $JAVA_HOME

export JAVA_HOME&＃61;/usr/lib/java/jdk1.8.0_45