Hadoop学习之路，YARN的配置与安装

篇首语：本文由编程笔记#小编为大家整理，主要介绍了Hadoop学习之路，YARN的配置与安装相关的知识，希望对你有一定的参考价值。

Hadoop中YARN的配置

etc/hadoop/mapred-site.xml文件编辑



mapreduce.framework.name</name>
yarn</value>
</property>
</configuration>


etc/hadoop/yarn-site.xml文件编辑



yarn.nodemanager.aux-services</name>
mapreduce_shuffle</value>
</property>
</configuration>


启动yarn

[vagary&＃64;vagary ~]$ jps
21049 DataNode
2187 Jps
21275 SecondaryNameNode
20926 NameNode
[vagary&＃64;vagary ~]$ start-yarn.sh
Starting resourcemanager
Starting nodemanagers
[vagary&＃64;vagary ~]$ jps
21049 DataNode
2842 Jps
21275 SecondaryNameNode
2493 NodeManager
2366 ResourceManager
20926 NameNode


然后查看对应端口号

[root&＃64;vagary ~]# netstat -nlp | grep 2366
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 2366/java
tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN 2366/java
tcp 0 0 0.0.0.0:8031 0.0.0.0:* LISTEN 2366/java
tcp 0 0 0.0.0.0:8032 0.0.0.0:* LISTEN 2366/java
tcp 0 0 0.0.0.0:8033 0.0.0.0:* LISTEN 2366/java


然后web界面查看&＃xff0c;这样子我们就算安装成功了&＃xff0c;如果访问不成功&＃xff0c;证明安全组没打开&＃xff0c;打开该端口安全组就行

然后因为8088端口经常会成为挖矿的一个点&＃xff0c;所以需要再进行一些配置&＃xff0c;伪装端口&＃xff0c;再打开yarn-site.xml文件&＃xff0c;再加一条设置&＃xff0c;这里我修改为9527了&＃xff0c;只要改成不常用的都可以


yarn.resourcemanager.webapp.address</name>
$yarn.resourcemanager.hostname:9527</value>
</property>


设置完成&＃xff0c;再重新启动yarn&＃xff0c;查看端口&＃xff0c;已经修改为9527了

[vagary&＃64;vagary hadoop]$ jps
17990 ResourceManager
18119 NodeManager
21049 DataNode
18521 Jps
21275 SecondaryNameNode
20926 NameNode
[root&＃64;vagary ~]# netstat -nlp | grep 17990
tcp 0 0 0.0.0.0:9527 0.0.0.0:* LISTEN 17990/java
tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN 17990/java
tcp 0 0 0.0.0.0:8031 0.0.0.0:* LISTEN 17990/java
tcp 0 0 0.0.0.0:8032 0.0.0.0:* LISTEN 17990/java
tcp 0 0 0.0.0.0:8033 0.0.0.0:* LISTEN 17990/java


将Hadoop的pid临时路径改到本地,编辑 hadoop-env.sh文件&＃xff0c;加入一下配置

[vagary&＃64;vagary hadoop]$ vi hadoop-env.sh


export HADOOP_PID_DIR&＃61;/home/vagary/tmp
export HADOOP_SECURE_PID_DIR&＃61;/home/vagary/tmp


运行一个mapreduce实例&＃xff0c;现在本地创建一个txt文件

[vagary&＃64;vagary data]$ vi wordcount.txt


文件内容&＃xff1a;

hadoop hdfs hdfs hive
hdfs sqoop flume java
Java Hadoop hadoop


在Hadoop上创建文件夹,这里我们选择创建级联文件夹&＃xff08;-p&＃xff09;

[vagary&＃64;vagary data]$ hdfs dfs -mkdir -p /wordcount/input


将文件上传到刚刚创建好的Hadoop文件目录中

[vagary&＃64;vagary data]$ hdfs dfs -put wordcount.txt /wordcount/input


运行个mapreduce任务&＃xff0c;调用jar&＃43;任务名&＃43;输入路径&＃43;输出路径,具体用法&＃xff1a;

[vagary&＃64;vagary hadoop]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wordcount/input /wordcount/output


会报以下错误&＃xff1a;

需要在etc/hadoop/mapred-site.xml文件下再加入一下配置&＃xff1a;


yarn.app.mapreduce.am.env</name>
HADOOP_MAPRED_HOME&＃61;$HADOOP_HOME</value>
</property>

mapreduce.map.env</name>
HADOOP_MAPRED_HOME&＃61;$HADOOP_HOME</value>
</property>

mapreduce.reduce.env</name>
HADOOP_MAPRED_HOME&＃61;$HADOOP_HOME</value>
</property>


yarn.app.mapreduce.am.env含义&＃xff1a;
用户为MR应用程序主进程添加了环境变量&＃xff0c;指定为逗号分隔的列表
mapreduce.map.env:为map添加环境变量
mapreduce.reduce.env&＃xff1a;为reduce添加环境变量
现在再重新运行任务&＃xff1a;

[vagary&＃64;vagary hadoop]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wordcount/input /wordcount/output


出现以下结果&＃xff0c;就成功了&＃xff1a;

查看结果显示&＃xff1a;

[vagary&＃64;vagary hadoop]$ hdfs dfs -cat /wordcount/output/*
2022-03-19 19:02:59,158 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted &＃61; false, remoteHostTrusted &＃61; false
Hadoop 1
Java 1
flume 1
hadoop 2
hdfs 3
hive 1
java 1
sqoop 1