热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

yarnhadoop2.3.0installationclusterCentos64bits

cluster:n0,n1,n2n0:NameNode,ResourceManager;n1.n2:DataNode,NodeManager;1.prerequiration1

cluster:
n0,n1,n2

n0:NameNode,ResourceManager
;

n1.n2:DataNode,NodeManager;

 

1.
prerequiration

 
1.1 添加用户hm

   
#useradd hm

   
#passwd hm

 
1.2 jdk 1.6/1.7

 
1.3 ssh 无密码登录

 



1.所有机器: 使用hm用户登录
$cd /home/hm
$mkdir .ssh
2. 在namenode上生成密钥对
$ ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
2.1 .ssh目录要设成700 有执行权限
2.2 authorized_keys要设成600 否则会出错
2.3 还有ssh 登陆要加入用户名的 比如(需要密码)
$ssh n1
$ssh n2
3. 复制公钥(需要密码)
$cd .ssh
$scp authorized_keys n1:/home/hm/.ssh
$scp authorized_keys n2:/home/hm/.ssh
4.测试 (!!不需要密码)
ssh n1
ssh n2

 

2.
hadoop 通用配置

  
2.1 hadoop-env.sh

  
2.2 slave 工作节点

3.
hadoop四大组件配置

 
3.1 组件core-site.xml 



xml version="1.0" encoding="UTF-8"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFSname>
<value>hdfs://n0:9000value>
property>
<property>
<name>io.file.buffer.sizename>
<value>131072value>
property>
<property>
<name>hadoop.tmp.dirname>
<value>file:/home/hm/tempvalue>
property>
<property>
<name>hadoop.proxyuser.hm.hostsname>
<value>*value>
property>
<property>
<name>hadoop.proxyuser.hm.groupsname>
<value>*value>
property>
configuration>

 

 
3.2 组件 hdfs-site.xml



xml version="1.0" encoding="UTF-8"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.namenode.secondary.http-addressname>
<value>n0:9001value>
property>
<property>
<name>dfs.namenode.name.dirname>
<value>file:/home/hm/dfs/namevalue>
property>
<property>
<name>dfs.datanode.data.dirname>
<value>file:/home/hm/dfs/datavalue>
property>
<property>
<name>dfs.replicationname>
<value>2value>
property>
<property>
<name>dfs.webhdfs.enabledname>
<value>truevalue>
property>
configuration>

 

 
3.3 组件yarn-site.xml



xml version="1.0"?>
<configuration>

<property>
<name>yarn.nodemanager.aux-servicesname>
<value>mapreduce_shufflevalue>
property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.classname>
<value>org.apache.hadoop.mapred.ShuffleHandlervalue>
property>
<property>
<name>yarn.resourcemanager.addressname>
<value>n0:8032value>
property>
<property>
<name>yarn.resourcemanager.scheduler.addressname>
<value>n0:8030value>
property>
<property>
<name>yarn.resourcemanager.resource-tracker.addressname>
<value>n0:8031value>
property>
<property>
<name>yarn.resourcemanager.admin.addressname>
<value>n0:8033value>
property>
<property>
<name>yarn.resourcemanager.webapp.addressname>
<value>n0:8088value>
property>
configuration>

 

 
3.4 组件mapred-site.xml 



xml version="1.0"?>
xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.namename>
<value>yarnvalue>
property>
<property>
<name>mapreduce.jobhistory.addressname>
<value>n0:10020value>
property>
<property>
<name>mapreduce.jobhistory.webapp.addressname>
<value>n0:19888value>
property>
configuration>

 

 

4.
启动和停止

 4.1
启动

     
sbin/start-dfs.sh

 
    sbin/start-yarn.sh

4.2
停止

     
sbin/stop-dfs.sh

 
    sbin/stop-yarn.sh

5.测试

 

 

 

 

 

yarn hadoop-2.3.0 installation cluster Centos 64bits,布布扣,bubuko.com


推荐阅读
author-avatar
nora抹抹茶I
这个家伙很懒,什么也没留下!
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有