Hadoop2.6.0+云centos+伪分布式只谈部署

1. 3.0.3玩不好&＃xff0c;现将2.6.0tar.gz上传到 / usr  , chmod -R hadoop:hadop hadoop-2.6.0 &＃xff0c; rm掉3.0.3
   

2.在/etc/profile中 配置java的环境配置  , hadoop环境配置

ssh免密登录配置 &＃xff08;查看之前记录&＃xff09;

3. 配置文件

hadoop-env.sh中配置java环境

core-sit.xml

官网上没有提到 端口9000这个配置&＃xff0c;但是如果不添加&＃xff0c; start-dfs.sh的时候会出现如下错误&＃xff1a;

Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.

hdfs-site.xml

4.格式化文件系统

# hadoop namenode –format

[root&＃64;zui hadoop]# hadoop namenode -format     (因为这里用到了root用户&＃xff0c; 所以start-dfs.sh如果不在root下执行&＃xff0c;启动不了namenode / datanode and secondnamenode , yarn没有关系)

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

18/07/23 17:03:28 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host &＃61; zui/182.61.17.191

STARTUP_MSG:   args &＃61; [-format]

STARTUP_MSG:   version &＃61; 2.6.0

STARTUP_MSG:   classpath &＃61;/***********各种jar包的path/

STARTUP_MSG:   build &＃61; https://git-wip-us.apache.org/repos/asf/hadoop.git -r e34                                                                                        96499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by &＃39;jenkins&＃39; on 2014-11-13T21:10 Z

STARTUP_MSG:   java &＃61; 1.8.0_152

************************************************************/

18/07/23 17:03:29 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

18/07/23 17:03:29 INFO namenode.NameNode: createNameNode [-format]

Formatting using clusterid: CID-cb98355b-6a1d-47a2-964c-48dc32752b55

18/07/23 17:03:30 INFO namenode.FSNamesystem: No KeyProvider found.

18/07/23 17:03:30 INFO namenode.FSNamesystem: fsLock is fair:true

18/07/23 17:03:30 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit&＃61;1000

18/07/23 17:03:30 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check&＃61;true

18/07/23 17:03:30 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000

18/07/23 17:03:30 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Jul 23 17:03:30

18/07/23 17:03:30 INFO util.GSet: Computing capacity for map BlocksMap

18/07/23 17:03:30 INFO util.GSet: VM type       &＃61; 64-bit

18/07/23 17:03:30 INFO util.GSet: 2.0% max memory 966.7 MB &＃61; 19.3 MB

18/07/23 17:03:30 INFO util.GSet: capacity      &＃61; 2^21 &＃61; 2097152 entries

18/07/23 17:03:30 INFO blockmanagement.BlockManager: dfs.block.access.token.enable&＃61;false

18/07/23 17:03:30 INFO blockmanagement.BlockManager: defaultReplication&＃61; 1

18/07/23 17:03:30 INFO blockmanagement.BlockManager: maxReplication&＃61; 512

18/07/23 17:03:30 INFO blockmanagement.BlockManager: minReplication&＃61; 1

18/07/23 17:03:30 INFO blockmanagement.BlockManager: maxReplicationStreams&＃61; 2

18/07/23 17:03:30 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks &＃61; false

18/07/23 17:03:30 INFO blockmanagement.BlockManager: replicationRecheckInterval&＃61; 3000

18/07/23 17:03:30 INFO blockmanagement.BlockManager: encryptDataTransfer&＃61; false

18/07/23 17:03:30 INFO blockmanagement.BlockManager: maxNumBlocksToLog&＃61; 1000

18/07/23 17:03:30 INFO namenode.FSNamesystem: fsOwner             &＃61; root (auth:SIMPLE)

18/07/23 17:03:30 INFO namenode.FSNamesystem: supergroup          &＃61; supergroup

18/07/23 17:03:30 INFO namenode.FSNamesystem: isPermissionEnabled &＃61; true

18/07/23 17:03:30 INFO namenode.FSNamesystem: HA Enabled: false

18/07/23 17:03:30 INFO namenode.FSNamesystem: Append Enabled: true

18/07/23 17:03:31 INFO util.GSet: Computing capacity for map INodeMap

18/07/23 17:03:31 INFO util.GSet: VM type       &＃61; 64-bit

18/07/23 17:03:31 INFO util.GSet: 1.0% max memory 966.7 MB &＃61; 9.7 MB

18/07/23 17:03:31 INFO util.GSet: capacity      &＃61; 2^20 &＃61; 1048576 entries

18/07/23 17:03:31 INFO namenode.NameNode: Caching file names occuring more than 10 times

18/07/23 17:03:31 INFO util.GSet: Computing capacity for map cachedBlocks

18/07/23 17:03:31 INFO util.GSet: VM type       &＃61; 64-bit

18/07/23 17:03:31 INFO util.GSet: 0.25% max memory 966.7 MB &＃61; 2.4 MB

18/07/23 17:03:31 INFO util.GSet: capacity      &＃61; 2^18 &＃61; 262144 entries

18/07/23 17:03:31 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct &＃61; 0.9990000128746033

18/07/23 17:03:31 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes &＃61; 0

18/07/23 17:03:31 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension&＃61; 30000

18/07/23 17:03:31 INFO namenode.FSNamesystem: Retry cache on namenode is enabled

18/07/23 17:03:31 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis

18/07/23 17:03:31 INFO util.GSet: Computing capacity for map NameNodeRetryCache

18/07/23 17:03:31 INFO util.GSet: VM type       &＃61; 64-bit

18/07/23 17:03:31 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB &＃61; 297.0 KB

18/07/23 17:03:31 INFO util.GSet: capacity      &＃61; 2^15 &＃61; 32768 entries

18/07/23 17:03:31 INFO namenode.NNConf: ACLs enabled? false

18/07/23 17:03:31 INFO namenode.NNConf: XAttrs enabled? true

18/07/23 17:03:31 INFO namenode.NNConf: Maximum size of an xattr: 16384

18/07/23 17:03:31 INFO namenode.FSImage: Allocated new BlockPoolId: BP-702429615-182.61.17.191-1532336611838

18/07/23 17:03:31 INFO common.Storage: Storage directory /usr/hadoop-2.6.0/data/tmp/dfs/name has been successfully formatted.

18/07/23 17:03:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >&＃61; 0

18/07/23 17:03:32 INFO util.ExitUtil: Exiting with status 0

18/07/23 17:03:32 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at zui/182.61.17.191

************************************************************/

[root&＃64;zui hadoop]# [root&＃64;zui hadoop]# hadoop namenode -format

-bash: [root&＃64;zui: command not found

[root&＃64;zui hadoop]# DEPRECATED: Use of this script to execute hdfs command is deprecated.

-bash: DEPRECATED:: command not found

[root&＃64;zui hadoop]# Instead use the hdfs command for it.

-bash: Instead: command not found

[root&＃64;zui hadoop]#

[root&＃64;zui hadoop]# 18/07/23 17:03:28 INFO namenode.NameNode: STARTUP_MSG:

-bash: 18/07/23: No such file or directory

[root&＃64;zui hadoop]# /************************************************************

-bash: /appd.log: Text file busy

[root&＃64;zui hadoop]# STARTUP_MSG: Starting NameNode

-bash: STARTUP_MSG:: command not found

[root&＃64;zui hadoop]# STARTUP_MSG:   host &＃61; zui/182.61.17.191

-bash: STARTUP_MSG:: command not found

[root&＃64;zui hadoop]# STARTUP_MSG:   args &＃61; [-format]

-bash: STARTUP_MSG:: command not found

[root&＃64;zui hadoop]# STARTUP_MSG:   version &＃61; 2.6.0

-bash: STARTUP_MSG:: command not found

格式化成功,这里我把打印的信息贴上了&＃xff0c;深入的学习是需要分析的

5.

执行 start-dfs.sh

检查 结果jps

6.通过浏览器访问 : http://公网ip:50070/ 

来张大图爽快一把

全文参考&＃xff1a; https://blog.csdn.net/liuge36/article/details/78353930

如有雷同&＃xff0c;全属抄袭

2018 07 23

Hadoop中的资源调度 &＃xff1a; yarn

mapreduce-site.xml

yarn-site.xml

切换到hadoop用户&＃xff0c;执行 start-yarn.sh&＃xff0c; 因为免密配置是在hadoop用户下操作的&＃xff0c;如果root用户&＃xff0c;需要一次次输入密码

因为之前start-dfs的操作是在root下操作的&＃xff0c;所以log文件对hadoop用户 Permission denied

检查如下;

将logs用户和组 assign给 hadoop (提示&＃xff1a;免密登录在什么用户下配置的&＃xff0c;后面hadoop任何操作都要在这个user下 1.其他用户操作不知要输入多少次密码&＃xff0c;如果一百次操作都要输入pwd你会晕挂的 2.假使前面用了root&＃xff0c;后面恍然大悟切回到hadoop用户了&＃xff0c;但是有些生成的文件是root用户和组&＃xff0c;如果hadoop下也需要操作这些目录那么明显没有权限&＃xff0c;运行检查发现100个文件&＃xff0c;运气好也许一个 chown -R就好了&＃xff0c;运气不好 100次 chown你来试试)

再次执行 start-yarn.sh

查看 &＃xff0c;为什么没有显示 namenode 和 datanode的进程&＃xff0c; 此时http://182.61.**.***:50070也还是可以访问的呀 &＃xff1f;&＃xff1f;&＃xff1f;&＃xff1f;

在浏览器输入&＃xff0c;OK&＃xff0c; 看到下面结果&＃xff0c;伪分布式搭建完成