Hadoop3.2.0Hadoop机架感知

Hadoop组件具有机架感知功能。例如&＃xff0c;通过将一个块的分片放在不同的机架上&＃xff0c;HDFS块放置将使用机架感知来实现容错。这可以在群集中发生网络切换故障或分区时提供数据可用性。

Hadoop主守护进程通过调用配置文件指定的外部脚本或java类来获取集群工作者的机架ID。使用java类或外部脚本进行拓扑&＃xff0c;输出必须遵循java org.apache.hadoop.net.DNSToSwitchMapping接口。接口需要保持一对一的对应关系&＃xff0c;并且拓扑信息的格式为&＃39;/ myrack/myhost&＃39;&＃xff0c;其中&＃39;/&＃39;是拓扑定界符&＃xff0c;&＃39;myrack&＃39;是机架标识符&＃xff0c;&＃39;myhost&＃39;是个人主持人。假设每个机架有一个/ 24个子网&＃xff0c;可以使用&＃39;/192.168.100.0/192.168.100.5&＃39;格式作为唯一的机架 - 主机拓扑映射。

要使用java类进行拓扑映射&＃xff0c;类名由配置文件中的net.topology.node.switch.mapping.impl参数指定。例如&＃xff0c;NetworkTopology.java包含在hadoop发行版中&＃xff0c;可以由Hadoop管理员自定义。使用Java类而不是外部脚本具有性能优势&＃xff0c;因为当新的工作节点注册自身时&＃xff0c;Hadoop不需要派生外部进程。

如果实现外部脚本&＃xff0c;将使用配置文件中的net.topology.script.file.name参数指定它。与java类不同&＃xff0c;外部拓扑脚本不包含在Hadoop发行版中&＃xff0c;由管理员提供。在分叉拓扑脚本时&＃xff0c;Hadoop将向ARGV发送多个IP地址。发送到拓扑脚本的IP地址数由net.topology.script.number.args控制&＃xff0c;默认为100.如果net.topology.script.number.args更改为1&＃xff0c;则每个拓扑脚本都会分叉DataNodes和/或NodeManagers提交的IP。

如果未设置net.topology.script.file.name或net.topology.node.switch.mapping.impl&＃xff0c;则会为任何传递的IP地址返回机架ID&＃39;/ default-rack&＃39;。虽然这种行为似乎是可取的&＃xff0c;但它可能会导致HDFS块复制出现问题&＃xff0c;因为默认行为是将一个复制块写入机架而无法执行此操作&＃xff0c;因为只有一个名为“/ default-rack”的机架。

python Example

#!/usr/bin/python
# this script makes assumptions about the physical environment.
# 1) each rack is its own layer 3 network with a /24 subnet, which
# could be typical where each rack has its own
# switch with uplinks to a central core router.
#
# &＃43;-----------&＃43;
# |core router|
# &＃43;-----------&＃43;
# / \
# &＃43;-----------&＃43; &＃43;-----------&＃43;
# |rack switch| |rack switch|
# &＃43;-----------&＃43; &＃43;-----------&＃43;
# | data node | | data node |
# &＃43;-----------&＃43; &＃43;-----------&＃43;
# | data node | | data node |
# &＃43;-----------&＃43; &＃43;-----------&＃43;
#
# 2) topology script gets list of IP&＃39;s as input, calculates network address, and prints &＃39;/network_address/ip&＃39;.import netaddr
import sys
sys.argv.pop(0) # discard name of topology script from argv list as we just want IP addressesnetmask &＃61; &＃39;255.255.255.0&＃39; # set netmask to what&＃39;s being used in your environment. The example uses a /24for ip in sys.argv: # loop over list of datanode IP&＃39;saddress &＃61; &＃39;{0}/{1}&＃39;.format(ip, netmask) # format address string so it looks like &＃39;ip/netmask&＃39; to make netaddr worktry:network_address &＃61; netaddr.IPNetwork(address).network # calculate and print network addressprint "/{0}".format(network_address)except:print "/rack-unknown" # print catch-all value if unable to calculate network address


bash Example

#!/usr/bin/env bash
# Here&＃39;s a bash example to show just how simple these scripts can be
# Assuming we have flat network with everything on a single switch, we can fake a rack topology.
# This could occur in a lab environment where we have limited nodes,like 2-8 physical machines on a unmanaged switch.
# This may also apply to multiple virtual machines running on the same physical hardware.
# The number of machines isn&＃39;t important, but that we are trying to fake a network topology when there isn&＃39;t one.
#
# &＃43;----------&＃43; &＃43;--------&＃43;
# |jobtracker| |datanode|
# &＃43;----------&＃43; &＃43;--------&＃43;
# \ /
# &＃43;--------&＃43; &＃43;--------&＃43; &＃43;--------&＃43;
# |datanode|--| switch |--|datanode|
# &＃43;--------&＃43; &＃43;--------&＃43; &＃43;--------&＃43;
# / \
# &＃43;--------&＃43; &＃43;--------&＃43;
# |datanode| |namenode|
# &＃43;--------&＃43; &＃43;--------&＃43;
#
# With this network topology, we are treating each host as a rack. This is being done by taking the last octet
# in the datanode&＃39;s IP and prepending it with the word &＃39;/rack-&＃39;. The advantage for doing this is so HDFS
# can create its &＃39;off-rack&＃39; block copy.
# 1) &＃39;echo $&＃64;&＃39; will echo all ARGV values to xargs.
# 2) &＃39;xargs&＃39; will enforce that we print a single argv value per line
# 3) &＃39;awk&＃39; will split fields on dots and append the last field to the string &＃39;/rack-&＃39;. If awk
# fails to split on four dots, it will still print &＃39;/rack-&＃39; last field valueecho $&＃64; | xargs -n 1 | awk -F &＃39;.&＃39; &＃39;{print "/rack-"$NF}&＃39;

原文链接: https://hadoop.apache.org/docs/r3.2.0/