Chapter 4. Configuring Rack Awareness on HDP

Create a topology script and data file. The topology script MUST be executable.

Sample Topology Script

File name: rack-topology.sh

#!/bin/bash

# Adjust/Add the property "net.topology.script.file.name"
# to core-site.xml with the "absolute" path the this
# file.  ENSURE the file is "executable".

# Supply appropriate rack prefix
RACK_PREFIX=default

# To test, supply a hostname as script input:
if [ $# -gt 0 ]; then

CTL_FILE=${CTL_FILE:-"rack_topology.data"}

HADOOP_CONF=${HADOOP_CONF:-"/etc/hadoop/conf"} 

if [ ! -f ${HADOOP_CONF}/${CTL_FILE} ]; then
  echo -n "/$RACK_PREFIX/rack "
  exit 0
fi

while [ $# -gt 0 ] ; do
  nodeArg=$1
  exec< ${HADOOP_CONF}/${CTL_FILE}
  result=""
  while read line ; do
    ar=( $line )
    if [ "${ar[0]}" = "$nodeArg" ] ; then
      result="${ar[1]}"
    fi
  done
  shift
  if [ -z "$result" ] ; then
    echo -n "/$RACK_PREFIX/rack "
  else
    echo -n "/$RACK_PREFIX/rack_$result "
  fi
done

else
  echo -n "/$RACK_PREFIX/rack "
fi

Sample Topology Data File

File name: rack_topology.data

# This file should be:
#  - Placed in the /etc/hadoop/conf directory
#    - On the Namenode (and backups IE: HA, Failover, etc)
#    - On the Job Tracker OR Resource Manager (and any Failover JT's/RM's) 
# This file should be placed in the /etc/hadoop/conf directory.

# Add Hostnames to this file. Format <host ip> <rack_location>
192.168.2.10 01
192.168.2.11 02
192.168.2.12 03

Copy both of these files to the /etc/hadoop/conf directory on all cluster nodes.

Run the rack-topology.sh script to ensure that it returns the correct rack information for each host.

Stop HDFS using the instructions on this page.

Add the following property to core-site.xml:

<property>
<name>net.topology.script.file.name</name> 
<value>/etc/hadoop/conf/rack-topology.sh</value>
</property>

By default the topology script will process up to 100 requests per invocation. You can also specify a different number of requests with the net.topology.script.number.args property. For example:

<property> 
<name>net.topology.script.number.args</name> 
<value>75</value>
</property>