Chapter 3. Setting Up the Hadoop Configuration

This section describes how to set up and edit the deployment configuration files for HDFS and MapReduce.

Use the following instructions to set up Hadoop configuration files:

We strongly suggest that you edit and source the files included in Download Companion Files).
Alternatively, you can also copy the contents to your ~/.bash_profile) to set up these environment variables in your environment.
From the downloaded scripts.zip file, extract the files from the configuration_files/core_hadoopdirectory to a temporary directory.

Modify the configuration files.

In the temporary directory, locate the following files and modify the properties based on your environment. Search for TODO in the files for the properties to replace. See Define Environment Parameters for more information.

Edit the core-site.xml file and modify the following properties:

<property>       
 <name>fs.default.name</name>       
 <value>hdfs://$namenode.full.hostname:8020</value>  
 <description>Enter your NameNode hostname</description>
</property>

<property>       
 <name>fs.checkpoint.dir</name>       
 <value>/grid/hadoop/hdfs/snn,/grid1/hadoop/hdfs/snn,/grid2/hadoop/hdfs/snn</value>  
 <description>A comma separated list of paths. Use the list of directories from $FS_CHECKPOINT_DIR. 
                For example, /grid/hadoop/hdfs/snn,sbr/grid1/hadoop/hdfs/snn,sbr/grid2/hadoop/hdfs/snn </description>
</property>

Edit the hdfs-site.xml file and modify the following properties:

<property>       
 <name>dfs.name.dir</name>       
 <value>/grid/hadoop/hdfs/nn,/grid1/hadoop/hdfs/nn</value>  
 <description>Comma separated list of paths. Use the list of directories from $DFS_NAME_DIR.  
                For example, /grid/hadoop/hdfs/nn,/grid1/hadoop/hdfs/nn.</description>
</property>

<property>       
 <name>dfs.data.dir</name>       
 <value>/grid/hadoop/hdfs/dn,/grid1/hadoop/hdfs/dn</value>  
 <description>Comma separated list of paths. Use the list of directories from $DFS_DATA_DIR.  
                For example, /grid/hadoop/hdfs/dn,/grid1/hadoop/hdfs/dn.</description>
</property>

<property>       
 <name>dfs.http.address</name>       
 <value>$namenode.full.hostname:50070</value>  
 <description>Enter your NameNode hostname for http access.</description>
</property>

<property>       
 <name>dfs.secondary.http.address</name>       
 <value>$secondarynamenode.full.hostname:50090</value>  
 <description>Enter your Secondary NameNode hostname.</description>
</property>

<property>       
 <name>dfs.https.address</name>       
 <value>$namenode.full.hostname:50470</value>  
 <description>Enter your NameNode hostname for https access.</description>
</property>

Note

	Note
The value of NameNode new generation size should be 1/8 of maximum heap size (`-Xmx`). Please check this value, as the default setting may not be accurate. To change the default value, edit the `/etc/hadoop/conf/hadoop-env.sh` file and change the value of the `-XX:MaxnewSize` parameter to 1/8th the value of the maximum heap size (`-Xmx`) parameter. Also ensure that the NameNode and Secondary NameNode have identical memory settings.

The value of NameNode new generation size should be 1/8 of maximum heap size (-Xmx). Please check this value, as the default setting may not be accurate. To change the default value, edit the /etc/hadoop/conf/hadoop-env.sh file and change the value of the -XX:MaxnewSize parameter to 1/8th the value of the maximum heap size (-Xmx) parameter. Also ensure that the NameNode and Secondary NameNode have identical memory settings.

Edit the mapred-site.xml file and modify the following properties:

<property>       
 <name>mapred.job.tracker</name>       
 <value>$jobtracker.full.hostname:50300</value>  
 <description>Enter your JobTracker hostname.</description>
</property>

<property>       
 <name>mapred.job.tracker.http.address</name>       
 <value>$jobtracker.full.hostname:50030</value>  
 <description>Enter your JobTracker hostname.</description>
</property>

<property>       
 <name>mapred.local.dir</name>       
 <value>/grid/hadoop/mapred,/grid1/hadoop/mapred</value>  
 <description>Comma separated list of paths. Use the list of directories from $MAPREDUCE_LOCAL_DIR</description>
</property>

<property>       
 <name>mapreduce.tasktracker.group</name>       
 <value>hadoop</value>  
 <description>Enter your group. Use the value of $HADOOP_GROUP</description>
</property>

<property>       
 <name>mapreduce.history.server.http.address</name>       
 <value>$jobtracker.full.hostname:51111</value>  
 <description>Enter your JobTracker hostname</description>
</property>

Edit the taskcontroller.cfg file and modify the following property:

<property>       
 <name>mapred.local.dir</name>       
 <value>/grid/hadoop/mapred,/grid1/hadoop/mapred</value>  
 <description>Comma separated list of paths. Use the list of directories from $MAPREDUCE_LOCAL_DIR</description>
</property>

Copy the configuration files.
1. Replace the installed Hadoop configs with the modified core_hadoop configuration files and set appropriate permissions.
```
rm -rf $HADOOP_CONF_DIR
mkdir -p $HADOOP_CONF_DIR
```
2. Copy all the modified configuration files in core_hadoop to $HADOOP_CONF_DIR on all nodes.
3. Set appropriate permissions.
```
chmod a+x $HADOOP_CONF_DIR/
chown -R $HDFS_USER:$HADOOP_GROUP $HADOOP_CONF_DIR/../
chmod -R 755 $HADOOP_CONF_DIR/../
```
where $HADOOP_CONF_DIR is the directory for storing the Hadoop configuration files. For example, /etc/hadoop/conf.

Legal notices