To customize the Hadoop configuration:

1. Copy the default configuration to your custom directory:

   $ sudo cp -r /etc/hadoop/conf.empty /etc/hadoop/conf.my_cluster

   You can call this configuration anything you like; in this example, it's called my_cluster.

2. Set alternatives to point to your custom directory, as follows.

   To manually set the configuration on Red Hat-compatible systems:

   $ sudo alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.my_cluster 50
   $ sudo alternatives --set hadoop-conf /etc/hadoop/conf.my_cluster

   To manually set the configuration on Ubuntu and SLES systems:

   $ sudo update-alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.my_cluster 50
   $ sudo update-alternatives --set hadoop-conf /etc/hadoop/conf.my_cluster

   For more information on alternatives, see the update-alternatives(8) man page on Ubuntu and SLES systems or the alternatives(8) man page On Red Hat-compatible systems.

  Important:

When performing the configuration tasks in this section, and when you go on to deploy MRv1 or YARN, edit the configuration files in your custom directory (for example /etc/hadoop/conf.my_cluster). Do not create your custom configuration in the default directory /etc/hadoop/conf.empty.

The following tables show the most important properties that you must configure for your cluster.

  Note:

For information on other important configuration properties, and the configuration files, see the Apache Cluster Setup page.

<property>
 <name>fs.defaultFS</name>
 <value>hdfs://namenode-host.company.com:8020</value>
</property>

<property>
 <name>dfs.permissions.superusergroup</name>
 <value>hadoop</value>
</property>

You need to specify, create, and assign the correct permissions to the local directories where you want the HDFS daemons to store data. You specify the directories by configuring the following two properties in the hdfs-site.xml file.

  Note:

dfs.data.dir and dfs.name.dir are deprecated; you should use dfs.datanode.data.dir and dfs.namenode.name.dir instead, though dfs.data.dir and dfs.name.dir will still work.

Sample configuration:

hdfs-site.xml on the NameNode:

<property>
 <name>dfs.namenode.name.dir</name>
 <value>file:///data/1/dfs/nn,file:///nfsmount/dfs/nn</value>
</property>

hdfs-site.xml on each DataNode:

<property>
 <name>dfs.datanode.data.dir</name>
 <value>file:///data/1/dfs/dn,file:///data/2/dfs/dn,file:///data/3/dfs/dn,file:///data/4/dfs/dn</value>
</property>

After specifying these directories as shown above, you must create the directories and assign the correct file permissions to them on each node in your cluster.

In the following instructions, local path examples are used to represent Hadoop parameters. Change the path examples to match your configuration.

Local directories:

• The dfs.name.dir or dfs.namenode.name.dir parameter is represented by the /data/1/dfs/nn and /nfsmount/dfs/nn path examples.
• The dfs.data.dir or dfs.datanode.data.dir parameter is represented by the /data/1/dfs/dn, /data/2/dfs/dn, /data/3/dfs/dn, and /data/4/dfs/dn examples.

To configure local storage directories for use by HDFS:

1. On a NameNode host: create the dfs.name.dir or dfs.namenode.name.dir local directories:

   $ sudo mkdir -p /data/1/dfs/nn /nfsmount/dfs/nn

     Important:

   If you are using High Availability (HA), you should not configure these directories on an NFS mount; configure them on local storage.

2. On all DataNode hosts: create the dfs.data.dir or dfs.datanode.data.dir local directories:

   $ sudo mkdir -p /data/1/dfs/dn /data/2/dfs/dn /data/3/dfs/dn /data/4/dfs/dn

3. Configure the owner of the dfs.name.dir or dfs.namenode.name.dir directory, and of the dfs.data.dir or dfs.datanode.data.dir directory, to be the hdfs user:

   $ sudo chown -R hdfs:hdfs /data/1/dfs/nn /nfsmount/dfs/nn /data/1/dfs/dn /data/2/dfs/dn /data/3/dfs/dn /data/4/dfs/dn

   Here is a summary of the correct owner and permissions of the local directories:

   Directory	Owner	Permissions (see Footnote 1)
   dfs.name.dir or dfs.namenode.name.dir	hdfs:hdfs	drwx------
   dfs.data.dir or dfs.datanode.data.dir	hdfs:hdfs	drwx------

   Footnote: 1 The Hadoop daemons automatically set the correct permissions for you on dfs.data.dir or dfs.datanode.data.dir. But in the case of dfs.name.dir or dfs.namenode.name.dir, permissions are currently incorrectly set to the file-system default, usually drwxr-xr-x (755). Use the chmod command to reset permissions for these dfs.name.dir or dfs.namenode.name.dir directories to drwx------ (700); for example:

   $ sudo chmod 700 /data/1/dfs/nn /nfsmount/dfs/nn

   or

   $ sudo chmod go-rx /data/1/dfs/nn /nfsmount/dfs/nn

     Note:

   If you specified nonexistent directories for the dfs.data.dir or dfs.datanode.data.dir property in the hdfs-site.xml file, CDH 5 will shut down. (In previous releases, CDH silently ignored nonexistent directories for dfs.data.dir.)

By default, the failure of a single dfs.data.dir or dfs.datanode.data.dir will cause the HDFS DataNode process to shut down, which results in the NameNode scheduling additional replicas for each block that is present on the DataNode. This causes needless replications of blocks that reside on disks that have not failed.

To prevent this, you can configure DataNodes to tolerate the failure of dfs.data.dir or dfs.datanode.data.dir directories; use the dfs.datanode.failed.volumes.tolerated parameter in hdfs-site.xml. For example, if the value for this parameter is 3, the DataNode will only shut down after four or more data directories have failed. This value is respected on DataNode startup; in this example the DataNode will start up as long as no more than three directories have failed.

  Note:

It is important that dfs.datanode.failed.volumes.tolerated not be configured to tolerate too many directory failures, as the DataNode will perform poorly if it has few functioning data directories.

Before starting the NameNode for the first time you need to format the file system.

  Important:

• Make sure you format the NameNode as user hdfs.
• If you are re-formatting the NameNode, keep in mind that this invalidates the DataNode storage locations, so you should remove the data under those locations after the NameNode is formatted.

$ sudo -u hdfs hdfs namenode -format

  Note:

If Kerberos is enabled, do not use commands in the form sudo -u <user> hadoop <command>; they will fail with a security error. Instead, use the following commands: $ kinit <user> (if you are using a password) or $ kinit -kt <keytab> <principal> (if you are using a keytab) and then, for each command executed by this user, $ <command>

You'll get a confirmation prompt; for example:

Re-format filesystem in /data/namedir ? (Y or N)

  Note: Respond with an upper-case Y; if you use lower case, the process will abort.

You should configure the NameNode to write to multiple storage directories, including one remote NFS mount. To keep NameNode processes from hanging when the NFS server is unavailable, configure the NFS mount as a soft mount (so that I/O requests that time out fail rather than hang), and set other options as follows:

tcp,soft,intr,timeo=10,retrans=10

These options configure a soft mount over TCP; transactions will be retried ten times (retrans=10) at 1-second intervals (timeo=10) before being deemed to have failed.

Example:

mount -t nfs -o tcp,soft,intr,timeo=10,retrans=10, <server>:<export> <mount_point>

where <server> is the remote host, <export> is the exported file system, and <mount_point> is the local mount point.

  Note:

Cloudera recommends similar settings for shared HA mounts, as in the example that follows.

Example for HA:

mount -t nfs -o tcp,soft,intr,timeo=50,retrans=12, <server>:<export> <mount_point>

Note that in the HA case timeo should be set to 50 (five seconds), rather than 10 (1 second), and retrans should be set to 12, giving an overall timeout of 60 seconds.

For more information, see the man pages for mount and nfs.

  Important:

The Secondary NameNode does not provide failover or High Availability (HA). If you intend to configure HA for the NameNode, skip this section: do not install or configure the Secondary NameNode (the Standby NameNode performs checkpointing). After completing the HA software configuration, follow the installation instructions under Deploying HDFS High Availability.

In non-HA deployments, configure a Secondary NameNode that will periodically merge the EditLog with the FSImage, creating a new FSImage which incorporates the changes which were in the EditLog. This reduces the amount of disk space consumed by the EditLog on the NameNode, and also reduces the restart time for the Primary NameNode.

A standard Hadoop cluster (not a Hadoop Federation or HA configuration), can have only one Primary NameNode plus one Secondary NameNode. On production systems, the Secondary NameNode should run on a different machine from the Primary NameNode to improve scalability (because the Secondary NameNode does not compete with the NameNode for memory and other resources to create the system snapshot) and durability (because the copy of the metadata is on a separate machine that is available if the NameNode hardware fails).

  Important:

The trash feature is disabled by default. Cloudera recommends that you enable it on all production clusters.

The Hadoop trash feature helps prevent accidental deletion of files and directories. If trash is enabled and a file or directory is deleted using the Hadoop shell, the file is moved to the .Trash directory in the user's home directory instead of being deleted. Deleted files are initially moved to the Current sub-directory of the .Trash directory, and their original path is preserved. If trash checkpointing is enabled, the Current directory is periodically renamed using a timestamp. Files in .Trash are permanently removed after a user-configurable time delay. Files and directories in the trash can be restored simply by moving them to a location outside the .Trash directory.

  Note:

The trash feature works by default only for files and directories deleted using the Hadoop shell. Files or directories deleted programmatically using other interfaces (WebHDFS or the Java APIs, for example) are not moved to trash, even if trash is enabled, unless the program has implemented a call to the trash functionality. (Hue, for example, implements trash as of CDH 4.4.)

Users can bypass trash when deleting files using the shell by specifying the -skipTrash option to the hadoop fs -rm -r command. This can be useful when it is necessary to delete files that are too large for the user's quota.

Trash is configured with the following properties in the core-site.xml file:

For example, to enable trash so that files deleted using the Hadoop shell are not deleted for 24 hours, set the value of the fs.trash.interval property in the server's core-site.xml file to a value of 1440.

  Note:

The period during which a file remains in the trash starts when the file is moved to the trash, not when the file is last modified.

You can configure HDFS to distribute writes on each DataNode in a manner that balances out available storage among that DataNode's disk volumes.

By default a DataNode writes new block replicas to disk volumes solely on a round-robin basis. You can configure a volume-choosing policy that causes the DataNode to take into account how much space is available on each volume when deciding where to place a new replica.

To configure storage balancing, set the following properties in hdfs-site.xml.

  Note: Keep in mind that if usage is markedly imbalanced among a given DataNode's storage volumes when you enable storage balancing, throughput on that DataNode will be affected initially, as writes are disproportionately directed to the under-utilized volumes.

  Note:

To configure HttpFs instead, see HttpFS Installation.

If you want to use WebHDFS, you must first enable it.

To enable WebHDFS:

Set the following property in hdfs-site.xml:

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>

To enable numeric usernames in WebHDFS:

^[A-Za-z_][A-Za-z0-9._-]*[$]?$

<property>
  <name>dfs.webhdfs.user.provider.user.pattern</name>
  <value>^[A-Za-z0-9_][A-Za-z0-9._-]*[$]?$</value>
</property>

  Important: The username pattern should be compliant with the requirements of the operating system in use. Hence, Cloudera recommends you use the default pattern and avoid modifying the dfs.webhdfs.user.provider.user.pattern property when possible.

  Note:

If you want to use WebHDFS in a secure cluster, you must set additional properties to configure secure WebHDFS. For instructions, see the CDH 5 Security Guide .

If you have installed LZO, configure it as follows.

To configure LZO:

Set the following property in core-site.xml.

  Note:

If you copy and paste the value string, make sure you remove the line-breaks and carriage returns, which are included below because of page-width constraints.

<property>
  <name>io.compression.codecs</name>
  <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

For more information about LZO, see Using LZO Compression.

To to deploy MRv1 or YARN, and start HDFS services if you have not already done so, see

• Deploying MapReduce v2 (YARN) on a Cluster or
• Deploying MapReduce v1 (MRv1) on a Cluster