Cloudera Docs

Replicating Directories with Thousands of Files and Subdirectories

Replicating data includes a directory with several hundred thousand files or subdirectories.

  1. On the destination Cloudera Manager instance, go to the HDFS service page.
  2. Click the Configuration tab.
  3. Expand SCOPE and select HDFS service name (Service-Wide) option.
  4. Locate the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh property.
  5. To increase the heap size, add the key-value pair HADOOP_CLIENT_OPTS=-Xmx[***memory_value***]. For example, if you enter HADOOP_CLIENT_OPTS=-Xmx1g, the heap size is set to 1 GB. This value should be adjusted depending on the number of files and directories being replicated.
  6. Enter a Reason for change, and then click Save Changes to commit the changes.