Cloudera Docs

Replicating directories with thousands of files and subdirectories

Before you replicate the data in directories that has thousands of files and subdirectories, increase the heap size in the hadoop-env.sh file.

  1. On the destination Cloudera Manager instance, go to the HDFS service page.
  2. Click the Configuration tab.
  3. Expand Scope and then select HDFS service name (Service-Wide) option.
  4. Expand CategoryAdvanced and select Advanced.
  5. Locate the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh property.
  6. Increase the heap size by adding a key-value pair, for instance, HADOOP_CLIENT_OPTS=-Xmx1g. In this example, 1g sets the heap size to 1 GB. This value should be adjusted depending on the number of files and directories being replicated.
  7. Enter a Reason for change, and then click Save Changes to commit the changes.