Replicating Directories with Thousands of Files and Subdirectories
You can increase the heap size in the hadoop-env.sh file before you replicate the data in directories that contain thousands of files and subdirectories.
- On the destination Cloudera Manager instance, go to the HDFS service page.
- Click the Configuration tab.
- Expand SCOPE and select HDFS service name (Service-Wide) option.
- Expand CATEGORY and select Advanced option.
- Locate the HDFS Replication Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh property.
-
To increase the heap size, add the key-value pair
HADOOP_CLIENT_OPTS=-Xmx<memory_value>
. For example, if you enterHADOOP_CLIENT_OPTS=-Xmx1g
, the heap size is set to 1 GB. This value should be adjusted depending on the number of files and directories being replicated. - Enter a Reason for change, and then click Save Changes to commit the changes.