Configuring concurrent moves

Learn how to configure concurrent moves for the DataNode and Balancer to prevent the balancer from taking too many resources from the DataNode and interfering with normal cluster operations.

The property dfs.datanode.balance.max.concurrent.moves sets the maximum number of threads used by the DataNode balancer for pending moves. It is a throttling mechanism to prevent the balancer from taking too many resources from the DataNode and interfering with normal cluster operations. Increasing the value allows the balancing process to complete more quickly. Decreasing the value allows rebalancing to complete more slowly, but is less likely to compete for resources with other tasks on the DataNode. To use this property, you need to set the value on both the DataNode and the Balancer.

Perform the following steps to set the dfs.datanode.balance.max.concurrent.moves property for the DataNode and the Balancer:

  1. In the Cloudera Admin Console, select Clusters > HDFS cluster.
  2. Select the Configuration tab.
  3. Search for HDFS Service Advanced Configuration Snippet.
  4. Add the following code to the configuration field, for example, setting the value to 50.
    <property>
      <name>dfs.datanode.balance.max.concurrent.moves</name>
      <value>50</value>
    </property>
  5. Click Save Changes.
  6. Restart the DataNode or the Balancer.