Cloudera Docs

Using dfs.datanode.max.transfer.threads with HBase

You must configure the dfs.datanode.max.transfer.threads with HBase to specify the maximum number of files that a DataNode can serve at any one time.

A Hadoop HDFS DataNode has an upper bound on the number of files that it can serve at any one time. The upper bound is controlled by the dfs.datanode.max.transfer.threads property. Before loading, make sure you have configured this property to at least 4096.
  1. In Cloudera Manager navigate to HDFS > Configuration.
  2. Find the DataNode Advanced Configuration Snippet (Safety-Valve) for hdfs-site.xml property.
  3. Click the plus icon to add a new property:
    • Name: dfs.datanode.max.transfer.threads
    • Value: at least 4096
  4. Clik Save Changes.
  5. Restart the HDFS service.
If the value is not set to an appropriate value, strange failures can occur and an error message about exceeding the number of transfer threads will be added to the DataNode logs. Other error messages about missing blocks are also logged, such as the following:
06/12/14 20:10:31 INFO hdfs.DFSClient: Could not obtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry…