Cloudera Docs

Advanced erasure coding configuration

You can customize the behavior of EC through a combination of the hdfs ec subcommand and the Cloudera Manager Admin Console.

This procedure provides information about configuring certain properties for EC with the Cloudera Manager Admin Console.

For information about the hdfs ec see Erasure coding CLI command.
  1. Select Clusters and choose the HDFS cluster you want to configure.
  2. Navigate to the Configuration tab and select the Erasure Coding category.
  3. Configure the advanced EC properties.
    • DataNode Striped Read Timeout: The timeout for striped reads during background data reconstruction in milliseconds.
    • DataNode Striped Read Threads: The number of threads that a DataNode can use during background data reconstruction.
    • Erasure Coding Reconstruction Weight: The relative weight of resources used by EC for data recovery. The number of blocks that must be read is based on the EC policy used. For example, RS-6-3-1024k requires six blocks to be read. Replication only requires one block to be read. Higher values result in fewer reconstruction tasks being able to run concurrently. The number of blocks required to be read to recover data is multiplied by this weight to determine the total weight of the recovery task. The total weight of the recovery task counts against the limit set with the dfs.namenode.replication.max-streams property.
    • Fallback Erasure Coding Policy: The fallback Erasure Coding policy that HDFS uses if no policy is specified when you run the -setPolicy command.