Cloudera Docs

HDFS storage demands due to retained HDFS trash

Using Cloudera Manager you can either change the HDFS trash setting or disable the use of the HDFS trash entirely, for your Operational Database powered by Apache Accumulo (OpDB) deployment.

By default, if HDFS trash is enabled, Accumulo uses it for all files it deletes. This includes write‐ahead logs and long‐term storage files that were blocked due to compaction. By default, the retention period for the HDFS trash is 24 hours. On Accumulo installations with a heavy write workload, this can result in a large amount of data accumulating in the trash folder for the service user.

There are three workarounds:
  • Periodically run the hdfs dfs ­-expunge command as the Accumulo service user. The command must be run twice each time you want to purge a backlog of data. On the first run, a trash checkpoint is created. On the second run, the checkpoint created with the first run is removed with immediate effect.
  • Change the HDFS trash settings in Cloudera Manager.
  • Disable OpDB’s use of the HDFS trash entirely.