Configuring HDFS trash for Solr index directory cleanup

Solr HDFS trash configuration preserves stale index directories for recovery before permanent deletion.

When Solr uses HdfsDirectoryFactory, it periodically removes old index directories that are no longer referenced after an index reload, optimize, or replication operation. By default, Solr deletes these directories permanently and immediately.

Required settings

You must configure both of the following settings. Setting only one has no effect.

Setting Location Description
solr.hdfs.trash.enabled solrconfig.xml, inside the directoryFactory element Solr-side setting. The default is false. When set to true, Solr attempts to move old index directories to HDFS Trash before deleting them. You can also set it with the -Dsolr.hdfs.trash.enabled=true JVM system property.
fs.trash.interval core-site.xml or the HDFS service configuration in Cloudera Manager HDFS setting that specifies, in minutes, how long files remain in Trash. The default is 0, which disables HDFS Trash. Because this setting is cluster-wide, it affects every HDFS client.

The HDFS fs.trash.checkpoint.interval setting controls how often Trash checkpoints are created. Set it to a value no greater than fs.trash.interval.

For example, the directoryFactory element in solrconfig.xml can contain the following setting:

<directoryFactory name="DirectoryFactory"
    class="${solr.directoryFactory:org.apache.solr.core.HdfsDirectoryFactory}">
  <bool name="solr.hdfs.trash.enabled">${solr.hdfs.trash.enabled:false}</bool>
</directoryFactory>

HDFS Trash behavior

The result depends on both settings after Solr has decided that an old index directory should be cleaned up.

solr.hdfs.trash.enabled fs.trash.interval Result
false Any value Solr deletes old index directories directly.
true Greater than 0 Solr moves old index directories to .Trash, where they remain for the configured interval.
true 0 HDFS Trash declines the request. Solr falls back to a direct delete and logs the fallback at INFO level.
true Greater than 0, but the move fails Solr logs a warning containing the underlying error and falls back to a direct delete.

Limitations

  • HDFS Trash is a best-effort recovery mechanism, not a guaranteed recovery or audit trail. Solr intentionally continues with the delete if moving the directory to Trash fails.
  • Changing fs.trash.interval changes Trash retention for every application that uses the HDFS cluster, not only Solr.
  • Directories that are successfully moved to Trash are stored under the standard Trash location for the Solr process user, typically /user/<user>/.Trash/Current/. HDFS permanently purges them when the configured interval expires.

Configure HDFS Trash for Solr

  1. Confirm with your HDFS administrator that fs.trash.interval is set to a value greater than 0. For example, set it to 1440 to retain files for 24 hours.
  2. Set solr.hdfs.trash.enabled=true in solrconfig.xml, or set the -Dsolr.hdfs.trash.enabled=true JVM system property for the Solr cores or collections where you want this behavior.
  3. After an index reload, optimize, or replication operation triggers cleanup, check the Solr logs for messages that indicate the old directory was moved to Trash instead of deleted.