Configuring HDFS trash for Solr index directory cleanup
Solr HDFS trash configuration preserves stale index directories for recovery before permanent deletion.
When Solr uses HdfsDirectoryFactory, it periodically removes old index
directories that are no longer referenced after an index reload, optimize, or replication
operation. By default, Solr deletes these directories permanently and immediately.
Required settings
You must configure both of the following settings. Setting only one has no effect.
| Setting | Location | Description |
|---|---|---|
solr.hdfs.trash.enabled |
solrconfig.xml, inside the
directoryFactory element |
Solr-side setting. The default is false. When set to
true, Solr attempts to move old index directories to HDFS Trash
before deleting them. You can also set it with the
-Dsolr.hdfs.trash.enabled=true JVM system property. |
fs.trash.interval |
core-site.xml or the HDFS service configuration in
Cloudera Manager |
HDFS setting that specifies, in minutes, how long files remain in Trash. The
default is 0, which disables HDFS Trash. Because this setting is
cluster-wide, it affects every HDFS client. |
The HDFS fs.trash.checkpoint.interval setting controls how often Trash
checkpoints are created. Set it to a value no greater than
fs.trash.interval.
For example, the directoryFactory element in
solrconfig.xml can contain the following setting:
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:org.apache.solr.core.HdfsDirectoryFactory}">
<bool name="solr.hdfs.trash.enabled">${solr.hdfs.trash.enabled:false}</bool>
</directoryFactory>
HDFS Trash behavior
The result depends on both settings after Solr has decided that an old index directory should be cleaned up.
solr.hdfs.trash.enabled |
fs.trash.interval |
Result |
|---|---|---|
false |
Any value | Solr deletes old index directories directly. |
true |
Greater than 0 |
Solr moves old index directories to .Trash, where they remain for the configured interval. |
true |
0 |
HDFS Trash declines the request. Solr falls back to a direct delete and logs the fallback at INFO level. |
true |
Greater than 0, but the move fails |
Solr logs a warning containing the underlying error and falls back to a direct delete. |
Limitations
- HDFS Trash is a best-effort recovery mechanism, not a guaranteed recovery or audit trail. Solr intentionally continues with the delete if moving the directory to Trash fails.
- Changing
fs.trash.intervalchanges Trash retention for every application that uses the HDFS cluster, not only Solr. - Directories that are successfully moved to Trash are stored under the standard Trash location for the Solr process user, typically /user/<user>/.Trash/Current/. HDFS permanently purges them when the configured interval expires.
Configure HDFS Trash for Solr
- Confirm with your HDFS administrator that
fs.trash.intervalis set to a value greater than0. For example, set it to1440to retain files for 24 hours. - Set
solr.hdfs.trash.enabled=trueinsolrconfig.xml, or set the-Dsolr.hdfs.trash.enabled=trueJVM system property for the Solr cores or collections where you want this behavior. - After an index reload, optimize, or replication operation triggers cleanup, check the Solr logs for messages that indicate the old directory was moved to Trash instead of deleted.
