Preventing Accidental Deletion of Files
You can prevent accidental deletion of files by enabling the Trash
feature for HDFS. For additional information regarding HDFS trash configuration, see
HDFS Architecture.
You might still cause irrecoverable data loss if the -skipTrash and
-R options are accidentally used on directories with a large number
of files. You can obtain an additional layer of protection by using the
-safely option to the fs shell -rm command. The
fs shell -rm command checks the
hadoop.shell.safely.delete.limit.num.files property from
core-site.xml file, even if you specify
-skipTrash. By specifying the -safely option, the
-rm command requires that you confirm if the number of files to
be deleted is greater than the limit specified by the assigned value. The default limit
for value is 100, referring to 100 files.
This confirmation warning is disabled if value is set at
0 or the -safely is not specified to the
-rm command.
To enable the hadoop.shell.safely.delete.limit.num.files
property, add the following lines to core-site.xml:
<property> <name>hadoop.shell.safely.delete.limit.num.files</name> <value>100</value> <description>Used by -safely option of hadoop fs shell -rm command to avoid accidental deletion of large directories.</description> </property>
In the following example, the
hadoop.shell.safely.delete.limit.num.files property with an
associated value of 10 has been added to
core-site.xml with -skipTrash . In this
example, fs shell -r prompts deletion of a directory with only
10 files. It does not prompt if trash is
enabled and -skipTrash is not.
[ambari-qa@c6405 current]$ hdfs dfs -ls -R /tmp/test1 -rw-r--r-- 3 ambari-qa hdfs 2413 2016-10-20 20:57 /tmp/test1/capacity-scheduler.xml -rw-r--r-- 3 ambari-qa hdfs 4435 2016-10-20 20:57 /tmp/test1/core-site.xml -rw-r--r-- 3 ambari-qa hdfs 1308 2016-10-20 20:57 /tmp/test1/hadoop-policy.xml -rw-r--r-- 3 ambari-qa hdfs 8071 2016-10-20 20:57 /tmp/test1/hdfs-site.xml -rw-r--r-- 3 ambari-qa hdfs 3518 2016-10-20 20:57 /tmp/test1/kms-acls.xml -rw-r--r-- 3 ambari-qa hdfs 5511 2016-10-20 20:57 /tmp/test1/kms-site.xml -rw-r--r-- 3 ambari-qa hdfs 7339 2016-10-20 20:57 /tmp/test1/mapred-site.xml -rw-r--r-- 3 ambari-qa hdfs 884 2016-10-20 20:57 /tmp/test1/ssl-client.xml -rw-r--r-- 3 ambari-qa hdfs 1000 2016-10-20 20:57 /tmp/test1/ssl-server.xml -rw-r--r-- 3 ambari-qa hdfs 20349 2016-10-20 20:57 /tmp/test1/yarn-site.xml [ambari-qa@c6405 current]$ hdfs dfs -rm -R /tmp/test1 16/10/20 20:58:37 INFO fs.TrashPolicyDefault: Moved: 'hdfs://c6403.ambari.apache.org:8020/tmp/test1' to trash at: hdfs://c6403.ambari.apache.org:8020/user/ambari-qa/.Trash/Current/tmp/test1
The following example deletes files without prompting or moving to the trash:
[ambari-qa@c6405 current]$ hdfs dfs -ls -R /tmp/test2 -rw-r--r-- 3 ambari-qa hdfs 2413 2016-10-20 20:59 /tmp/test2/capacity-scheduler.xml -rw-r--r-- 3 ambari-qa hdfs 4435 2016-10-20 20:59 /tmp/test2/core-site.xml -rw-r--r-- 3 ambari-qa hdfs 1308 2016-10-20 20:59 /tmp/test2/hadoop-policy.xml -rw-r--r-- 3 ambari-qa hdfs 8071 2016-10-20 20:59 /tmp/test2/hdfs-site.xml -rw-r--r-- 3 ambari-qa hdfs 3518 2016-10-20 20:59 /tmp/test2/kms-acls.xml -rw-r--r-- 3 ambari-qa hdfs 5511 2016-10-20 20:59 /tmp/test2/kms-site.xml -rw-r--r-- 3 ambari-qa hdfs 7339 2016-10-20 20:59 /tmp/test2/mapred-site.xml -rw-r--r-- 3 ambari-qa hdfs 884 2016-10-20 20:59 /tmp/test2/ssl-client.xml -rw-r--r-- 3 ambari-qa hdfs 1000 2016-10-20 20:59 /tmp/test2/ssl-server.xml -rw-r--r-- 3 ambari-qa hdfs 20349 2016-10-20 20:59 /tmp/test2/yarn-site.xml [ambari-qa@c6405 current]$ hdfs dfs -rm -R -skipTrash /tmp/test2 Deleted /tmp/test2
The following example prompts for you to confirm file deletion if the number of files
to be deleted is greater than the value specified to
hadoop.shell.safely.delete.limit.num.files:
[ambari-qa@c6405 current]$ hdfs dfs -ls -R /tmp/test3 -rw-r--r-- 3 ambari-qa hdfs 2413 2016-10-20 21:00 /tmp/test3/capacity-scheduler.xml -rw-r--r-- 3 ambari-qa hdfs 4435 2016-10-20 21:00 /tmp/test3/core-site.xml -rw-r--r-- 3 ambari-qa hdfs 1308 2016-10-20 21:00 /tmp/test3/hadoop-policy.xml -rw-r--r-- 3 ambari-qa hdfs 8071 2016-10-20 21:00 /tmp/test3/hdfs-site.xml -rw-r--r-- 3 ambari-qa hdfs 3518 2016-10-20 21:00 /tmp/test3/kms-acls.xml -rw-r--r-- 3 ambari-qa hdfs 5511 2016-10-20 21:00 /tmp/test3/kms-site.xml -rw-r--r-- 3 ambari-qa hdfs 7339 2016-10-20 21:00 /tmp/test3/mapred-site.xml -rw-r--r-- 3 ambari-qa hdfs 884 2016-10-20 21:00 /tmp/test3/ssl-client.xml -rw-r--r-- 3 ambari-qa hdfs 1000 2016-10-20 21:00 /tmp/test3/ssl-server.xml -rw-r--r-- 3 ambari-qa hdfs 20349 2016-10-20 21:00 /tmp/test3/yarn-site.xml [ambari-qa@c6405 current]$ hdfs dfs -rm -R -skipTrash -safely /tmp/test3 Proceed deleting 10 files? (Y or N) N Delete aborted at user request.
![[Warning]](../common/images/admon/warning.png) | Warning |
|---|---|
Using the |