Deleting Objects on S3
The rm command deletes objects and directories full of objects. If the
object store is eventually consistent, fs ls commands and other accessors may
briefly return the details of the now-deleted objects; this is an artifact of object stores
which cannot be avoided.
If the filesystem client is configured to copy files to a trash directory, the trash
directory is in the bucket. The rm operation then takes time proportional to
the size of the data. Furthermore, the deleted files continue to incur storage costs.
To make sure that your deleted files are no longer incurring costs, you can do two things:
Use the the
-skipTrashoption when removing files:hadoop fs -rm -skipTrash s3a://bucket1/datasetUse the
expungecommand to purge any data that has been previously moved to the.Trashdirectory:hadoop fs -expunge -D fs.defaultFS=s3a://bucket1/As the
expungecommand only works with the default filesystem, you need to use the-Doption to make the target object store the default filesystem. This will change the default configuration.