Limitations When Using DistCp with S3
When using DistCp with data in S3, consider the following limitations:
The
-append
option is not supported.The
-diff
option is not supported.The
-atomic
option causes a rename of the temporary data, so significantly increases the time to commit work at the end of the operation. Furthermore, as S3A does not offer atomic renames of directories, the-atomic operation doesn't actually deliver what is promised.
Avoid using this option.All
-p
options, including those to preserve permissions, user and group information, attributes checksums, and replication are ignored.CRC checking will not be performed, irrespective of the value of the
-skipCrc
flag.