Using Fast Upload with Amazon S3
data to Amazon S3 is subject to limitations of the s3a
OutputStream implementation, which buffers the entire file to disk before
uploading it to S3. This can cause the upload to proceed very slowly and can require a large
amount of temporary disk space on local disks.
You can configure a cluster to use the Fast Upload feature. This feature implements several performance improvements and has tunable parameters for buffering to disk (the default) or to memory, tuning the number of threads, and for specifying the disk directories used for buffering.