S3 Performance Checklist
There are various types of checklists that you can use to ensure optimal performance when working with data in S3.
Checklist for Data
-
[ ]
Amazon S3 bucket is in same region as the EC2-hosted cluster. -
[ ]
The directory layout is "shallow". For directory listing performance, the directory layout prefers "shallow" directory trees with many files over deep directory trees with only a few files per directory. -
[ ]
The "pseudo" block size set infs.s3a.block.size
is appropriate for the work to be performed on the data. -
[ ]
Copy to HDFS any data that needs to be repeatedly read to HDFS.
Checklist for Cluster Configurations
-
[ ]
Setyarn.scheduler.capacity.node-locality-delay
to 0 to improve container launch times.
Checklist for Code
-
[ ]
Application does not makerename()
calls. Where it does, it does not assume the operation is immediate. -
[ ]
Application does not assume thatdelete()
is near-instantaneous. -
[ ]
Application usesFileSystem.listFiles(path, recursive=true)
to list a directory tree. -
[ ]
Application prefers forward seeks through files, rather than full random IO. -
[ ]
If making "random" IO throughseek()
andread()
sequences or and Hadoop'sPositionedReadable
API,fs.s3a.experimental.input.fadvise
is set torandom
.