S3 Performance Checklist

There are various types of checklists that you can use to ensure optimal performance when working with data in S3.

Checklist for Data

  • [ ] Amazon S3 bucket is in same region as the EC2-hosted cluster.

  • [ ] The directory layout is "shallow". For directory listing performance, the directory layout prefers "shallow" directory trees with many files over deep directory trees with only a few files per directory.

  • [ ] The "pseudo" block size set in fs.s3a.block.size is appropriate for the work to be performed on the data.

  • [ ] Copy to HDFS any data that needs to be repeatedly read to HDFS.

Checklist for Cluster Configurations

  • [ ] Set yarn.scheduler.capacity.node-locality-delay to 0 to improve container launch times.

Checklist for Code

  • [ ] Application does not make rename() calls. Where it does, it does not assume the operation is immediate.

  • [ ] Application does not assume that delete() is near-instantaneous.

  • [ ] Application uses FileSystem.listFiles(path, recursive=true) to list a directory tree.

  • [ ] Application prefers forward seeks through files, rather than full random IO.

  • [ ] If making "random" IO through seek() and read() sequences or and Hadoop's PositionedReadable API, fs.s3a.experimental.input.fadvise is set to random.

More