Speeding up Job Commits by Increasing the Number of Threads

When an individual job writes many files to S3, the time to commit the job can increase.

It can be speeded up by increasing the value of fs.s3a.committer.threads>. However, the value of fs.s3a.connection.maximum must be at least this size otherwise the limit on the number of parallel uploads is still limited.

<property>
  <name>fs.s3a.committer.threads</name>
  <value>8</value>
</property>
<property>
  <name>fs.s3a.connection.maximum</name>
  <value>15</value>
</property>