Speeding up Job Commits by Increasing the Number of Threads
When an individual job writes many files to S3, the time to commit the job can increase.
It can be speeded up by increasing the value of
fs.s3a.committer.threads>. However, the value of
fs.s3a.connection.maximum must be at least this size otherwise the limit
on the number of parallel uploads is still limited.
<property> <name>fs.s3a.committer.threads</name> <value>8</value> </property> <property> <name>fs.s3a.connection.maximum</name> <value>15</value> </property>
