Enabling the Directory Committer in Hadoop
For backwards compatibility, output to S3 defaults to using the original file committer.
To switch to the Directory Committer in Ambari, edit core-site.xml
and set the property fs.s3a.committer.name
to directory
.
<property> <name>fs.s3a.committer.name</name> <value>directory</value> </property>
The other values for this option are file
, partitioned
and
magic
; file
is the original file output committer, which uses
file and directory renames to commit output. The partitioned
committer is a
variant of the directory committer, with different conflict resolution policies. The
magic
committer is a potentially faster committer, but which should be
considered less tested.