Accessing Cloud Data
Also available as:
PDF
loading table of contents...

Enabling the Directory Committer in Hadoop

For backwards compatibility, output to S3 defaults to using the original file committer.

To switch to the Directory Committer in Ambari, edit core-site.xml and set the property fs.s3a.committer.name to directory.

<property>
  <name>fs.s3a.committer.name</name>
  <value>directory</value>
</property>

The other values for this option are file, partitioned and magic; file is the original file output committer, which uses file and directory renames to commit output. The partitioned committer is a variant of the directory committer, with different conflict resolution policies. The magic committer is a potentially faster committer, but which should be considered less tested.