Working with Third-party S3-compatible Object Stores

The S3A Connector can work with third-party object stores; some vendors test the connector against their stores —and even actively collaborate in developing the connector in the open source community.

Option Change to
fs.s3a.endpoint (the hostname of the S3 Store)
fs.s3a.endpoint.region Specify the AWS region for the region.
fs.s3a.path.style.access true
fs.s3a.signing-algorithm If the default signing mechanism is rejected, another mechanism may work from: "QueryStringSignerType", "S3SignerType", "AWS3SignerType", "AWS4SignerType", "AWS4UnsignedPayloadSignerType" and "NoOpSignerType".
fs.s3a.connection.ssl.enabled Set to "false" if HTTP is used instead of HTTPS
fs.s3a.multiobjectdelete.enable Set to "false" if bulk delete operations are not supported.
fs.s3a.list.version Set to "1" if the list directories with the default "2" option fails with an error.
The following aspects need to be considered when using using S3 connector with third-party object stores:
  • Third-party object stores are generally consistent; there is no need for S3Guard. The S3A Committers will still offer better performance, and should be used for MapReduce and Spark.
  • Except for versions of HBase specifically designed to work with S3 storage, HBase must not use s3a:// paths for HBase storage.
  • S3 can not be used as a replacement for HDFS as the cluster filesystem in CDP. S3 can be used as a source and destination of work.
  • Encryption may or may not be supported, refer to the documentation of the third-party object store.
  • Security permissions are likely to be implemented differently from the IAM role mode —again, refer to the documentation of the third-party object store to see what is available.