Working with Third-party S3-compatible Object Stores

The S3A Connector can work with third-party object stores; some vendors test the connector against their stores —and even actively collaborate in developing the connector in the open source community.

Option Change to
fs.s3a.endpoint (the hostname of the S3 Store)
fs.s3a.endpoint.region Specify the AWS region for the region. true
fs.s3a.signing-algorithm If the default signing mechanism is rejected, another mechanism may work from: "QueryStringSignerType", "S3SignerType", "AWS3SignerType", "AWS4SignerType", "AWS4UnsignedPayloadSignerType" and "NoOpSignerType".
fs.s3a.connection.ssl.enabled Set to "false" if HTTP is used instead of HTTPS
fs.s3a.multiobjectdelete.enable Set to "false" if bulk delete operations are not supported.
fs.s3a.list.version Set to "1" if the list directories with the default "2" option fails with an error.

Third-party object stores are generally consistent; there is no need for S3Guard. The S3A Committers will still offer better performance, and should be used for MapReduce and Spark.

Encryption may or may not be supported: consult the documentation.

Security permissions are likely to be implemented differently from the IAM role mode —again, consult the documentation to see what is available.