Working with Third-party S3-compatible Object Stores
The S3A Connector can work with third-party object stores; some vendors test the connector against their stores —and even actively collaborate in developing the connector in the open source community.
| Option | Change to |
|---|---|
fs.s3a.endpoint |
(the hostname of the S3 Store) |
fs.s3a.endpoint.region |
Specify the AWS region for the region. |
fs.s3a.path.style.access |
true |
fs.s3a.signing-algorithm |
If the default signing mechanism is rejected, another mechanism may work from: "QueryStringSignerType", "S3SignerType", "AWS3SignerType", "AWS4SignerType", "AWS4UnsignedPayloadSignerType" and "NoOpSignerType". |
fs.s3a.connection.ssl.enabled |
Set to "false" if HTTP is used instead of HTTPS |
fs.s3a.multiobjectdelete.enable |
Set to "false" if bulk delete operations are not supported. |
fs.s3a.list.version |
Set to "1" if the list directories with the default "2" option fails with an error. |
Third-party object stores are generally consistent; there is no need for S3Guard. The S3A Committers will still offer better performance, and should be used for MapReduce and Spark.
Encryption may or may not be supported: consult the documentation.
Security permissions are likely to be implemented differently from the IAM role mode —again, consult the documentation to see what is available.
