Working with Third-party S3-compatible Object Stores
The S3A Connector can work with third-party object stores; some vendors test the connector against their stores —and even actively collaborate in developing the connector in the open source community.
You need to configure the S3A connector in Cloudera Manager to be able to use it with
third-party objects stores.
- Log in to Cloudera Manager, and select your cluster.
- Select the Configuration tab.
- Search for Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml.
- Use the + button to add each of the following configuration parameter and provide its value.
- Click Save, and restart the service.
| Option | Change to |
|---|---|
fs.s3a.endpoint |
(the hostname of the S3 Store) |
fs.s3a.endpoint.region |
Specify the AWS region for the region. |
fs.s3a.path.style.access |
true |
fs.s3a.signing-algorithm |
If the default signing mechanism is rejected, another mechanism may work from: "QueryStringSignerType", "S3SignerType", "AWS3SignerType", "AWS4SignerType", "AWS4UnsignedPayloadSignerType" and "NoOpSignerType". |
fs.s3a.connection.ssl.enabled |
Set to "false" if HTTP is used instead of HTTPS |
fs.s3a.multiobjectdelete.enable |
Set to "false" if bulk delete operations are not supported. |
fs.s3a.list.version |
Set to "1" if the list directories with the default "2" option fails with an error. |
The following aspects need to be considered when using using S3 connector with third-party
object stores:
- Third-party object stores are generally consistent; there is no need for S3Guard. The S3A Committers will still offer better performance, and should be used for MapReduce and Spark.
- Except for versions of HBase specifically designed to work with S3 storage, HBase must
not use
s3a://paths for HBase storage. - S3 can not be used as a replacement for HDFS as the cluster filesystem in CDP. S3 can be used as a source and destination of work.
- Encryption may or may not be supported, refer to the documentation of the third-party object store.
- Security permissions are likely to be implemented differently from the IAM role mode —again, refer to the documentation of the third-party object store to see what is available.
