Configuring the endpoint for an S3 bucket in the HDFS cluster-wide configuration
Configure the endpoint for an S3 bucket in the HDFS cluster-wide configuration to manage bucket-specific settings and regional endpoints.
- Go to Cloudera Manager > HDFS > Configuration.
-
Add the following configuration in the Cluster-wide Advanced
Configuration Snippet (Safety Valve) for core-site.xml advanced
configuration snippet:
Key: fs.s3a.bucket.<BUCKET-NAME>.endpoint Value: s3.<REGION-NAME>.amazonaws.comThe configuration setting described above is used to specify a bucket-specific endpoint for Amazon S3. This configuration is also used in case where multiple buckets are managed in different regions. There are also global configuration settings that control the default behavior for all S3 buckets:fs.s3a.endpoint: Sets the default endpoint for all S3 buckets.fs.s3a.endpoint.region: Specifies the region used for signing when combined withfs.s3a.endpoint, and all S3 buckets point to the same region.
Example values for these configurations:f3.s3a.endpoint: s3.amazonaws.comf3.s3a.endpoint.region: us-east-1/us-east-2/us-west-1/us-west-2/eu-west-1/..For more information on the endpoints and regions supported, refer to Amazon Simple Storage Service endpoints and quotas.
For detailed information on configuration behavior and how to connect to an Amazon S3 bucket using the S3A Connector, please refer to Connecting to an Amazon S3 Bucket through the S3A Connector.
