Configuring the endpoint for an S3 bucket in the HDFS cluster-wide configuration

Configure the endpoint for an S3 bucket in the HDFS cluster-wide configuration to manage bucket-specific settings and regional endpoints.

  1. Go to Cloudera Manager > HDFS > Configuration.
  2. Add the following configuration in the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml advanced configuration snippet:
    Key: fs.s3a.bucket.<BUCKET-NAME>.endpoint 
    Value: s3.<REGION-NAME>.amazonaws.com
    The configuration setting described above is used to specify a bucket-specific endpoint for Amazon S3. This configuration is also used in case where multiple buckets are managed in different regions. There are also global configuration settings that control the default behavior for all S3 buckets:
    • fs.s3a.endpoint: Sets the default endpoint for all S3 buckets.
    • fs.s3a.endpoint.region: Specifies the region used for signing when combined with fs.s3a.endpoint, and all S3 buckets point to the same region.
    Example values for these configurations:

    For detailed information on configuration behavior and how to connect to an Amazon S3 bucket using the S3A Connector, please refer to Connecting to an Amazon S3 Bucket through the S3A Connector.