Accessing Cloud Data
Also available as:
loading table of contents...

Customizing Per-Bucket Secrets Held in Credential Files

Although most properties which are set per-bucket are automatically propagated from their fs.s3a.bucket. custom entry to that of the base fs.s3a. option, supporting secrets kept in Hadoop credential files is slightly more complex: property values are kept in these files, and they cannot be dynamically patched.

Instead, callers need to create different configuration files for each bucket, setting the base secrets, then declare the path to the appropriate credential file in a bucket-specific version of the property


  1. Set base properties for fs.s3a.secret.key and fs.s3a.access.key in core-site.xml or in your job submission.

  2. Set similar properties per-bucket for a bucket called "frankfurt-1". These will override the base properties when talking to the bucket "frankfurt-1".

  3. When setting properties in a JCEKS file, you must use the base property names — even if you only intend to use them for a specific bucket.

    For example, in the JCEKS file called hdfs://users/steve/frankfurt.jceks, set the base parameters fs.s3a.secret.key, fs.s3a.access.key to your "frankfurt-1" values from step 2.

  4. Next, set the path to the JCEKS file as a per-bucket option.

    For example, should be set to hdfs://users/steve/frankfurt.jceks.

  5. When the credentials for "frankfurt-1" are set up, the property will be read, and the secrets from that file used to set the options to access the bucket.

Related Links

Using Per-Bucket Credentials to Authenticate

Credential Provider API