Customizing Per-Bucket Secrets Held in Credential Files
Although most properties which are set per-bucket are automatically propagated from
their fs.s3a.bucket. custom entry to that of the base fs.s3a.
option, supporting secrets kept in Hadoop credential files is slightly more complex:
property values are kept in these files, and they cannot be dynamically patched.
Instead, callers need to create different configuration files for each bucket,
setting the base secrets, then declare the path to the appropriate credential file in a
bucket-specific version of the property
fs.s3a.security.credential.provider.path.
Example
Set base properties for
fs.s3a.secret.keyandfs.s3a.access.keyincore-site.xmlor in your job submission.Set similar properties per-bucket for a bucket called "frankfurt-1". These will override the base properties when talking to the bucket "frankfurt-1".
When setting properties in a JCEKS file, you must use the base property names — even if you only intend to use them for a specific bucket.
For example, in the JCEKS file called
hdfs://users/steve/frankfurt.jceks, set the base parameters fs.s3a.secret.key, fs.s3a.access.key to your "frankfurt-1" values from step 2.Next, set the path to the JCEKS file as a per-bucket option.
For example, fs.s3a.bucket.frankfurt-1.security.credential.provider.path should be set to
hdfs://users/steve/frankfurt.jceks.When the credentials for "frankfurt-1" are set up, the property
fs.s3a.bucket.frankfurt-1.security.credential.provider.pathwill be read, and the secrets from that file used to set the options to access the bucket.
Related Links