Accessing multiple buckets managed by the same cloud account credential
You can also access multiple buckets present in your Amazon S3 or Amazon S3-compatible object stores.
-
Assign at least the following set of AWS permissions to an AWS IAM role used by
IDBroker for Ranger RAZ support in a multiple buckets scenario:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AccessToBucketObjects", "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectVersion", "s3:GetObjectVersionAcl", "s3:PutObject", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::${BUCKET-01}/*", "arn:aws:s3:::${BUCKET-02}/*", "arn:aws:s3:::${BUCKET-03}/*" ] }, { "Sid": "AccessToBucket", "Effect": "Allow", "Action": [ "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:GetBucketVersioning", "s3:GetEncryptionConfiguration", "s3:ListBucket", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::${BUCKET-01}", "arn:aws:s3:::${BUCKET-02}", "arn:aws:s3:::${BUCKET-03}", ] } ] } -
In Cloudera Manager > HDFS > Configuration, add the following S3 endpoint and region configurations to the
Cluster-wide Advanced Configuration Snippet (Safety Valve) for
core-site.xml advanced configuration snippet for the buckets you
defined in your AWS IAM policy:
Key : fs.s3a.bucket.<BUCKET-01-NAME>.endpoint Value : s3.<BUCKET-01-REGION-NAME>.amazonaws.com Key : fs.s3a.bucket.<BUCKET-02-NAME>.endpoint Value : s3.<BUCKET-02-REGION-NAME>.amazonaws.com
