Accessing multiple buckets managed by the same cloud account credential

You can also access multiple buckets present in your Amazon S3 or Amazon S3-compatible object stores.

  1. Assign at least the following set of AWS permissions to an AWS IAM role used by IDBroker for Ranger RAZ support in a multiple buckets scenario:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AccessToBucketObjects",
                "Effect": "Allow",
                "Action": [
                    "s3:AbortMultipartUpload",
                    "s3:DeleteObject",
                    "s3:DeleteObjectVersion",
                    "s3:GetObject",
                    "s3:GetObjectAcl",
                    "s3:GetObjectVersion",
                    "s3:GetObjectVersionAcl",
                    "s3:PutObject",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET-01}/*",
                    "arn:aws:s3:::${BUCKET-02}/*",
                    "arn:aws:s3:::${BUCKET-03}/*"
                ]
            },
            {
                "Sid": "AccessToBucket",
                "Effect": "Allow",
                "Action": [
                    "s3:GetBucketAcl",
                    "s3:GetBucketLocation",
                    "s3:GetBucketVersioning",
                    "s3:GetEncryptionConfiguration",
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET-01}",
                    "arn:aws:s3:::${BUCKET-02}",
                    "arn:aws:s3:::${BUCKET-03}",
                ]
            }
        ]
    }
  2. In Cloudera Manager > HDFS > Configuration, add the following S3 endpoint and region configurations to the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml advanced configuration snippet for the buckets you defined in your AWS IAM policy:
    Key : fs.s3a.bucket.<BUCKET-01-NAME>.endpoint
    Value : s3.<BUCKET-01-REGION-NAME>.amazonaws.com
    
    Key : fs.s3a.bucket.<BUCKET-02-NAME>.endpoint
    Value : s3.<BUCKET-02-REGION-NAME>.amazonaws.com