Cloudera Docs

Configuring S3Guard for Cluster Access to S3

  1. Specify the AWS credentials for the Amazon S3 instance where you want to enable S3Guard.
    • Add a new AWS credential.

      After adding the credential, the Edit S3Guard dialog box displays.

    • Use an existing AWS credential:
        1. Go to Administration > AWS Credentials.
        2. Locate the credential you want to use and click Actions > Edit S3Guard.

          The Edit S3Guard dialog box displays.

  2. Select Enable S3Guard.
  3. Edit the following S3Guard configuration properties:
    Table 1. S3Guard Configuration Properties
    Property Description
    Automatically Create S3Guard Metadata Table

    (fs.s3a.s3guard.ddb.table.create)

    API Name:

    s3guard_table_auto_create

    When Yes is selected, the DynamoDB table that stores the S3Guard metadata is automatically created if it does not exist.

    When No is selected and the table does not exist, running the Prune command, queries, or other jobs on S3 will fail.
    S3Guard Metadata Table Name

    (fs.s3a.s3guard.ddb.table)

    API Name: s3guard_table_name

    The name of the DynamoDB table that stores the S3Guard metadata.

    By default, the table is named s3guard-metadata.

    S3Guard Metadata Region Name

    (fs.s3a.s3guard.ddb.region)

    API Name: s3guard_region

    The DynamoDB region to connect to for access to the S3Guard metadata. Set this property to a valid region.
    Expand the Advanced section to configure the following properties:
    S3Guard Metadata Pruning Age

    (fs.s3a.s3guard.cli.prune.age)

    API Name: s3guard_cache_prune_age_ms

    Maximum age for S3Guard metadata. Whenever the Prune command runs, entries in the S3Guard metadata cache older than this age will be deleted.

    You can enter this value in milliseconds, seconds, minutes, hours, or days.

    S3Guard Metadata Table Read Capacity

    (fs.s3a.s3guard.ddb.table.capacity.read)

    API Name: s3guard_table_capacity_read

    Provisioned throughput requirements, in capacity units, for read operations from the DynamoDB table used for the S3Guard metadata. This value is only used when creating a new DynamoDB table. After the table is created, you can monitor the throughput and adjust the read capacity using the DynamoDB AWS Management Console.
    S3Guard Metadata Table Write Capacity

    (fs.s3a.s3guard.ddb.table.capacity.write)

    API Name: s3guard_table_capacity_write

    Provisioned throughput requirements, in capacity units, for write operations to the DynamoDB table used for the S3Guard metadata. This value is only used when creating a new DynamoDB table. After the table is created, you can monitor the throughput and adjust the write capacity as needed using the DynamoDB AWS Management Console.
  4. Click Save.

    The Connect to Amazon Web Services dialog box displays.

  5. To enable cluster access to S3 using the S3 Connector Service, click the Enable for Cluster Name link in the Cluster Access to S3 section.

    Follow the prompts to add the S3 Connector Service.