Accessing Cloud Data
Also available as:
PDF
loading table of contents...

Verifying that S3Guard is Enabled on a Bucket

When S3Guard is working, apart from some messages in the logs, there is no obvious clue that it is enabled. To verify that a bucket does have S3Guard enabled, use the command line command hadoop s3guard bucket-info. This will print bucket information, and can be used to explicitly check that a bucket has s3guard enabled

On a guarded bucket, it will list details about the bucket, the S3Guard Database on DynamoDB, and some client information.

hadoop s3guard bucket-info -guarded s3a://guarded-table/

Filesystem s3a://guarded-table
Location: eu-west-1
Filesystem s3a://guarded-table is using S3Guard with store DynamoDBMetadataStore{region=eu-west-1, tableName=guarded-table}
Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false
Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:eu-west-1:980678866538:table/guarded-table
description=S3Guard metadata store in DynamoDB
name=guarded-table
read-capacity=100
region=eu-west-1
retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS)
size=61261
status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
  {AttributeName: parent,AttributeType: S}],
  TableName: guarded-table,
  KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],
  TableStatus: ACTIVE
  ,CreationDateTime: Sat Apr 28 22:14:22 BST 2018,
  ProvisionedThroughput: {LastDecreaseDateTime: Thu May 31 15:09:04 BST 2018,NumberOfDecreasesToday: 2,ReadCapacityUnits: 100,WriteCapacityUnits: 20},
  TableSizeBytes: 61261,ItemCount: 419,
  TableArn: arn:aws:dynamodb:eu-west-1:980678866538:table/guarded-table,
  TableId: dc465257-8aaf-4a80-ad3e-c7fc708322fb,}
write-capacity=20
The "magic" committer is supported

S3A Client
Endpoint: fs.s3a.endpoint=s3.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal

When invoked against an unguarded bucket the same command will fail.

hadoop s3guard bucket-info -guarded s3a://unguarded/

Filesystem s3a://unguarded
Location: eu-west-1
Filesystem s3a://unguarded is not using S3Guard
The "magic" committer is supported

S3A Client
Endpoint: fs.s3a.endpoint=s3-eu-west-1.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal
2018-05-31 16:37:16,691 [main] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 46: 46: S3Guard is not enabled for s3a://unguarded