Verifying that S3Guard is Enabled on a Bucket
When S3Guard is working, apart from some messages in the logs, there is no obvious clue that
it is enabled. To verify that a bucket does have S3Guard enabled, use the command line
command hadoop s3guard bucket-info
. This will print bucket information, and can
be used to explicitly check that a bucket has s3guard enabled
On a guarded bucket, it will list details about the bucket, the S3Guard Database on DynamoDB, and some client information.
hadoop s3guard bucket-info -guarded s3a://guarded-table/ Filesystem s3a://guarded-table Location: eu-west-1 Filesystem s3a://guarded-table is using S3Guard with store DynamoDBMetadataStore{region=eu-west-1, tableName=guarded-table} Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false Metadata Store Diagnostics: ARN=arn:aws:dynamodb:eu-west-1:980678866538:table/guarded-table description=S3Guard metadata store in DynamoDB name=guarded-table read-capacity=100 region=eu-west-1 retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS) size=61261 status=ACTIVE table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}], TableName: guarded-table, KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}], TableStatus: ACTIVE ,CreationDateTime: Sat Apr 28 22:14:22 BST 2018, ProvisionedThroughput: {LastDecreaseDateTime: Thu May 31 15:09:04 BST 2018,NumberOfDecreasesToday: 2,ReadCapacityUnits: 100,WriteCapacityUnits: 20}, TableSizeBytes: 61261,ItemCount: 419, TableArn: arn:aws:dynamodb:eu-west-1:980678866538:table/guarded-table, TableId: dc465257-8aaf-4a80-ad3e-c7fc708322fb,} write-capacity=20 The "magic" committer is supported S3A Client Endpoint: fs.s3a.endpoint=s3.amazonaws.com Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal
When invoked against an unguarded bucket the same command will fail.
hadoop s3guard bucket-info -guarded s3a://unguarded/ Filesystem s3a://unguarded Location: eu-west-1 Filesystem s3a://unguarded is not using S3Guard The "magic" committer is supported S3A Client Endpoint: fs.s3a.endpoint=s3-eu-west-1.amazonaws.com Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal 2018-05-31 16:37:16,691 [main] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 46: 46: S3Guard is not enabled for s3a://unguarded