S3 Data Extraction for Navigator

To enable bulk and incremental extraction, you create the following policy document by copying the policy text and pasting it in the policy editor, and then attaching it to the Navigator user in AWS.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1481678612000",
            "Effect": "Allow",
            "Action": [
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SetQueueAttributes"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1481678744000",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

To set up bulk extraction only, add the following to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties in Cloudera Manager, and then restart the metadata server:

nav.s3.extractor.incremental.enable=false

To enable bulk extraction only, create the following policy document by copying the policy text and pasting it in the policy editor, and then attaching it to the Navigator user in AWS.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1481676614000",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

To set up event notification for an external queue, you must create a queue from which Navigator can extract, and then configure S3 event notifications to send to Navigator.

If you create that queue instead of having Navigator set it up for you, do the following:

1. Stop Cloudera Navigator.
2. Open the Amazon SQS console and create a queue with the following settings:
   • Default Visibility Timeout: 10 minutes
   • Message Retention Period: 14 days
   • Delivery Delay: 0 seconds
   • Receive Message Wait Time: 0 seconds
3. Select the queue you created, click the Permissions tab, click Add a Permission, and configure the following in the Add a Permision to... dialog box:
   • Effect: Allow
   • Principal: Everybody
   • Actions: SendMessage
     In the Conditions (optional) area, set the following values:

     • Qualifier: None
     • Condition: ArnLike
     • Key: aws:SourceArn
     • Value: arn:aws:s3::*:*

     When finished, click Add Condition, and then click Add Condition.

4. Set up a queue in each region in which you have buckets.
5. Configure event notification for every bucket:
   • Name: nav-send-metadata-on-change
   • Events: ObjectCreated(All) and ObjectRemoved(All)
   • Send to: SQS queue
   • SQS queue: The name of your queue
6. Configure SNS fanout if you have existing S3 event notification. For more information about SNS fanout, see Common SNS Scenarios.
7. In Cloudera Manager, add the following to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties in Cloudera Manager:

   nav.s3.extractor.incremental.enable=true
   nav.s3.extractor.incremental.auto_setup.enable=false
   nav.s3.extractor.incremental.queues=queue_json

   queue_json must have the following JSON format, with no spaces:

   [{"region":"us-west-1"\\,"queueUrl":"https://sqs.us-west-1.amazonaws.com/account_num/queue_name"}\\,{queue_2}\\,...{queue_n}]

8. Restart Navigator.

To enable event notification for an external queue, create the following policy document by copying the policy text and pasting it in the policy editor, and then attaching it to the Navigator user in AWS.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1481678612000",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:GetQueueAttributes",
                "sqs:ReceiveMessage"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1481678744000",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}