Cloudera Docs

Configure S3 for system log using CDP CLI

To configure S3 using CDP CLI, follow the below procedure.

  • This operation requires CDP CLI version 0.9.84 or above.
  • This procedure is applicable under following conditions:
    • If you did not have Ozone when you created the base cluster.
    • Later you decide to use CDE on the existing base cluster and add Ozone to the existing base cluster.
  1. Create an S3 bucket using the following steps:
    1. Make use of the Cloudera Manager API call /clusters/{clusterName}/getOzoneS3GatewayInfo.
    2. The clusterName is the base cluster that has the Ozone service.
    3. You must provide a bucket name by following the guidelines mentioned in the Bucket naming rules documentation. You must note down the bucket name as this is different from the return bucket field mentioned below.
      The API call is in progress and you must wait for the API call to complete.
    4. After the API call is complete, save the returned json output. This looks similar to { “awsAccessKey”: … “awsSecret”: … “restUrl”: … “bucket”: … }
  2. Run the following cdpcli command using the values returned above:

    cdp environments --endpoint-url <console-url> --no-verify-tls --no-parameter-expansion --set-environment-setting --environment-name <environment-name> --settings ‘{“logType”: “OZONE”, “ozoneS3Key”: “<awsAccessKey>”, “ozoneS3Secret”: “<awsSecret>”, “ozoneS3RestUrl: “<restUrl>”, “ozoneLogsBucket”: “<bucket-name>”, “ozoneLogsPath”: “<bucket>”}’

  3. Verify that the environment settings have been correctly configured by running cdp environments

    --no-verify-tls get-environment-setting --environment-name <environment-name> --attrs logType ozoneS3Key ozoneS3Secret ozoneS3RestUrl ozoneLogsBucket ozoneLogsPath