Cloudera Docs

Authentication methods to use AWS credentials in replication policies

You can choose long-term AWS cloud credentials or temporary AWS session credentials when you want to replicate HDFS data, Hive external tables, and HBase data from Cloudera Private Cloud Base clusters to S3 buckets on Cloudera Public Cloud.

Long-term cloud credentials

You can use long-term credentials to replicate data to the cloud using replication policies. To use long-term cloud credentials in a replication policy, you must:
  • have an AWS account, and access key and secret key for it.
  • register an external account in Cloudera Manager using AWS access key and AWS secret key.

    You can add an external account on the Cloudera Manager > Administration > External Accounts page. The external account serves as an authentication method during data replication, using replication policies, from Cloudera Private Cloud Base clusters to cloud.

  • add the cloud credential in Cloudera Replication Manager.
The following use cases illustrate scenarios where you can use long-term AWS credentials:
  • Environments where you have multiple users and multi-tenancy – In this instance, you can add an Add Access Key Credentials external account in Cloudera Manager for Cloudera Private Cloud Base cluster, add the cloud credentials in the Cloudera Replication Manager, and then create a replication policy.
  • Single user cluster, or where all the users of the cluster have the same privileges to the data in Amazon S3 – In this instance, you can add IAM role-based authentication in Cloudera Manager for Cloudera Private Cloud Base cluster, add the cloud credentials in the Cloudera Replication Manager, and then create a replication policy.

Temporary AWS session credentials

You can use temporary AWS session credentials to provide just-in-time, minimum required access to replicate data using replication policies. Before you use temporary AWS session credentials in a replication policy, you must:
  1. have an AWS account with an IAM role that has the required permissions to access the target S3 bucket and has the necessary trust relationships set up.
  2. install and configure IDBroker on the Cloudera Private Cloud Base cluster.
  3. add the cloud credential in Cloudera Replication Manager.

    Alternatively, you can add an external account for the IDBroker topology in Cloudera Manager.