Cloudera Replication Manager

• Cloudera on cloud Release Summaries
• Cloudera on cloud Preview Features

Cloudera Replication Manager is a service for copying and migrating data between environments within the enterprise data cloud. It is a simple, easy-to-use, and feature-rich data movement capability to move existing data and metadata to the cloud to fuel new workloads.

Use the following links to access the service, manage replication policies, and troubleshoot operations.

Get Started with Replication Manager

• Access Replication Manager in Cloudera on cloud - You can access the Replication Manager service by logging into Cloudera Data Platform.

  When you log into Cloudera Data Platform, the Cloudera on cloud web interface appears. Click Replication Manager to view the Overview page of the Replication Manager.

  Replication Manager has the following pages:

  • Overview
  • Classic Clusters
  • Cloud Credentials
  • Replication Policies

• Explore Replication Policies - Use Replication Manager to replicate HDFS, Hive external tables, HBase data, Ranger policies and roles for HDFS, Hive, and HBase services, and Iceberg tables from Cloudera Base on premises clusters to Cloudera on cloud clusters in Amazon S3 (AWS), Microsoft Azure ADLS Gen2 (ABFS), and Google Cloud Platform (GCP).

Review Replication Policies

You can create replication policies in Replication Manager to copy and migrate data from CDH (version 5.13 and higher) clusters (HDFS, Hive, and HBase data) and Cloudera Base on premises (version 7.1.4 and higher) clusters (HDFS, Hive external tables, and HBase data) to Cloudera on cloud clusters. You can also replicate HDFS data from cloud storage to classic clusters (CDH or Cloudera Base on premises clusters), and Hive external tables to Data Hubs. The supported Cloudera on cloud services include Amazon S3 and Microsoft Azure ADLS Gen2 (ABFS).

• Review HDFS Policies - You can use the HDFS replication policies in Cloudera Replication Manager to replicate HDFS data. The HDFS replication policies can replicate HDFS data and metadata from Classic Clusters (CDH, Cloudera Base on premises, and HDP) to Cloudera on cloud storage buckets such as S3 and ABFS, and from cloud storage to Classic Clusters (CDH or Cloudera Base on premises clusters). To use an on-premises cluster (CDH or Cloudera Base on premises) in the replication policy, you must register it as a classic cluster in the Management Console. To use the cloud storage for data replication, you must register the cloud credentials in Replication Manager so that the Replication Manager service can access the cloud storage. You must also verify cluster access and configure minimum ports for replication before you create HDFS replication policies.

• Review Hive Policies - To create a Hive replication policy in Cloudera Replication Manager, you must configure the required Ranger policy in Ranger, register the on-premises cluster (CDH or Cloudera Base on premises) as a classic cluster in Cloudera Management Console, register cloud account credentials in the Replication Manager service, verify cluster access, and configure minimum ports for replication. The replication load happens on the source on-premises cluster. You can replicate data on-premises to the cloud with a single cluster if the Metastore is running on the cloud.

• Review HBase Policies - To create an HBase replication policy in Cloudera Replication Manager, you must register the on-premises cluster (CDH or Cloudera Base on premises) as a classic cluster in Cloudera Management Console, register cloud account credentials in the Replication Manager service, verify cluster access, and configure minimum ports for replication.

• Review Iceberg Policies - The Iceberg replication policies in Cloudera Replication Manager can replicate Iceberg tables between Data Lakes through Data Hubs in Cloudera on cloud 7.3.2 or higher versions using AWS. The Data Lakes can be located in a single AWS region or across multiple regions.

  Before you create an Iceberg replication policy, you must deploy the source Iceberg Replication Data Hub in the source Data Lake and the target Iceberg Replication Data Hub in the target Data Lake. The deployed Data Hubs provide the Hive database the details about the table metadata, source location of the tables, and optionally compute resources for the replication process. The replication occurs between S3-backed Data Lakes using HDFS protocols (or DistCp).

• Review Ranger Policies - You can create Ranger replication policies in Cloudera Replication Manager to migrate the Ranger policies and roles for HDFS, Hive, and HBase services. You can migrate these Ranger policies from Kerberos-enabled Cloudera Base on premises 7.3.2 or higher clusters using Cloudera Manager 7.13.2 to Cloudera on cloud 7.3.2 clusters. The Ranger replication policies can also migrate the Ranger audit logs in HDFS.

Create Replication Policies

You can configure new replication policies to schedule automated data movement from CDH or Cloudera Base on premises to Cloudera on cloud:

• Create HDFS Replication Policies - After you register the Classic Clusters in Management Console and register the cloud credentials in Cloudera Replication Manager, you can create an HDFS replication policy to replicate HDFS data. An HDFS replication policy can replicate HDFS data from a classic cluster (CDH, Cloudera Base on premises, and HDP clusters) to cloud storage, and from cloud storage to Classic Clusters (CDH or Cloudera Base on premises clusters). You can replicate data on-premises to cloud storage account with a single cluster.

• Create Hive Replication Policies - Before you create a Hive replication policy in Cloudera Replication Manager, you must ensure that the prerequisites are complete and the required Ranger policy is set in Ranger.

• Create HBase Replication Policies - You can replicate HBase data from a source classic cluster (CDH or Cloudera Base on premises cluster), Cloudera Operational Database, or Cloudera Data Hub to a target Cloudera Data Hub or Cloudera Operational Database cluster in Cloudera Replication Manager.

• Create Iceberg Replication Policies - You can create replication policies to copy Apache Iceberg tables, data files, snapshot manifests, and metadata to cloud target locations.

• Create Ranger Replication Policies - Create Ranger replication policies in Cloudera Replication Manager to migrate the Ranger policies, roles for HDFS, Hive, and HBase services, and audit logs for HDFS.

Create Replication Policies using the CDP CLI

You can use CDP CLI commands to create and manage HDFS, Hive, and HBase replication policies in Cloudera Replication Manager. You can also register the ABFS and AWS cloud credentials to use in Replication Manager. The CDP CLI commands for Replication Manager are under the “replicationmanager” CDP CLI option.

Troubleshoot Replication Manager

Diagnose and resolve common replication failures, review transfer logs, and verify service connectivity in Cloudera Replication Manager.