Cloudera Docs

Support matrix for CDP Public Cloud Replication Manager

You can use Replication Manager or other alternate replication methods to replicate HDFS, Hive external tables, and HBase data between on-premises clusters (CDH clusters, CDP Private Cloud Base clusters, HDP clusters) and CDP Public Cloud (Amazon S3 and Microsoft Azure ADLS Gen2 (ABFS)) clusters. Replication Manager from HDP clusters to CDP Public Cloud Azure is a beta feature and is not available for general use.

See the other sections in this topic for the supported cluster and runtime versions.

Replication Manager provides the following functionalities that you can use to accomplish your data replication goals:
  • HDFS replication policies replicate HDFS data and metadata from:
    • on-premises clusters (CDH, CDP Private Cloud Base, and HDP) to cloud storage.
    • cloud storage to classic clusters (CDH or CDP Private Cloud Base clusters).

    You can choose the frequency during policy creation to replicate the data.

  • Hive replication policies support table-level replication and can replicate Hive external tables from on-premises clusters (CDH and CDP Private Cloud Base) to cloud storage and to Data Hubs. They also can:
    • replicate data stored in Hive tables, Hive metadata, data in Hive metastore, and Impala metadata (catalog server metadata) associated with Impala tables registered in the Hive metastore, and
    • migrate Sentry permissions to Ranger.

      You can choose the frequency during policy creation to replicate the data.

  • HBase replication policies replicate HBase data from a source classic cluster (CDH or CDP Private Cloud Base cluster), COD, or Data Hub to a target Data Hub or COD cluster. You can also copy or replicate HBase data between different environments within a Virtual Private Cloud (VPC) using these policies.
    Table 1. Supported cluster and runtime versions for HBase replication policies
    Source Cluster Type Lowest Supported Source CDH/CDP Version Lowest Supported Source Cloudera Manager Version Target Cluster Type Lowest Supported Target CDP Version Lowest Supported Target Cloudera Manager Version
    CDP Private Cloud Base 7.1.6* 7.3.1 Data Hub 7.2.14 7.6.0
    CDH 6.3.3 7.3.1 Data Hub 7.2.14 7.6.0
    CDH 5.16.2 7.4.4 (patch-5017) COD (AWS) 7.2.14 -
    CDH 5.16.2
    • 7.6.1 (patch-5610)
    • 7.6.7 CHF1 and higher
    		 COD (Azure) 7.2.14 -
    COD (AWS/Azure) 7.2.14 - COD (AWS/Azure) 7.2.14 -
    COD (GCP) 7.2.16 - COD (GCP) 7.2.16
    *CDP Private Cloud Base 7.1.6 and higher clusters must be Kerberos enabled to use them as source classic clusters in an HBase replication policy.

    HBase replication policies replicate all the data from the specified tables and then continue to replicate the changed data automatically without user intervention.

Replicate data from CDP Private Cloud Base and CDP Public Cloud source clusters

Replication Manager replicates HDFS (CDP Private Cloud Base source clusters and CDP Public Cloud source clusters), Hive external tables (CDP Private Cloud Base source clusters), and HBase (CDP Private Cloud Base source clusters) data to CDP Public Cloud (Amazon S3 and Microsoft Azure ADLS Gen2 (ABFS)) clusters. You can use the replication plugin as an alternate replication method to replicate HBase data for scenarios that are not supported by Replication Manager.

The following tables list the minimum source and destination cluster versions, minimum Cloudera Manager versions, supported cloud providers, and supported scenarios:

Replicate data from CDP Private Cloud Base source clusters

Source cluster Lowest supported source Cloudera Manager version Lowest supported source Cloudera Runtime version Cloud provider Supported services on Replication Manager Services that require alternate replication methods
CDP Private Cloud Base 7.1.1 7.1.1 AWS

Azure
  • HDFS
  • Hive external tables
 HBase

To replicate HBase data, see COD replication in a Nutshell and HBase data replication.
CDP Private Cloud Base 7.9.0 7.1.1 Data Hub Hive external tables Not applicable
CDP Private Cloud Base 7.3.1 7.1.6 AWS

Azure

 HBase -

Replicate data from CDP Public Cloud source clusters

Consider the following limitations while using CDP Public Cloud source and CDP Public Cloud target clusters:
  • Replication across cross-cloud providers, that is from AWS to Azure and vice-versa is not supported.
  • The source and target clusters must use the same account.
Source cluster Destination cluster Supported services on Replication Manager Services that require alternate replication methods
CDP Public Cloud

(AWS* / Azure)

 CDH 5.x

CDH 6.x

HDP 2.x

HDP 3.x

 Not applicable HBase

To replicate HBase data, see COD replication in a Nutshell and HBase data replication.
CDP Public Cloud

(AWS*)

 CDH 5.9.0 and higher

CDP Private Cloud Base 7.1.7 SP1 and higher

 HDFS -
CDP Public Cloud

(Azure)

 CDH 6.1.0 and higher

CDP Private Cloud Base 7.1.7 SP1 and higher

 HDFS -
AWS AWS HBase -
Azure Azure HBase -

GCP

 GCP HBase -
*Replication Manager does not support S3 as a source or destination when S3 is configured to use SSE-KMS.

Replicate data from CDH and HDP source clusters

Replication Manager replicates HDFS data (CDH source clusters and HDP source clusters), Hive external tables (CDH source clusters), and HBase data (CDH 6 source clusters) to CDP Public Cloud (Amazon S3 and Microsoft Azure ADLS Gen2 (ABFS)) clusters. Replication Manager from HDP clusters to CDP Public Cloud Azure is a beta feature and is not available for general use. You can use alternate methods to replicate Hive external tables and HBase data for scenarios that are not supported by Replication Manager.

The following tables list the minimum CDH and HDP source cluster versions, minimum Cloudera Manager versions, supported cloud providers, and supported scenarios:

Table 2. Replicate data from CDH 5 source clusters
Source cluster Lowest supported source Cloudera Runtime version Lowest supported source Cloudera Manager version Cloud provider Supported services on Replication Manager Services that require alternate replication methods
CDH 5 5.10 6.3.0 AWS
  • HDFS
  • Sentry to Ranger*
  • Hive external table
 HBase

To replicate HBase data, see COD replication in a Nutshell, Migrating HBase data, and HBase data replication.

CDH 5 5.10 6.3.4 Azure
  • HDFS
  • Sentry to Ranger*
  • Hive external table
 HBase

To replicate HBase data, see COD replication in a Nutshell, Migrating HBase data, and HBase data replication.

CDH 5 5.10 7.9.0 Data Hub Hive external tables Not applicable
*To perform the Sentry policy replication, you must be running the Sentry service on CDH 5.12 or higher, or any CDH 6.x version.
Table 3. Replicate data from CDH 6 source clusters
Source cluster Lowest supported source Cloudera Runtime version Lowest supported source Cloudera Manager version Cloud provider Supported services on Replication Manager Services that require alternate replication methods
CDH 6 6.1 6.3.0 AWS
  • HDFS
  • Sentry to Ranger*
  • Hive external table
 HBase

To replicate HBase data, see COD replication in a Nutshell, Migrating HBase data, and HBase data replication.

CDH 6 6.1 7.1.1 / 6.3.4 Azure
  • HDFS
  • Sentry to Ranger*
  • Hive external table
 HBase

To replicate HBase data, see COD replication in a Nutshell, Migrating HBase data, HBase data replication.

CDH 6 6.3.3 7.3.1

AWS

Azure

 HBase -
*To perform the Sentry policy replication, you must be running the Sentry service on CDH 5.12 or higher, or any CDH 6.x version.
Table 4. Replicate data from HDP 2 and HDP 3 source clusters
Lowest supported source HDP version Cloud provider Supported services on Replication Manager Services that require alternate replication methods
HDP 2.6.5* AWS HDFS
HDP 2.6.5* Azure HDFS HBase

To replicate HBase data, see COD replication in a Nutshell and HBase data replication.
HDP 3.1.1*

AWS

Azure

 HDFS
*No alternate replication methods are available for HDFS, Ranger, and Atlas replication.