Hive/Impala replication
Hive/Impala replication enables you to copy (replicate) your Hive metastore and data from one cluster to another and synchronize the Hive metastore and data set on the destination cluster with the source, based on a specified replication policy.
This page contains references to CDH 5 components or features that have been removed from CDH 6. These references are only applicable if you are managing a CDH 5 cluster with Cloudera Manager 6.
Minimum Required Role: Replication Administrator (also provided by Full Administrator)
The destination cluster must be managed by the Cloudera Manager Server where the replication is being set up, and the source cluster can be managed by that same server or by a peer Cloudera Manager Server.
- When you replicate from a CDH cluster to a CDP Private Cloud Base cluster, all tables become External tables during Hive replication. This is because the default table type is ACID in Hive3, which is the only managed table type. As of this release, Replication Manager does not support Hive2 -> Hive3 replication into ACID tables and all the tables will necessarily be replicated as External tables.
- Replicated tables will be created under external Hive warehouse directory set
by
hive.metastore.warehouse.external.dir
Hive configuration parameter. Users have to make sure that this has a different value thanhive.metastore.warehouse.dir
Hive configuration parameter, that is the location of Managed tables. - If users want to replicate the same database from Hive2 to Hive3 (that will have different paths by design), they need to use Force Overwrite option per policy to avoid any mismatch issues.
- If the
hadoop.proxyuser.hive.groups
configuration has been changed to restrict access to the Hive Metastore Server to certain users or groups, thehdfs
group or a group containing thehdfs
user must also be included in the list of groups specified for Hive/Impala replication to work. This configuration can be specified either on the Hive service as an override, or in the core-site HDFS configuration. This applies to configuration settings on both the source and destination clusters. - If you configured on the target cluster for the directory where HDFS data is copied during Hive/Impala replication, the permissions that were copied during replication, are overwritten by the HDFS ACL synchronization and are not preserved
- Replicate to and from Amazon S3 from CDH 5.14+ and Cloudera Manager version
5.13+.
Replication Manager does not support S3 as a source or destination when S3 is configured to use SSE-KMS.
- Replicate to and from Microsoft ADLS Gen1 from CDH 5.13+ and Cloudera Manager 5.15, 5.16, 6.1+.
- Replicate to Microsoft ADLS Gen2 (ABFS) from CDH 5.13+ and Cloudera Manager 6.1+.