Properties for overriding HA configuration

You must set certain configuration properties in the hdfs-site.xml file of both the source and target clusters for High Availability (HA) support during data replication using Hive. You must add these properties in the Override field so that the replication job completes successfully.

In the following table, the source nameservice is referred to as ns1 and target nameservice is referred to as ns2. You must update these example values based on source and target details. You can use Cloudera Manager APIs to automate the creation of Hive replication policy, where the APIs pass the following policy options as part of the REST API call.

Table 1. Source cluster
Key Value
dfs.client.failover.proxy.provider.ns1 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.automatic-failover.enabled.ns1 true
dfs.ha.namenodes.ns1 namenode1546332965,namenode1546335166
dfs.namenode.http-address.ns1.namenode1546332965 quasar-syrawe-1.quasar-syrawe.root.hwx.site:20101
dfs.namenode.http-address.ns1.namenode1546335166 quasar-syrawe-2.quasar-syrawe.root.hwx.site:20101
dfs.namenode.https-address.ns1.namenode1546332965 quasar-syrawe-1.quasar-syrawe.root.hwx.site:20102
dfs.namenode.https-address.ns1.namenode1546335166 quasar-syrawe-2.quasar-syrawe.root.hwx.site:20102
dfs.namenode.rpc-address.ns1.namenode1546332965 quasar-syrawe-1.quasar-syrawe.root.hwx.site:8020
dfs.namenode.rpc-address.ns1.namenode1546335166 quasar-syrawe-2.quasar-syrawe.root.hwx.site:8020
dfs.namenode.servicerpc-address.ns1.namenode1546332965 quasar-syrawe-1.quasar-syrawe.root.hwx.site:8022
dfs.namenode.servicerpc-address.ns1.namenode1546335166 quasar-syrawe-2.quasar-syrawe.root.hwx.site:8022
dfs.namenode.kerberos.principal hdfs/_HOST@AD.HALXG.CLOUDERA.COM,hdfs/_HOST@AD.HALXG.CLOUDERA.COM
dfs.nameservices ns1, ns2
ha.zookeeper.quorum quasar-syrawe-2.quasar-syrawe.root.hwx.site:2181,quasar-syrawe-1.quasar-syrawe.root.hwx.site:2181,quasar-syrawe-3.quasar-syrawe.root.hwx.site:2181,quasar-rlltjy-1.quasar-rlltjy.root.hwx.site:2181,quasar-rlltjy-5.quasar-rlltjy.root.hwx.site:2181,quasar-rlltjy-2.quasar-rlltjy.root.hwx.site:2181
Table 2. Target cluster
Key Value
dfs.client.failover.proxy.provider.ns2 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.automatic-failover.enabled.ns2 true
dfs.ha.namenodes.ns2 namenode1546335216,namenode1546332943
dfs.namenode.http-address.ns2.namenode1546332943 quasar-rlltjy-5.quasar-rlltjy.root.hwx.site:20101
dfs.namenode.http-address.ns2.namenode1546335216 quasar-rlltjy-1.quasar-rlltjy.root.hwx.site:20101
dfs.namenode.https-address.ns2.namenode1546332943 quasar-rlltjy-5.quasar-rlltjy.root.hwx.site:20102
dfs.namenode.https-address.ns2.namenode1546335216 quasar-rlltjy-1.quasar-rlltjy.root.hwx.site:20102
dfs.namenode.rpc-address.ns2.namenode1546332943 quasar-rlltjy-5.quasar-rlltjy.root.hwx.site:8020
dfs.namenode.rpc-address.ns2.namenode1546335216 quasar-rlltjy-1.quasar-rlltjy.root.hwx.site:8020
dfs.namenode.servicerpc-address.ns2.namenode1546332943 quasar-rlltjy-5.quasar-rlltjy.root.hwx.site:8022
dfs.namenode.servicerpc-address.ns2.namenode1546335216 quasar-rlltjy-1.quasar-rlltjy.root.hwx.site:8022
dfs.namenode.kerberos.principal hdfs/_HOST@AD.HALXG.CLOUDERA.COM,hdfs/_HOST@AD.HALXG.CLOUDERA.COM
dfs.nameservices ns1, ns2
ha.zookeeper.quorum quasar-syrawe-2.quasar-syrawe.root.hwx.site:2181,quasar-syrawe-1.quasar-syrawe.root.hwx.site:2181,quasar-syrawe-3.quasar-syrawe.root.hwx.site:2181,quasar-rlltjy-1.quasar-rlltjy.root.hwx.site:2181,quasar-rlltjy-5.quasar-rlltjy.root.hwx.site:2181,quasar-rlltjy-2.quasar-rlltjy.root.hwx.site:2181