Before creating the required entity definitions in Falcon, you must modify several configuration files. These files can be modified from Ambari or from the command line. To access the service properties in Ambari, click the Services menu and select the service you want, then click the Configs tab. For some services, you must also click the Advanced tab.

​auth-to-local Setting

For Kerberos principals to be mapped correctly across clusters, changes must be made to the hadoop.security.auth_to_local setting in the core-site.xml file on both clusters. These entries should be made for the ambariqa, hbase, and hdfs principals.

For example, to add the principal mapping for CLUSTER02 into CLUSTER01 add the following lines into the hadoop.security.auth_to_local field in CLUSTER01. Note, these lines must be inserted before the wildcard line RULE:[1:$1@$0](.*@HADOOP)s/@.*//:

RULE:[1:$1@$0](ambari-qa-CLUSTER02@HADOOP)s/.*/ambari-qa/
RULE:[1:$1@$0](hbase-CLUSTER02@HADOOP)s/.*/hbase/
RULE:[1:$1@$0](hdfs-CLUSTER02@HADOOP)s/.*/hdfs/

	Important
	Ensure the hadoop.security.auth_to_local setting in the HDFS core-site.xml file is consistent across all clusters. Inconsistencies in rules for hadoop.security.auth_to_local can lead to issues with delegation token renewals.

To allow the Falcon Oozie workflows to query the primary status of both clusters, the Oozie server hosts must be whitelisted as proxy hosts to allow cross-site Oozie access. On both clusters, the hadoop.proxyuser.oozie.hosts setting in the HDFS core-site.xml must contain the hostnames of all the Oozie servers in both clusters. The hostnames of all the Oozie servers must also be added to the hadoop.proxyuser.hive.hosts because the Oozie server proxies as the Hive user during replication.

Also, to allow cross-site Apache Hive access for Hive replication, Hive and HCatalog hosts for both clusters must be whitelisted as proxy hosts. On both clusters, the hadoop.proxyuser.hive.hosts and hadoop.proxyuser.hcat.hosts settings in the HDFS core-site.xml must contain the hostnames of all the Hive Metastore and Hive Server2 hosts for Hive and WebHCat hosts for HCatalog.

You can change these settings through Ambari.

For cross-site DistCp to function, both clusters must be able to resolve the nameservice of the other cluster. In Ambari, modify the following properties in the hdfs-site section to add the appropriate cluster names.

dfs.internal.nameservices:CLUSTER01 
dfs.client.failover.proxy.provider.CLUSTER02=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailover c ,! ProxyProvider 
dfs.ha.namenodes.CLUSTER02=nn1,nn2 
dfs.namenode.http-address.CLUSTER02.nn1=namenode01.hostname:50070 
dfs.namenode.http-address.CLUSTER02.nn2=namenode02.hostname:50070 
dfs.namenode.https-address.CLUSTER02.nn1=namenode01.hostname:50470 
dfs.namenode.https-address.CLUSTER02.nn2=namenode02.hostname:50470 
dfs.namenode.rpc-address.CLUSTER02.nn1=namenode01.hostname:8020 
dfs.namenode.rpc-address.CLUSTER02.nn2=namenode02.hostname:8020

You must configure several Hadoop and Hive properties to set up a secure environment before you can implement HiveDR jobs.

What's Next?

You need to create both the source and the destination cluster entities from the Falcon UI.

"Creating a Cluster Entity Definition Using the Web UI"

Before running a HiveDR job, you must create an initial copy of any table or database that you intend to replicate on an ongoing basis. This initialization process is sometimes referred to as bootstrapping. Falcon uses the initialized data on the target cluster as a baseline to do incremental replications.

You perform an initialization by using the export and import commands. When you export the source data, you assign a base event ID. The event ID is incremented each time a new replication job runs. When the exported data is imported, a check is done to determine if a table or database already exists that has the same base event ID. If the incremented event ID of the imported data indicates that the imported data is newer, then the existing data with the same base ID is replaced. If the imported data is older, no update is performed and no error is generated.

Prerequisites

In the following steps, replaceable text in code examples is shown in italics.

For more information, see the Apache Hive LanguageManual ImportExport wiki page.

You can quickly mirror Apache Hive databases or tables between source and target clusters with HiveServer2 endpoints. You can also enable TDE encryption for your mirror.