Chapter 7. Failing Over Manually
If a source cluster used in a replication policy is offline and will not be brought online for an extended period, you should manually fail over the destination cluster to serve as the new source. After failover, the new source cluster will receive read and write requests. You might also want to designate a new destination cluster to which data will be copied from the new source.
Make the Destination Cluster the New Source
If the source cluster becomes unavailable for an extended period, you can configure the destination cluster to serve as the new source. Read and write requests from clients will then be redirected from the old source to the new source cluster.
Prerequisites
You need the name of the cluster that is offline.
Steps
Log in to the DPS UI as Infra Admin.
Access the DLM UI by clicking the DPS icon in the upper left of the page and then clicking the Data Lifecycle Manager icon.
Identify the set of replication policies for which the offline cluster is the source in a replication relationship.
Click Policies in the navigation pane.
In the Filter field, type the name of the offline cluster.
A list appears that displays the cluster name as a source or a destination cluster.
From the list, select the cluster name under Source Cluster.
The page content shows only the policies that use the selected cluster as the source for replication.
Delete all replication policies that use the offline cluster as the replication source.
At the end of each row in the policies list, click the
(Actions) icon.Click Delete in the drop-down menu, and then click OK to confirm deletion.
If a replication policy is in the process of running a job, the job aborts when you delete the policy.
![[Important]](../common/images/admon/important.png)
Important After a replication policy is deleted, it cannot be retrieved.
If the Ranger deny policy is enabled, remove the deny policy that is on the destination cluster.
Determine if the Ranger deny policy is enabled.
Navigate to the Ambari UI.
In the services list, click DLM Engine.
Click Configs>Advanced.
Scroll to the parameter
beacon.ranger.plugin.create.denypolicyand verify if the Ranger Deny Policy is enabled or disabled.
If the Ranger Deny Policy is enabled, you must disable it.
Log in to the destination cluster, access Ranger, and then navigate to Ranger admin resource policies.
Identify Ranger policies that start with “<sourcecluster>_beacon deny policy for”.
Activate a New Destination Cluster
If you have not prepared a cluster in advance to serve as an alternate destination in a failover scenario, then you must install the DLM Engine, configure the clusters for use by DLM, and pair the clusters before you can create new replication policies and begin copying data to the new destination.
Prerequisites
You must have the name of the cluster you want to configure as the new destination.
Steps
Identify the Ambari-managed cluster to use as the new destination.
Install the DLM Engine on the new destination, if it is not already installed.
Follow the instructions in Setting Up the DPS Services for the following tasks, as needed:
Register Clusters with DPS
Enable Services
Pair the clusters you are using as source and destination, if they are not already paired.
Ensure that the HDFS folders or Hive databases to be copied either do not exist or are empty on the new destination cluster.
This is required prior to bootstrapping data from the source cluster to the destination cluster. Otherwise, the initial copy job fails.
Create and submit new replication policies between the source and destination clusters.
The first time a new policy is submitted, the entire contents of the source dataset is copied to the destination. Depending on the size of each dataset, these initial bootstrap copies can take a significant amount of time. After the initial copy, subsequent copies are incremental.
More Information