DLM Administration
Also available as:
PDF

Chapter 7. Failing Over Manually

If a source cluster used in a replication policy is offline and will not be brought online for an extended period, you should manually fail over the destination cluster to serve as the new source. After failover, the new source cluster will receive read and write requests. You might also want to designate a new destination cluster to which data will be copied from the new source.

Make the Destination Cluster the New Source

If the source cluster becomes unavailable for an extended period, you can configure the destination cluster to serve as the new source. Read and write requests from clients will then be redirected from the old source to the new source cluster.

Prerequisites

You need the name of the cluster that is offline.

Steps

  1. Log in to the DPS UI as Infra Admin.

  2. Access the DLM UI by clicking the DPS icon in the upper left of the page and then clicking the Data Lifecycle Manager icon.

  3. Identify the set of replication policies for which the offline cluster is the source in a replication relationship.

    1. Click Policies in the navigation pane.

    2. In the Filter field, type the name of the offline cluster.

      A list appears that displays the cluster name as a source or a destination cluster.

    3. From the list, select the cluster name under Source Cluster.

      The page content shows only the policies that use the selected cluster as the source for replication.

  4. Delete all replication policies that use the offline cluster as the replication source.

    1. At the end of each row in the policies list, click the (Actions) icon.

    2. Click Delete in the drop-down menu, and then click OK to confirm deletion.

      If a replication policy is in the process of running a job, the job aborts when you delete the policy.

      [Important]Important

      After a replication policy is deleted, it cannot be retrieved.

  5. If the Ranger deny policy is enabled, remove the deny policy that is on the destination cluster.

    1. Determine if the Ranger deny policy is enabled.

      1. Navigate to the Ambari UI.

      2. In the services list, click DLM Engine.

      3. Click Configs>Advanced.

      4. Scroll to the parameter beacon.ranger.plugin.create.denypolicy and verify if the Ranger Deny Policy is enabled or disabled.

    2. If the Ranger Deny Policy is enabled, you must disable it.

      1. Log in to the destination cluster, access Ranger, and then navigate to Ranger admin resource policies.

      2. Identify Ranger policies that start with “<sourcecluster>_beacon deny policy for”.

Activate a New Destination Cluster

If you have not prepared a cluster in advance to serve as an alternate destination in a failover scenario, then you must install the DLM Engine, configure the clusters for use by DLM, and pair the clusters before you can create new replication policies and begin copying data to the new destination.

Prerequisites

You must have the name of the cluster you want to configure as the new destination.

Steps

  1. Identify the Ambari-managed cluster to use as the new destination.

  2. Install the DLM Engine on the new destination, if it is not already installed.

    Installing DPS Services, Engines, and Agents

  3. Follow the instructions in Setting Up the DPS Services for the following tasks, as needed:

    • Register Clusters with DPS

    • Enable Services

  4. Pair the clusters you are using as source and destination, if they are not already paired.

    Pair Clusters for Replication

  5. Ensure that the HDFS folders or Hive databases to be copied either do not exist or are empty on the new destination cluster.

    This is required prior to bootstrapping data from the source cluster to the destination cluster. Otherwise, the initial copy job fails.

  6. Create and submit new replication policies between the source and destination clusters.

    Create a Replication Policy

    The first time a new policy is submitted, the entire contents of the source dataset is copied to the destination. Depending on the size of each dataset, these initial bootstrap copies can take a significant amount of time. After the initial copy, subsequent copies are incremental.

More Information

Hortonworks DPS Installation