Hortonworks Docs » Data Lifecycle Manager 1.1.2 » DLM Administration

DLM Administration

Also available as:

PDF

loading table of contents...

Replication Concepts
Replicating Data
Failing Over Manually
- Make the destination cluster the new source
  - Remove the Ranger deny policy
- Activate a new destination cluster
DLM policy parameters
DLM version Information
Roles Required to Work with DLM
Tuning DLM Engine
Troubleshooting DLM

Replication Concepts

HDFS cloud replication
DLM supports replication of HDFS data from cluster to cloud storage and vice versa. The replication policy runs on the cluster and either pushes or pulls the data from cloud storage.
Hive cloud replication
DLM supports replication of the Hive database from a cluster with underlying HDFS to another cluster with cloud storage. It uses push-based replication, with the replication job running on the cluster with HDFS. Hive replication from cloud storage to HDFS is not supported.
Non-support of replication of Hive-Managed tables written by Spark applications.
DLM Hive replication for Managed tables relies on replication events being published by Hive in Hive Metastore for every change that is made by Hive.
Cloud replication guidelines and considerations
DLM supports replication of HDFS and Hive data between underlying HDFS and AWS S3 cloud storage.
Pairing guidelines and considerations
You should take into consideration the following items when pairing clusters in DLM.
Policy guidelines and considerations
You should take into consideration the following items when creating or modifying a replication policy.
Snapshot guidelines and considerations
Before enabling snapshots on data to be replicated, consider the following items.
Retention of Ranger policies, HDFS permissions, and ACLs
When a DLM replication job is run, data, metadata, and any Ranger policies that are associated with the replicated data are automatically exported to the target. The data on the destination is marked as read-only by adding a deny policy on the replicated data in Ranger in the destination cluster. This prevents accidental writes on the copy.
Hive replication bootstrap
DLM allows you to replicate Hive databases from a source cluster to a target location on a destination cluster.

© 2012–2019, Hortonworks, Inc.

Document licensed under the Creative Commons Attribution ShareAlike 4.0 License.

Hortonworks.com | Documentation | Support | Community