Cloudera Docs

Migrating HDFS data to HDFS

Copy curated HDFS directories from a source CDP Private Cloud Base cluster to HDFS on a target cluster without altering source data, using Cloudera Replication Manager-backed executions.

Planning completes with Cloudera Replication Manager replication policies copying every selected directory into mapped HDFS paths on the target cluster. The source HDFS data remains intact.

  • Both source and target clusters must be displayed on the Clusters page. For instructions, see Registering clusters.

  • The target cluster must run HDFS and Cloudera Replication Manager.

  • CMA Agents must operate on source and target clusters.

  • Cloudera Replication Manager must be reachable. Cloudera Migration Assistant submits policies through Cloudera Replication Manager APIs.

  • Secure clusters authenticate with Kerberos, so you must confirm that the migration user principal works for filesystem access.

Whenever Discovery trees look stale before mapping, rerun the steps in Scanning clusters. Collections exported from scans help bound execution scopes . For more information, see Creating collections for migration if you segregate phased cutovers.

Step 1 — Scan the source cluster (HDFS)

  1. Open Clusters, then select the source cluster.
  2. Click Start Scanning.
  3. Select Hdfs data scan.
  4. Optional: Constrain the scan path, for example, /tmp or /user, to shorten runs on massive namespaces.
  5. Click Scan selected and watch the progress drawer until completion.

    After success, Discovery lists HDFS trees with sizing and ownership snapshots.

Step 2 — Scan the target cluster (HDFS)

  1. From Clusters, open the target cluster.
  2. Click Start Scanning.
  3. Select Hdfs data scan.
  4. Click Scan selected and wait until the status turns green.

    HDFS hierarchies populate inside Discovery afterward.

Step 3 — Create a migration plan

  1. Go to Plans.
  2. Click Create Plan.
  3. Set Name, select the source HDFS cluster, and the target HDFS cluster.
  4. Complete the wizard (Create) and land on the plan detail surface.

Step 4 — Map HDFS directories to HDFS paths on the Data tab

The Data tab displays source HDFS on the left and target HDFS on the right.

  1. Go to Plan > Data.
  2. On the target side, browse to the parent directory that should receive migrated data.
  3. On the source side, go to the parent directory that contains data you want to migrate.
  4. Hover the directory and click Copy. Cloudera Migration Assistant persists a structural mapping.
  5. Switch to the Review view any time to view a list of your current structure mappings.
  6. Repeat the mapping workflow for additional directories whenever you broaden scope.

    You can map each directory to separate target prefixes or combine them under shared parents. The layout must meet the Cloudera Replication Manager requirements for your tenancy model.

Step 5 — Create and run an execution

  1. Open Execution.
  2. Select Create Execution.
  3. Complete the steps in the configuration wizard (General, Scope, Review), and then click Finish.
  4. Open the execution dashboard and trigger Run All.

    During execution the following stages occur:

    • Policy creation – Cloudera Migration Assistant registers Cloudera Replication Manager policies mapping HDFS prefixes to destination HDFS paths.

    • Data replication – Cloudera Replication Manager invokes DistCp-style transfers governed by Cloudera Replication Manager settings.

    • Verification – CMA Master tracks step health until completion (COMPLETED or FAILED statuses per step).

Step 6 — Verify the migration

  1. Confirm that the execution steps display COMPLETED.
  2. Inspect HDFS paths inside Discovery on the target cluster.
  3. Optional: Validate migrated paths using the HDFS CLI on the target cluster. 
    hdfs dfs -ls /path/to/migrated/data