Moving the HDFS data using the distcp command

Use the hadoop distcp command to move the content from the HDFS source cluster.

You must consider the following before running the distcp command:

Ensure that the distcp user can run a MapReduce job on YARN. Otherwise, you must tweak the following configurations to enable the distcp user:
- allowed.system.users
- banned.users
- min.user.id
For source directories with a very high file count, consider creating a manual copy listing, as specified in the following example.
```
> hdfs dfs -ls hdfs://<hdfs-nameservice>/user/john.doe/application1/* > src_files
```
The copy listing output file can be read, and submitted as input one by one to a distcp job.

Considering the example of a user john.doe whose data from the /user/john.doe/application1/ directory you want to transfer to Ozone, run the distcp command as specified.

> hadoop distcp -direct hdfs://<hdfs-nameservice>/user/john.doe/application1 ofs://<ozone.service.id>/user/john.doe/

note

You must use the -direct flag because Ozone does not support atomic renames. Therefore, a rename at a directory level internally means multiple client calls to rename the directory in every subdirectory and key with the required prefix, and eventually ends with renaming of the top-level directory. Hence writing to a temp directory in target, and then renaming a top level directory is a costly operation in Ozone.
Ozone does not support file appends, concatenations, and snapshots. Therefore, Ozone also does not support the distcp flags append, -diff, -rdiff, and -blocksperchunk.
From HDFS storage to DistCp in CDH5 and HDP2 to Apache Ozone in CDP, you must set the -skipcrccheck parameter to skip the file checksum comparison. From HDFS storage to DistCp in CDH6, HDP3, and CDP to Apache Ozone in CDP, you must set the -Ddfs.checksum.combine.mode=COMPOSITE_CRC parameter to enable the file checksum between the two filesystems because the file-level checksum type (HDFS-13056) is only available in CDH/HDP/CDP based on newer Hadoop versions.