Moving the HDFS data using the distcp command

Use the hadoop distcp command to move the content from the HDFS source cluster.

You must consider the following before running the distcp command:

Ensure that the distcp user can run a MapReduce job on YARN. Otherwise, you must tweak the following configurations to enable the distcp user:
- allowed.system.users
- banned.users
- min.user.id
For source directories with a very high file count, consider creating a manual copy listing, as specified in the following example.
```
> hdfs dfs -ls hdfs://<hdfs-nameservice>/user/john.doe/application1/* > src_files
```
The copy listing output file can be read, and submitted as input one by one to a distcp job.

Considering the example of a user john.doe whose data from the /user/john.doe/application1/ directory you want to transfer to Ozone, run the distcp command as specified.

> hadoop distcp -direct hdfs://<hdfs-nameservice>/user/john.doe/application1 ofs://<ozone.service.id>/user/john.doe/

note

If your bucket type is OBS (object-store), you must use the -direct flag. Ozone supports atomic renames with FSO (file-system optimized) buckets. FSO is the default bucket layout starting OZONE-Parcel 718.2.0 version.
If using the object store (OBS) bucket type, a rename at a directory level internally means multiple client calls to rename the directory in every subdirectory and key with the required prefix, eventually ending with the top-level directory's renaming. Hence, writing to a temp directory in target and then renaming a top-level directory is a costly operation in Ozone for OBS buckets, unlike FSO buckets.
Ozone does not support file appends, concatenations, and snapshots. Therefore, Ozone also does not support the distcp flags append, -diff, -rdiff, and -blocksperchunk.
From HDFS storage to DistCp in CDH5 and HDP2 to Apache Ozone in CDP, you must set the -skipcrccheck parameter to skip the file checksum comparison. From HDFS storage to DistCp in CDH6, HDP3, and CDP to Apache Ozone in CDP, you must set the -Ddfs.checksum.combine.mode=COMPOSITE_CRC parameter to enable the file checksum between the two filesystems because the file-level checksum type (HDFS-13056) is only available in CDH/HDP/CDP based on newer Hadoop versions.