Moving data from HDFS to Ozone using the distcp command

Use the hadoop distcp command to move the content from the HDFS source cluster.

You must consider the following before running the distcp command:

Execute the distcp command from the destination cluster.
Ensure that the distcp user can run a MapReduce job on YARN. Otherwise, you must tweak the following configurations to enable the distcp user:
- allowed.system.users
- banned.users
- min.user.id
If the source directories have a high file count, you can create a manual copy listing as specified in the following example.
```
> hdfs dfs -ls hdfs://<hdfs-nameservice>/user/john.doe/application1/* > src_files
```
The copy listing output file can be read and submitted as input one by one to a distcp job.

Consider the example of a user john.doe whose data is from the /user/john.doe/application1/ directory and you want to transfer to Ozone, run the distcp command as specified.

> hadoop distcp -direct hdfs://<hdfs-nameservice>/user/john.doe/application1 ofs://<ozone.service.id>/user/john.doe/

note

If your bucket type is OBS (object-store), you must use the -direct flag. Ozone supports atomic renames with FSO (file-system optimized) buckets. FSO is the default bucket layout starting OZONE-Parcel 718.2.0 version.
If using the object store (OBS) bucket type, a rename at a directory level internally means multiple client calls to rename the directory in every subdirectory and key with the required prefix, eventually ending with the top-level directory's renaming. Hence, writing to a temp directory in target and then renaming a top-level directory is a costly operation in Ozone for OBS buckets, unlike FSO buckets.
Ozone does not support file appends, concatenations, and snapshots. Therefore, Ozone also does not support the distcp flags append, -diff, -rdiff, and -blocksperchunk.
From HDFS storage to DistCp in CDH5 and HDP2 to Apache Ozone in CDP, you must set the -skipcrccheck parameter to skip the file checksum comparison. From HDFS storage to DistCp in CDH6, HDP3, and CDP to Apache Ozone in CDP, you must set the -Ddfs.checksum.combine.mode=COMPOSITE_CRC -Dozone.client.checksum.type=CRC32C parameter to enable the file checksum between the two filesystems because the file-level checksum type (HDFS-13056) is only available in CDH/HDP/CDP based on newer Hadoop versions.
If Kerberos is enabled on the Ozone cluster, add the ozone.om.kerberos.principal.pattern = * parameter.

For example, to distcp from a Kerberized CDP HDFS cluster ns1 /tmp directory to a Kerberized CDP Ozone cluster ozone1 under volume v1, bucket b1, directory /dst, execute the following command at a destination cluster host:

hadoop distcp \
-Ddfs.checksum.combine.mode=COMPOSITE_CRC \ -Dozone.client.checksum.type=CRC32C \
-Dozone.om.kerberos.principal.pattern=* \
 hdfs://ns1/tmp/ \
 ofs://ozone1707264383/v1/b1/dst/