Copying data with Hadoop DistCp

DistCp (distributed copy) is a tool used to copy files in large inter-cluster and intra-cluster environments. It uses MapReduce to affect its distribution, error handling and recovery, and reporting. It expands a list of files and directories into input to map tasks, each of which copy a partition of the files specified in the source list.

The following are some of the examples of distcp commands with object stores:

Copying between directories in an object store

~ hadoop distcp 
abfs://abfscontainer@abfstorageacc.dfs.core.windows.net/myDir/testingFile.txt \ abfs://abfscontainer@abfstorageacc.dfs.core.windows.net/test/

20/05/21 08:48:09 INFO mapreduce.Job: Job job_1589987399184_0005 
completed successfully

~hadoop fs -ls abfs://abfscontainer@abfstorageacc.dfs.core.windows.net/test/

Found 1 items

-rw-r--r--   1 hdfs hdfs         41 2020-05-21 08:48 
abfs://abfscontainer@abfstorageacc.dfs.core.windows.net/test/testingFile.txt

Copying between two different object stores

~ hadoop distcp 
abfs://abfscontainer@abfstorageacc.dfs.core.windows.net/myDir/testingFile.txt \ abfs://mycontainer@mystoragehastoexist.dfs.core.windows.net/newtest/

20/05/21 08:53:26 INFO mapreduce.Job: Job job_1589987399184_0007 
completed successfully

~ hadoop fs -ls 
abfs://mycontainer@mystoragehastoexist.dfs.core.windows.net/newtest/

Found 1 items

-rw-r--r--   1 hdfs hdfs         41 2020-05-21 08:53 
abfs://mycontainer@mystoragehastoexist.dfs.core.windows.net/newtest/testingFile.txt

For more information about the DistCp commands, see DistCP documentation.