Using DistCp with Azure ADLS and WASB
The
-append
option is not supported.The
-diff
option is not supported.The
-atomic
option causes a rename of the temporary data, which slows down the upload. Avoid using this option.ADLS implements the same permissions model as HDFS, so some of the
-p
options work.WASB supports getting and setting the permissions, but these permissions do not control access to the data.. What they can do is ensure is that permissions can be restored after a back-up.
You can tune
fs.azure.selfthrottling.read.factor
andfs.azure.selfthrottling.write.factor
. Refer to Maximizing HDInsight throughput to Azure Blob Storage blog post.