Accessing Cloud Data
Also available as:
PDF
loading table of contents...

Using DistCp with Azure ADLS and WASB

  • The -append option is not supported.

  • The -diff option is not supported.

  • The -atomic option causes a rename of the temporary data, which slows down the upload. Avoid using this option.

  • ADLS implements the same permissions model as HDFS, so some of the -p options work.

  • WASB supports getting and setting the permissions, but these permissions do not control access to the data.. What they can do is ensure is that permissions can be restored after a back-up.

  • You can tune fs.azure.selfthrottling.read.factor and fs.azure.selfthrottling.write.factor. Refer to Maximizing HDInsight throughput to Azure Blob Storage blog post.