Protecting the Azure Credentials for ADLS with Credential Providers
All ADLS credential properties can be protected by credential providers.
To provision the credentials:
hadoop credential create fs.adl.oauth2.client.id -value 123 -provider localjceks://file/home/foo/adls.jceks hadoop credential create fs.adl.oauth2.refresh.token -value 123 -provider localjceks://file/home/foo/adls.jceks
Next, configure the following configuration properties, either on the command line or in
the core-site.xml
configuration file:
<property> <name>fs.adl.oauth2.access.token.provider.type</name> <value>RefreshToken</value> </property> <property> <name>hadoop.security.credential.provider.path</name> <value>localjceks://file/home/foo/adls.jceks</value> </property>
The hadoop.security.credential.provider.path
should indicate the
path to interrogate for protected credentials.
You may optionally add the provider path property to the distcp
command line
instead of adding a job-specific configuration to a generic core-site.xml
. The
options enclosed in square brackets illustrate this capability.
hadoop distcp [-D fs.adl.oauth2.access.token.provider.type=RefreshToken -D hadoop.security.credential.provider.path=localjceks://file/home/user/adls.jceks] hdfs://<NameNode Hostname>:9001/user/foo/srcDir adl://<Account Name>.azuredatalakestore.net/tgtDir/
Related Links