Configuring ADLS Connectivity

1. To create your ADLS account, see the Microsoft documentation.
2. Create the service principal in the Azure portal. See the Microsoft documentation on creating a service principal.
3. Grant the service principal permission to access the ADLS account. See the Microsoft documentation on Authorization and access control. Review the section, "Using ACLs for operations on file systems" for information about granting the service principal permission to access the account.

   You can skip the section on RBAC (role-based access control) because RBAC is used for management and you only need data access.

4. Configure your CDH cluster to access your ADLS account. To access ADLS storage from a CDH cluster, you provide values for the following properties when submitting jobs:

   ADLS Access Properties

   Property Description	Property Name
   Provider Type	dfs.adls.oauth2.access.token.provider.type The value of this property should be ClientCredential
   Client ID	dfs.adls.oauth2.client.id
   Client Secret	dfs.adls.oauth2.credential
   Refresh URL	dfs.adls.oauth2.refresh.url

   There are several methods you can use to provide these properties to your jobs. There are security and other considerations for each method. Select one of the following methods to access data in ADLS:

   • User-Supplied Key for Each Job
   • Single Master Key for Cluster-Wide Access
   • User-Supplied Key stored in a Hadoop Credential Provider
   • Create a Hadoop Credential Provider and reference it in a customized copy of the core-site.xml file for the service

1. After configuring access, test your configuration by running the following command that lists files in your ADLS account:

   hadoop fs -ls adl://your_account.azuredatalakestore.net/

   If your configuration is correct, this command lists the files in your account.

2. After successfully testing your configuration, you can access the ADLS account from MRv2, Hive, Hive-on-Spark , Spark 1.6, Spark 2.1, or HBase by using the following URI:

   adl://your_account.azuredatalakestore.net

Use Cloudera Manager to save the values in the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml.

• Advantages: All users can access the ADLS storage
• Disadvantages: This is a highly insecure means of providing access to ADLS for the following reasons:
  • The credentials will appear in all Cloudera Manager-managed configuration files for all services in the cluster.
  • The credentials will appear in the Job History server.

1. Open the Cloudera Manager Admin Console and go to Cluster Name > Configuration > Advanced Configuration Snippets.
2. Enter the following in the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml:

   <property>
     <name>dfs.adls.oauth2.access.token.provider.type</name>
     <value>ClientCredential</value>
   </property>
   <property>
     <name>dfs.adls.oauth2.client.id</name>
     <value>CLIENT ID</value>
   </property>
   <property>
     <name>dfs.adls.oauth2.credential</name>
     <value>CLIENT SECRET</value>
   </property>
   <property>
     <name>dfs.adls.oauth2.refresh.url</name>
     <value>REFRESH URL</value>
   </property>

3. Click Save Changes.
4. Click Restart Stale Services so the cluster can read the new configuration information.

• Advantages: Credentials are securely stored in the credential provider.
• Disadvantages: Works with MapReduce2 and Spark only (Hive, Impala, and HBase are not supported).

1. Create a Credential Provider.
   1. Create a password for the Hadoop Credential Provider and export it to the environment:

      export HADOOP_CREDSTORE_PASSWORD=password

   2. Provision the credentials by running the following commands:

      hadoop credential create dfs.adls.oauth2.client.id -provider jceks://hdfs/user/USER_NAME/adls-cred.jceks -value client ID
      hadoop credential create dfs.adls.oauth2.credential -provider jceks://hdfs/user/USER_NAME/adls-cred.jceks -value client secret
      hadoop credential create dfs.adls.oauth2.refresh.url -provider jceks://hdfs/user/USER_NAME/adls-cred.jceks -value refresh URL

      You can omit the -value option and its value and the command will prompt the user to enter the value.

      For more details on the hadoop credential command, see Credential Management (Apache Software Foundation).

2. Reference the Credential Provider on the command line when submitting jobs:

   hadoop command
        -Ddfs.adls.oauth2.access.token.provider.type=ClientCredential \
        -Dhadoop.security.credential.provider.path=jceks://hdfs/user/USER_NAME/adls-cred.jceks \
        adl://<store>.azuredatalakestore.net/

• Advantages: all users can access the ADLS storage
• Disadvantages: you must pass the path to the credential store on the command line.

1. Create a Credential Provider:
   1. Create a password for the Hadoop Credential Provider and export it to the environment:

      export HADOOP_CREDSTORE_PASSWORD=password

   2. Provision the credentials by running the following commands:

      hadoop credential create dfs.adls.oauth2.client.id -provider jceks://hdfs/user/USER_NAME/adlskeyfile.jceks -value client ID
      hadoop credential create dfs.adls.oauth2.credential -provider jceks://hdfs/user/USER_NAME/adlskeyfile.jceks -value client secret
      hadoop credential create dfs.adls.oauth2.refresh.url -provider jceks://hdfs/user/USER_NAME/adlskeyfile.jceks -value refresh URL

      You can omit the -value option and its value and the command will prompt the user to enter the value.

      For more details on the hadoop credential command, see Credential Management (Apache Software Foundation).

2. Copy the contents of the /etc/service/conf directory to a working directory. The service can be one of the following verify list:
   • yarn
   • spark
   • spark2
   Use the --dereference option when copying the file so that symlinks are correctly resolved. For example:

   cp -r --dereference /etc/spark/conf ~/my_custom_config_directory

   Change the ownership so that you can edit the files:

   sudo chown --recursive $USER ~/custom-conf-file/*

3. Add the following to the core-site.xml file in the working directory:

   <property>
     <name>hadoop.security.credential.provider.path</name>
     <value>jceks://hdfs/path_to_credential_store_file</value>
   </property>
   <property>
     <name>dfs.adls.oauth2.access.token.provider.type</name>
     <value>ClientCredential</value>
   </property> 

   The value of the path_to_credential_store_file should be the same as the value for the --provider option in the hadoop credential create command described in step 1.

4. Set the HADOOP_CONF_DIR environment variable to the location of the working directory:

   export HADOOP_CONF_DIR=path_to_working_directory

You can use a Hadoop Credential Provider to specify ADLS credentials, which allows you to run jobs without having to enter the access key and secret key on the command line. This prevents these credentials from being exposed in console output, log files, configuration files, and other artifacts. Running the command in this way requires that you provision a credential store to securely store the access key and secret key. The credential store file is saved in HDFS.