Reference the managed configuration files and add AWS credentials

You can manage configuration files with Cloudera Manager and still having a version of the configuration that contains the Amazon S3 credentials.

This option allows you to continue to use the configuration files managed by Cloudera Manager. If you deploy new configuration files, the new values are included by reference in your copy of the configuration files while also maintaining a version of the configuration that contains the Amazon S3 credentials.

  1. Create a local directory under your home directory.
  2. Copy the configuration files from /etc/hadoop/conf to the new directory.
  3. Set the permissions for the configuration files appropriately for your environment.
  4. Remove all elements within the <configuration> element in each configuration file.
  5. Add an XML <include> element within the <configuration> element to reference the configuration files managed by Cloudera Manager.
    For example:
    <property>
        <name>fs.s3a.access.key</name>
        <value>Amazon S3 Access Key</value>
    </property>
    
    <property>
        <name>fs.s3a.secret.key</name>
        <value>Amazon S3 Secret Key</value>
    </property>
  6. Reference these versions of the configuration files when submitting jobs by running the following command:
    export HADOOP_CONF_DIR=path to local configuration directory