Make a modified copy of the configuration files

You can add the S3 credentials to a copy of the configuration files.

Everytime you update the client configuration files on Cloudera Manager, you must repeat these steps to use the new configurations.

  1. Copy the contents of the /etc/hadoop/conf directory to a local working directory under the home directory of the host where you will submit the job.
  2. Set the permissions for the configuration files appropriately for your environment. Ensure that unauthorized users cannot access sensitive configurations in these files.
  3. Add the following to the core-site.xml file within the <configuration> element:
    <property>
        <name>fs.s3a.access.key</name>
        <value>Amazon S3 Access Key</value>
    </property>
    
    <property>
        <name>fs.s3a.secret.key</name>
        <value>Amazon S3 Secret Key</value>
    </property>
  4. Reference these verions of the configuration files when submitting jobs by running the following command:
    export HADOOP_CONF_DIR=path to local configuration directory