Using a credential provider to secure S3 credentials

You can run the distcp command without having to enter the access key and secret key on the command line. This prevents these credentials from being exposed in console output, log files, configuration files, and other artifacts.

Running the distcp command in this way requires that you provision a credential store to securely store the access key and secret key. The credential store file is saved in HDFS.

Provision the credentials by running the following commands:
```
hadoop credential create fs.s3a.access.key -value access_key -provider jceks://hdfs/path_to_credential_store_file
hadoop credential create fs.s3a.secret.key -value secret_key -provider jceks://hdfs/path_to_credential_store_file
```
For example:
```
hadoop credential create fs.s3a.access.key -value foobar -provider jceks://hdfs/user/alice/home/keystores/aws.jceks
hadoop credential create fs.s3a.secret.key -value barfoo -provider jceks://hdfs/user/alice/home/keystores/aws.jceks
```
You can omit the -value option and its value and the command will prompt the user to enter the value.

For more details on the hadoop credential command, see Credential Management (Apache Software Foundation).
note
If you configure the hadoop.security.credential.provider.path property to jceks://hdfs/path_to_credential_store_file, ZKFC might fail to start and NameNodes might remain in standby state. So, Cloudera recommends not to add or change this property in Cloudera Manager.
Copy the contents of the /etc/hadoop/conf directory to a working directory.

Add the following to the core-site.xml file in the working directory:

<property>
<name>hadoop.security.credential.provider.path</name>
<value>jceks://hdfs/path_to_credential_store_file</value>
</property>

Set the HADOOP_CONF_DIR environment variable to the location of the working directory:
```
export HADOOP_CONF_DIR=path_to_working_directory
```

After completing these steps, you can run the distcp command using the following syntax:

hadoop distcp source_path s3a://destination_path

You can also reference the credential store on the command line, without having to enter it in a copy of the core-site.xml file. You also do not need to set a value for HADOOP_CONF_DIR. Use the following syntax:

hadoop distcp source_path s3a://bucket_name/destination_path
-Dhadoop.security.credential.provider.path=jceks://hdfspath_to_credential_store_file