Copying Cluster Data Using DistCp

• hftp:// is the source protocol.
• namenode-location is the CDH 4 (source) NameNode hostname as defined by its configured fs.default.name.
• 50070 is the NameNode's HTTP server port, as defined by the configured dfs.http.address.

• hdfs:// is the destination protocol
• nameservice-id or namenode-location is the CDH 5 (destination) NameNode hostname as defined by its configured fs.defaultFS.
• basePath in both examples refers to the directory you want to copy, if one is specifically needed.

1. Provision the credentials by running the following commands:

   hadoop credential create fs.s3a.access.key -value access_key -provider path_to_credential_store_file
   hadoop credential create fs.s3a.secret.key -value secret_key -provider path_to_credential_store_file

   Enter the value for the -provider option using the following syntax:

   provider_type://file/path_to_credential_store_file

   provider_type can be one of the following:

   • jceks – retrieves credentials from a JavaKeystoreProvider. Use when the credential provider is located in HDFS and requires access from multiple hosts.
   • localjceks – retrieves credentials from a LocalJavaKeystoreProvider. Use when the credential provider is located on the local filesystem and access is only required from this single host. Cloudera recommends localjceks.
   For example:

   hadoop credential create fs.s3a.access.key -value foobar -provider localjceks://file/home/keystores/aws.jceks
   hadoop credential create fs.s3a.secret.key -value barfoo -provider localjceks://file/home/keystores/aws.jceks

   For more details on the hadoop credential command, see Credential Management (Apache Software Foundation).

2. Copy the contents of the /etc/hadoop/conf directory to a working directory.
3. Add the following to the core-site.xml file in the working directory:

   <property>
   <name>hadoop.security.credential.provider.path</name>
   <value>provider_type://path_to_credential_store_file</value>
   </property>

4. Set the HADOOP_CONF_DIR environment variable to the location of the working directory:

   export HADOOP_CONF_DIR=path_to_working_directory

hadoop distcp hdfs://source_path s3a://bucket_name/destination_path
-D hadoop.security.credential.provider.path=path_to_credential_store_file

[realms]
HADOOP.QA.domain.COM = { kdc = kdc.domain.com:88 admin_server = admin.test.com:749
default_domain = domain.com supported_enctypes = arcfour-hmac:normal des-cbc-crc:normal
des-cbc-md5:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3 } 

[domain_realm]
.domain.com = HADOOP.test.domain.COM
domain.com = HADOOP.test.domain.COM
test03.domain.com = HADOOP.QA.domain.COM

1. Open the Cloudera Manager Admin Console.
2. Go to the HDFS service.
3. Click the Configuration tab.
4. Select Scope > HDFS-1 (Service-Wide)
5. Select Category > Security.
6. Locate the Hadoop RPC Protection property and select authentication.
7. Click Save Changes to commit the changes.

1. Open the Cloudera Manager Admin Console.
2. Go to the HDFS service.
3. Click the Configuration tab.
4. Select Scope > Gateway
5. Select Category > Advanced.
6. Edit the HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml property to add:

   <property>
     <name>dfs.namenode.kerberos.principal.pattern</name>
     <value>*</value>
   </property>

7. Click Save Changes to commit the changes.

<property>
<name>ssl.client.truststore.location</name>
<value>path_to_truststore</value>
</property>

<property>
<name>ssl.client.truststore.password</name>
<value>XXXXXX</value>
</property>

<property>
<name>ssl.client.truststore.type</name>
<value>jks</value>
</property>

hadoop distcp hdfs://test01.domain.com:8020/user/alice hdfs://test02.domain.com:8020/user/alice

[libdefaults]
udp_preference_limit = 1