To copy data between HA clusters, use the dfs.internal.nameservices
property
in the hdfs-site.xml
file to explicitly specify the name services belonging to
the local cluster, while continuing to use the dfs.nameservices
property to
specify all of the name services in the local and remote clusters.
Use the following steps to copy data between HA clusters:
Modify the following properties in the hdfs-site.xml
file for both cluster A
and cluster B:
-
Add both name services to
dfs.nameservices
= HAA, HAB
-
Add the
dfs.internal.nameservices
property:
-
Add
dfs.ha.namenodes.<nameservice>
to both clusters:
-
In cluster A
dfs.ha.namenodes.HAB = nn1,nn2
-
In cluster B
dfs.ha.namenodes.HAA = nn1,nn2
-
Add the
dfs.namenode.rpc-address.<cluster>.<nn>
property:
-
In Cluster A:
dfs.namenode.rpc-address.HAB.nn1 = <NN1_fqdn>:8020
dfs.namenode.rpc-address.HAB.nn2 = <NN2_fqdn>:8020
-
In Cluster B:
dfs.namenode.rpc-address.HAA.nn1 = <NN1_fqdn>:8020
dfs.namenode.rpc-address.HAA.nn2 = <NN2_fqdn>:8020
-
Add the following properties to enable
distcp
over WebHDFS and secure
WebHDFS:
-
In Cluster A:
dfs.namenode.http-address.HAB.nn1 = <NN1_fqdn>:50070
dfs.namenode.http-address.HAB.nn2 = <NN2_fqdn>:50070
dfs.namenode.https-address.HAB.nn1 = <NN1_fqdn>:50470
dfs.namenode.https-address.HAB.nn2 = <NN2_fqdn>:50470
-
In Cluster B:
dfs.namenode.http-address.HAA.nn1 = <NN1_fqdn>:50070
dfs.namenode.http-address.HAA.nn2 = <NN2_fqdn>:50070
dfs.namenode.https-address.HAA.nn1 = <NN1_fqdn>:50470
dfs.namenode.https-address.HAA.nn2 = <NN2_fqdn>:50470
-
Add the
dfs.client.failover.proxy.provider.<cluster>
property:
-
In cluster A:
dfs.client.failover.proxy.provider. HAB =
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
-
In cluster B:
dfs.client.failover.proxy.provider. HAA =
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
-
Restart the HDFS service, then run the
distcp
command using the
NameService. For example:
hadoop distcp hdfs://HAA/tmp/testDistcp hdfs://HAB/tmp/