Scaling Namespaces and Optimizing Data Storage
Also available as:
PDF
loading table of contents...

Pathnames on clusters with federated and non-federated NameNodes

The pathnames of the namespace URIs used in clusters with federated and non-federated NameNodes depend on the value configured for the default file system.

In cluster without federated NameNodes, the default file system is set to the NameNode of the cluster. For example, fs.defaultFS= hdfs://nnOfThatCluster:port/. In a federation with ViewFs, the default file system for each cluster is set to the name of the cluster. For example, fs.defaultFS=viewfs://clusterX/.

The value configured for the default file system determines the pathnames of the various URIs used in clusters.

The following table contains examples of pathnames in URIs used in clusters without federated NameNodes:

Pathname Example Description
/foo/bar The relative path equivalent to hdfs://nnOfClusterX:port/foo/bar.
hdfs://nnOfClusterX:port/foo/bar The fully qualified path to the foo/bar directory on cluster X.
Note
Note
You should consider using the relative path instead of the absolute path as that approach helps in moving an application and its data transparently across clusters when required.
hdfs://nnOfClusterY:port/foo/bar The URI for referring to the foo/bar directory on cluster Y. For example, the command for copying files from one cluster to another can be as follows:
distcp hdfs://namenodeClusterY:port/pathSrc hdfs://namenodeClusterZ:port/pathDest
hftp://nnClusterX:port/foo/bar The URI for accessing the foo/bar directory on cluster X through the HFTP file system.
webhdfs://nnClusterX:port/foo/bar The URI for accessing the foo/bar directory on cluster X through the WebHDFS file system.
http://namenodeClusterX:http_port/webhdfs/v1/foo/bar and http://proxyClusterX:http_port/foo/bar The HTTP URLs respectively for accessing files through WebHDFS REST API and HDFS proxy.

The following table contains examples of pathnames in URIs used in clusters with ViewFs:

Pathname Example Description
/foo/bar The relative path equivalent to viewfs://clusterX/foo/bar.
viewfs://clusterX/foo/bar The fully qualified path to the foo/bar directory on cluster X. Unlike a non-federated environment where you must specify the NameNode in the path to each namespace volume, you can use the mount table name in a federated environment to qualify the paths to all the namespace volumes of the cluster.
Note
Note
You should consider using the relative path instead of the absolute path as that approach helps in moving an application and its data transparently across clusters when required.
viewfs://clusterY/foo/bar The URI for referring to the foo/bar directory on cluster Y. For example, the command for copying files from one cluster to another can be as follows:
distcp viewfs://clusterY/pathSrc viewfs://clusterZ/pathDest
viewfs://clusterX-hftp/foo/bar The URI for accessing the foo/bar directory on cluster X through the HFTP file system.
viewfs://clusterX-webhdfs/foo/bar The URI for accessing the foo/bar directory on cluster X through the WebHDFS file system.
http://namenodeClusterX:http_port/webhdfs/v1/foo/bar and http://proxyClusterX:http_port/foo/bar The HTTP URLs respectively for accessing files through WebHDFS REST API and HDFS proxy.
Note
Note
You cannot rename files or directories across NameNodes or clusters in both federated and non-federated environments. For example, while the following renaming operation can be performed within the scope of a NameNode, the same is not possible if the source and destination paths belong to different NameNodes:
rename /user/joe/myStuff /data/foo/bar