Scaling a cluster using HDFS federation
An HDFS federation scales a cluster horizontally by providing support for multiple independent NameNodes and namespaces, with the DataNodes available as common block storage for all the NameNodes. The support for multiple namespaces improves cluster scalability and provides isolation in a multitenanted environment.
The earlier HDFS configurations without support for federations can be constrained by a single namespace, and consequently, a single NameNode for the entire cluster. In this non-federated environment, the NameNode stores the entire file system metadata in memory. This limits the number of blocks, files, and directories supported on the file system to what can be accommodated in the memory of a single NameNode. In addition, file system operations are limited to the throughput of a single NameNode. These issues of scalability and performance are addressed through an HDFS federation.
Note | |
---|---|
You can use ViewFs to create personalized
namespace views. ViewFs is analogous to client side mount tables in UNIX or Linux
systems. |