Typical Hadoop Cluster
Hadoop and HBase clusters have two types of machines:
Masters -- HDFS NameNode, YARN ResourceManager, and HBase Master.
Slaves -- HDFS DataNodes, YARN NodeManagers, and HBase RegionServers.
The DataNodes, NodeManagers, and HBase RegionServers are co-located or co-deployed for optimal data locality.
In addition, HBase requires the use of a separate component (ZooKeeper) to manage the HBase cluster.
Hortonworks recommends separating master and slave nodes because:
Task/application workloads on the slave nodes should be isolated from the masters.
Slaves nodes are frequently decommissioned for maintenance.
For evaluation purposes, it is possible to deploy Hadoop using a single-node installation (all the masters and slave processes reside on the same machine).
For a small two-node cluster, the NameNode and the ResourceManager are both on the master node, with the DataNode and NodeManager on the slave node.
Clusters of three or more machines typically use a single NameNode and ResourceManager with all the other nodes as slave nodes. A High-Availability (HA) cluster would use a primary and secondary NameNode , and might also use a primary and secondary ResourceManager .
Typically, a medium-to -large Hadoop cluster consists of a two-level or three-level architecture built with rack-mounted servers. Each rack of servers is interconnected using a 1 Gigabyte Ethernet (GbE) switch. Each rack-level switch is connected to a cluster-level switch (which is typically a larger port-density 10GbE switch). These cluster-level switches may also interconnect with other cluster-level switches or even uplink to another level of switching infrastructure.