This section provides information on the various components of the Apache Hadoop ecosystem and setting them up for high availability.
HDPs Full-Stack HA Architecture
Hortonworks Data Platform (HDP) is an open source distribution powered by Apache Hadoop. HDP provides you with the actual Apache-released versions of the stack with all the necessary bug fixes to make all the components in the stack interoperable in your production environments. This stack uses multiple ‘master’ services whose failure would cause functionality outage in your Hadoop cluster. Hortonworks’ Full-Stack High Availability architecture provides a common framework to make all the master services resilient to failures.
HDP uses industry proven HA technologies in order to provide a reliable HA solution.
The Hadoop stack contains multiple services (HDFS, MapReduce, HBase, etc.) and each of these services have their own co-dependencies. A client application, that interacts with Hadoop, can depend on one or more of these services. A highly available Hadoop platform must ensure that the NameNode master service as well as client applications are resilient to critical failure services. Hortonworks’ Full-Stack HA architecture considers this global view.
Also see. Hortonworks blog on NameNode HA with Hadoop 1.0 The HDP HA architecture has the following key properties:
It provides high availability for the NameNode master daemon service.
When the NameNode master daemon fails over, the HA solution initiates the following actions:
Dependent services (like JobTracker) automatically detect the failure or fail over of the co-dependent component (NameNode) and these dependent services pause, retry, and recover the failed service. (For example, the JobTracker does not launch new jobs or kill jobs that have been waiting for the NameNode.)
Applications running inside and outside the Hadoop cluster also automatically pause and retry their connection to the failed service.
The above actions are highlighted in the following illustration. This illustration shows how HDFS clients and MapReduce services (Jobtracker daemon) handle the NameNode fail over.
To configure High Availability for your Hadoop cluster see:
High Availability using Red Hat
