Hive Metastore leader election
Leader election lets Hive Metastore (HMS) elect a single instance to run housekeeping tasks, which prevents multiple instances from duplicating the same work and overloading the metastore database.
HMS runs housekeeping tasks such as generating and running compaction tasks, discovering partitions for external tables, cleaning up expired data. When a cluster runs multiple HMS instances, these tasks must run on a single instance. Otherwise, every instance runs the same tasks and duplicates the work.
Duplicated housekeeping work generates a large number of read or write operations against the shared metastore database. In environments that run many concurrent metadata operations, such as replication jobs that use multiple threads to fetch metadata, this additional load can degrade performance for all workloads that share the database.
Tasks that run only on the leader
When leader election is configured, HMS runs the following housekeeping tasks only on the elected leader:
- Compaction tasks, including the initiator, cleaner, and table status updater
- Automatic partition discovery and partition management
- Cleanup of the changed-message (CM) directory
- Transaction and event cleanup tasks that run only in remote server mode
Static configuration compared to dynamic election
HMS supports two ways to elect the leader:
- Static leader configuration
- You set the metastore.housekeeping.leader.hostname property to the host name of the HMS instance that runs the housekeeping tasks. This approach does not fail over when the leader stops responding, is difficult to configure correctly in hybrid environments, and concentrates all tasks on one instance.
- Dynamic leader election
- HMS instances acquire an exclusive Hive lock to elect the leader. The instance that acquires the lock becomes the leader and runs the housekeeping tasks.
How dynamic leader election works
Dynamic leader election uses a Hive lock as a lease. The HMS instance that acquires the exclusive lock becomes the leader and sends heartbeats to renew the lock before it times out. Each non-leader instance checks the lock periodically and asks HMS to clean up timed-out locks, so that a new leader can be elected if the current leader restarts, is upgraded, or stops responding.
When leadership changes, HMS notifies registered listeners through
takeLeadership and loseLeadership events. The new leader
starts the housekeeping tasks, and the previous leader stops them, which ensures that the
tasks run on only one instance at a time.
HMS records each leader election event in a Hive audit table. You can run a query against this table to identify which instance was the leader at a given time.
Upgrade considerations
Enable dynamic leader election based on how your HMS instances are deployed:
- No leader host name is configured
- Every HMS instance acts as a leader. Enable the feature and housekeeping tasks on all instances.
- A leader host name is configured and all instances are in a single environment
- Enable the feature on all instances. During the upgrade might be at most two leaders, and only one leader remains afterward.
- Instances are distributed across multiple environments
- Enable the feature and housekeeping tasks on all instances in each environment. Cloudera recommends that you upgrade the environment that contains the leader first. If you upgrade another environment first, the cluster can have more than one leader temporarily, which still works but adds load to the metastore database.
