Hive Metastore leader election
Leader election lets Hive Metastore (HMS) elect a single instance to run housekeeping tasks, which prevents multiple instances from duplicating the same work and overloading the metastore database.
HMS runs housekeeping tasks such as generating and running compaction tasks, discovering partitions for external tables, cleaning up expired data. When a cluster runs multiple HMS instances, these tasks must run on a single instance. Otherwise, every instance runs the same tasks and duplicates the work.
Duplicated housekeeping work generates a large number of read or write operations against the shared metastore database. In environments that run many concurrent metadata operations, such as replication jobs that use multiple threads to fetch metadata, this additional load can degrade performance for all workloads that share the database.
Tasks that run only on the leader
When leader election is configured, HMS runs the following housekeeping tasks only on the elected leader:
- Compaction tasks, including the initiator, cleaner, and table status updater
- Automatic partition discovery and partition management
- Cleanup of the changed-message (CM) directory
- Transaction and event cleanup tasks that run only in remote server mode
Static configuration compared to dynamic election
HMS supports the following ways to elect the leader:
- Static leader configuration
Assigns a single HMS instance to run housekeeping tasks using the metastore.housekeeping.leader.hostname property. This approach does not support automated failover if the leader stops responding, can be difficult to configure correctly in hybrid environments, and concentrates all tasks on one instance.
- Dynamic leader election
Uses an exclusive Hive lock to elect a leader automatically across active HMS instances. The instance that acquires the lock becomes the leader and runs the housekeeping tasks.
How dynamic leader election works
Dynamic leader election uses a Hive lock as a lease. The HMS instance that acquires the exclusive lock becomes the leader and sends heartbeats to renew the lock before it times out. Each non-leader instance checks the lock periodically and asks HMS to clean up timed-out locks, so that a new leader can be elected if the current leader restarts, is upgraded, or stops responding.
When leadership changes, HMS notifies registered listeners through
takeLeadership and loseLeadership events. The new leader
starts the housekeeping tasks, and the previous leader stops them, which ensures that the
tasks run on only one instance at a time.
HMS records each leader election event in a Hive audit table. You can run a query against this table to identify which instance was the leader at a given time.
Upgrade considerations
Enable dynamic leader election based on the following HMS deployment scenarios:
- No leader hostname is
configured
Enable the feature and housekeeping tasks on all instances. Every HMS instance acts as a leader in this setup.
- A leader hostname is configured and all instances are in a
single environment
Enable the feature on all instances. During the upgrade, the cluster can temporarily have up to two leaders, but only one leader remains after the upgrade completes.
- Instances are distributed across multiple
environments
Enable the feature and housekeeping tasks on all instances in each environment. Cloudera recommends upgrading the environment that contains the leader first. If you upgrade another environment first, the cluster can temporarily have more than one leader, which still functions but increases load on the metastore database.
