High availability for Cloudera Lakehouse Optimizer
High availability (HA) for Cloudera Lakehouse Optimizer ensures that the service is running when a Cloudera Lakehouse Optimizer instance fails during planned maintenance. In an HA deployment, you can run multiple Cloudera Lakehouse Optimizer service instances within the same cluster or Data Hub, and the system elects one leader instance to accept requests and modify shared state.
At any time, exactly one Cloudera Lakehouse Optimizer instance is the leader. The leader is the only instance that accepts API requests and performs state-mutating operations. Standby instances wait to take over if the leader fails. Leader election is coordinated through ZooKeeper, and database-level fencing tokens prevent split-brain writes if a former leader tries to continue writing after losing leadership.
How HA protects against split-brain
When multiple Cloudera Lakehouse Optimizer instances run in parallel, HA uses three layers of protection:
- Leader election (ZooKeeper): ZooKeeper elects one leader at a time. If the leader fails, a new leader is elected automatically.
- In-memory pre-check: Before a state-mutating operation, the instance verifies that it is still the leader. If leadership was lost, the request is rejected immediately.
- Database-level fencing: Each leader receives a unique fencing token. Every database write includes this token. The database rejects writes from a stale token, even if a former leader attempts to write after failover.
Operational modes
Cloudera Lakehouse Optimizer can run in the following modes:
| Mode | Description | When it occurs |
|---|---|---|
| Full HA mode | Multiple Cloudera Lakehouse Optimizer instances participate in leader election through ZooKeeper. | HA is enabled and all Cloudera Lakehouse Optimizer instances can reach ZooKeeper at startup. |
| Degraded mode | Cloudera Lakehouse Optimizer started with HA enabled but cannot reach ZooKeeper and runs as a standalone instance without per-write fencing protection. | ZooKeeper is unreachable when Cloudera Lakehouse Optimizer starts, but HA was configured. |
| Standalone mode | A single Cloudera Lakehouse Optimizer instance runs without leader election or fencing. | HA is not configured, or only one Cloudera Lakehouse Optimizer instance is deployed. |
Failover behavior
When the leader fails, ZooKeeper detects the failure, elects a new leader, and the new leader
begins accepting requests. Failover is automatic and typically completes in less than 30
seconds. During failover, passive instances can return HTTP 503 Service
Unavailable or redirect clients to the leader, depending on your configuration.
When clients access Cloudera Lakehouse Optimizer through Apache Knox, the elected leader can update the Knox service descriptor so Knox routes traffic to the current leader automatically.
