Cloudera Docs

Configuring high availability for Cloudera Lakehouse Optimizer

You can configure high availability (HA) for Cloudera Lakehouse Optimizer in Cloudera Manager by deploying multiple Cloudera Lakehouse Optimizer service instances and setting HA properties on each instance.

Before you configure HA, ensure the following requirements are met:
  • ZooKeeper is running and healthy on at least three hosts in the cluster or Data Hub.
  • Multiple Cloudera Lakehouse Optimizer service instances deployed on separate hosts within the same cluster or Data Hub.
  • All Cloudera Lakehouse Optimizer instances connect to the same shared database (PostgreSQL or MySQL).
  • All Cloudera Lakehouse Optimizer instances can reach ZooKeeper and the shared database.
  • Apache Knox is configured if you use Knox for REST API access.
Configure HA on every Cloudera Lakehouse Optimizer instance in the cluster or Data Hub. All instances must use the same ZooKeeper connection string, leader latch path, and shared database settings.

To configure HA on a Cloudera Lakehouse Optimizer instance:

  1. On Cloudera on cloud, provision a Cloudera Lakehouse Optimizer Data Hub and deploy multiple service instances.
    1. Provision one Cloudera Lakehouse Optimizer Data Hub for your environment.
      For instructions, see Provisioning the Lakehouse Optimizer Data Hub.
    2. In Cloudera Manager > Clusters > [***CLOUDERA LAKEHOUSE OPTIMIZER***], add Cloudera Lakehouse Optimizer roles to additional hosts so that at least two instances are running.
  2. Navigate to Configuration > conf/dlm-client.properties_role_safety_valve.
    1. Set dlm.ha.enabled to true.
    2. Set dlm.ha.zookeeper.connect to the ZooKeeper connection string for your cluster. For example, zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181.
    3. Set dlm.ha.zookeeper.latch.path to the ZooKeeper path for leader election. For example, /dlm/leader.
    If you access Cloudera Lakehouse Optimizer through Apache Knox, configure the Knox discovery properties so the elected leader updates the Knox service descriptor. For the complete list of HA properties, see Cloudera Lakehouse Optimizer high availability configuration properties.
  3. Optionally, set dlm.ha.advertise.address on each instance if the auto-detected API URL is not reachable by other Cloudera Lakehouse Optimizer instances or API clients.
  4. Restart the Cloudera Lakehouse Optimizer service on all instances.
  5. Verify HA status using the GET /config/health API with scope=full.
    For instructions, see Verifying service health and viewing default Lakehouse Optimizer policy.
    When HA is enabled, the health check response includes a High Availability entry that reports whether the instance is the leader or a standby.

After you configure HA:

  • Test failover by stopping the leader instance and verifying that a standby instance becomes the leader.
  • Monitor Cloudera Lakehouse Optimizer logs for leader election messages.
  • Restore ZooKeeper and restart Cloudera Lakehouse Optimizer if an instance starts in degraded mode.