Cloudera Docs

Adding independent Livy services with active/passive HA

Each Livy service uses active/passive high availability (not load-balanced active/active). This procedure adds Livy Server roles or a separate Livy service and configures unique ports, session state store paths, and ZooKeeper key prefixes.

Use this procedure when you need more than one independent Livy service in a cluster, each with its own HA configuration. For example, you might run Livy-for-Spark3-1 on port 28998 and add Livy-for-Spark3-2 as a separate service. Adding Livy Server roles to one service creates passive instances that redirect requests to the single active instance; see Livy active/passive high availability.

Prerequisites

  • An existing Livy service (for example, Livy-for-Spark3-1) is running.
  • Administrative access to Cloudera Manager.
  • Required hosts and resources are available in the cluster.
Add Livy roles to the existing Livy service
  1. In Cloudera Manager, select Clusters > Livy-for-Spark3-1 > Instances.

    To enable active/passive HA within an existing Livy service, add Livy Server role instances. One instance is active at a time; the others are passive redirectors.

  2. Click Add Role Instances.
  3. For Livy Server, click Select Hosts, select the desired hosts, and click OK.
  4. Click Continue to deploy the roles.
  5. On the Instances tab, verify that the new Livy Server roles are listed.
  6. Select the status checkboxes for all Livy instances, then from the Actions for Selected menu, select Restart and confirm.
Add a new Livy service
  1. In Cloudera Manager, select Clusters > Cluster 1.
  2. From the Actions menu, select Add Service.
  3. Select Livy-for-Spark3 as the service type and click Continue.
  4. On the dependencies step, select No dependencies, then click Continue.
  5. Assign gateways and hosts for the new Livy service roles.
  6. Click Continue and complete the Add Service wizard. The new Livy service starts automatically.
Resolve port conflicts
  1. If the new Livy service fails to start with an error similar to the following, the default port is already in use by another Livy service: 
    Exception in thread "main" java.io.IOException: Failed to bind to /0.0.0.0:28998
Caused by: java.net.BindException: Address already in use
  2. In Cloudera Manager, select the new Livy service (for example, Livy-for-Spark3-2) and open Configuration.
  3. Search for livy.server.port.
  4. Change the port from the default 28998 to a unique value (for example, 28999).
  5. Click Save Changes, confirm the configuration changes, and click Continue.
  6. When the service restarts with the updated port, click Finish.
Configure unique session state store paths
  1. Each independent Livy service must use a unique HDFS recovery path (livy.server.recovery.state-store.url) to prevent conflicts and enable independent session recovery.
  2. For Livy-for-Spark3-1, open Configuration, search for livy.server.recovery.state-store.url, and verify that the path is unique (for example, /user/livy/recovery_for_livy3).
  3. For Livy-for-Spark3-2, open Configuration, search for livy.server.recovery.state-store.url, and set a different path (for example, /user/livy/recovery_for_livy3_1).
  4. Click Save Changes for each service.
  5. Restart both Livy services to apply the configuration.
Configure unique ZooKeeper key prefixes
  1. Each independent Livy service must use a unique ZooKeeper key prefix (livy.server.zk.key-prefix) so that leader election namespaces do not conflict.
  2. For Livy-for-Spark3-1, open Configuration, search for livy.server.zk.key-prefix, and verify that the prefix is unique (for example, livy3).
  3. For Livy-for-Spark3-2, open Configuration, search for livy.server.zk.key-prefix, and set a different prefix (for example, livy3_1).
  4. Click Save Changes for each service.
  5. Restart both Livy services to apply the configuration.