Configuring Knox topologies for multiple Livy for Spark 3 services

When a cluster has more than one Livy for Spark 3 service, Knox service discovery adds the Livy Server roles of every service to the Knox topologies. Set the Livy URLs explicitly in Knox Simplified Topology Management, so that Knox exposes only the Livy servers that you intend.

Knox discovers services through Cloudera Manager. For the LIVY_FOR_SPARK3 service, discovery collects every Livy Server role of every Livy for Spark 3 service in the cluster and merges them into one service entry in the following topologies:

  • cdp-proxy
  • cdp-proxy-api
  • cdp-proxy-token

For example, with three Livy for Spark 3 services of three Livy Server roles each, the LIVY_FOR_SPARK3 entry of each topology lists nine URLs instead of the URLs of one service. Knox then routes Livy requests to roles that you did not intend to expose through Knox, and the Livy UI can be empty or show sessions from the wrong service.

When you set the LIVY_FOR_SPARK3 URLs explicitly in Knox Simplified Topology Management, the explicit URLs override the URLs that Knox discovers. Typically, you expose the Livy Server roles of the first Livy for Spark 3 service, which is also the only service that supports the Livy gateway role.

  • The Livy Server hosts and REST ports of the Livy for Spark 3 service that you want to expose through Knox. The port is the value of livy.server.port for that service.
  • Permission to change the Knox configuration in Cloudera Manager.
  1. In Cloudera Manager, select the Knox service and click Configuration.
  2. Search for Knox Simplified Topology Management.
  3. In Knox Simplified Topology Management - cdp-proxy, add one LIVY_FOR_SPARK3 entry for each Livy Server role that you want to expose.
    For example, to expose the two Livy Server roles of the first Livy for Spark 3 service:
    LIVY_FOR_SPARK3:url=https://livy-primary-a.example.com:28998
    LIVY_FOR_SPARK3:url=https://livy-primary-b.example.com:28998
  4. Add the same entries to Knox Simplified Topology Management - cdp-proxy-api and Knox Simplified Topology Management - cdp-proxy-token.
  5. Click Save Changes.
  6. Click the Refresh needed indicator next to the Knox service, and wait until the refresh finishes.
  7. Verify that each topology lists only the URLs that you set.

    Open the topology through the Knox Admin API, for example https://knox.example.com:8443/gateway/admin/api/v1/topologies/cdp-proxy, and check that the LIVY_FOR_SPARK3 service contains one URL for each entry that you added. Repeat the check for cdp-proxy-api and cdp-proxy-token.

  8. Verify that the Livy UI opens through Knox.

    Open the Livy tile on the Knox Homepage, or go to https://knox.example.com:8443/gateway/cdp-proxy/livy_for_spark3/ui, and check that the Sessions and Batches pages list the sessions of the service that you exposed.

Knox routes Livy requests only to the Livy servers that you set, regardless of how many Livy for Spark 3 services the cluster has.

Custom Knox service definitions for Livy

If you maintain a custom Knox service definition for Livy, for example to show a separate Knox Homepage tile for each Livy for Spark 3 service, follow these guidelines:

  • In service.xml, include the metadata element with the context, description, short description, and type of the service. The Knox Homepage uses this metadata to display the service.
  • In rewrite.xml, keep the rewrite rules minimal. The Livy UI uses relative paths, and rules that rewrite HTML, CSS, or JavaScript content can break the Sessions and Batches pages.