Configuring Knox topologies for multiple Livy for Spark 3 services
When a cluster has more than one Livy for Spark 3 service, Knox service discovery adds the Livy Server roles of every service to the Knox topologies. Set the Livy URLs explicitly in Knox Simplified Topology Management, so that Knox exposes only the Livy servers that you intend.
Knox discovers services through Cloudera Manager. For the
LIVY_FOR_SPARK3 service, discovery collects every Livy Server role of
every Livy for Spark 3 service in the cluster and merges them into one service entry in the
following topologies:
cdp-proxycdp-proxy-apicdp-proxy-token
For example, with three Livy for Spark 3 services of three Livy Server roles each, the
LIVY_FOR_SPARK3 entry of each topology lists nine URLs instead of the
URLs of one service. Knox then routes Livy requests to roles that you did not intend to
expose through Knox, and the Livy UI can be empty or show sessions from the wrong
service.
When you set the LIVY_FOR_SPARK3 URLs explicitly in Knox Simplified
Topology Management, the explicit URLs override the URLs that Knox discovers. Typically, you
expose the Livy Server roles of the first Livy for Spark 3 service, which is also the only
service that supports the Livy gateway role.
- The Livy Server hosts and REST ports of the Livy for Spark 3 service that you want to
expose through Knox. The port is the value of
livy.server.portfor that service. - Permission to change the Knox configuration in Cloudera Manager.
Knox routes Livy requests only to the Livy servers that you set, regardless of how many Livy for Spark 3 services the cluster has.
Custom Knox service definitions for Livy
If you maintain a custom Knox service definition for Livy, for example to show a separate Knox Homepage tile for each Livy for Spark 3 service, follow these guidelines:
- In service.xml, include the
metadataelement with the context, description, short description, and type of the service. The Knox Homepage uses this metadata to display the service. - In rewrite.xml, keep the rewrite rules minimal. The Livy UI uses relative paths, and rules that rewrite HTML, CSS, or JavaScript content can break the Sessions and Batches pages.
