Using Livy for Interactive SQL and BI

Use Cases

Customers who used Spark Thrift Server (STS) to interact with Spark from third-party tools, such as Qlik Sense, can use the Livy named sessions feature to migrate to Livy.

The Livy Named Sessions feature has the following key capabilities:

• The ability to reestablish a connection with a long-running session using a persistent name.
• A Livy thrift session that can run indefinitely, treating Spark as an in-memory database.
• The ability to use a temporary view from multiple client sessions.

How It Works

Instead of relying on a transient session ID, you can assign a unique and persistent session name to a Livy session at session creation. The session name acts as a stable alias that clients, such as those using JDBC or ODBC drivers, can use to connect to that specific, long-running session. Session names must be unique and cannot be reused for another active session.

To ensure the session remains available, session timeout checks can be disabled, allowing the session to run indefinitely until explicitly destroyed.

Limitations

The Livy Named Sessions feature has the following limitations:

• Cannot be used in environments with Kerberos enabled for authentication.
• High Availabilty is not supported for JDBC or ODBC Livy Thrift Server sessions

Next Steps

• See Configuring the Livy Thrift Server for instructions on how to enable indefinite sessions.
• See Connecting to the Livy Thrift Server for an example of how to connect to a named session.