Connecting to the Apache Livy Thrift Server

You can connect to the Apache Livy Thrift Server using the Beeline client that is included with Apache Hive.

The Livy Thrift Server is disabled by default. Make sure that you have enabled it before proceeding. For instructions, see Configuring the Livy Thrift Server.

The information required to form a JDBC URL is included in the Livy client configuration file, /etc/livy/conf/livy-client.conf. Make sure that you are on a gateway cluster host managed by Cloudera Manager to ensure that the client configuration is included. You can designate a cluster host as a Livy client by assigning it the Gateway role in the Livy service configuration. For more information about role assignments, see Managing Roles.

  1. On a host designated as a Livy Gateway, view the /etc/livy/conf/livy-client.conf file and note the values of the following configuration parameters:
    livy.uri
    This is the URI for the Livy server. The hostname in this URI is the hostname to use for the Thrift server connection.
    livy.server.thrift.port
    This is the port to use in the JDBC connection URL.
    livy.server.thrift.transport.mode
    This specifies the transport mode (either binary or http).
  2. Determine the JDBC connection URL using the following table:
    Connection ModeJDBC Connection URL
    binary jdbc:hive2://<livy_host>:<port>/<database>
    binary with Kerberos jdbc:hive2://<livy_host>:<port>/<database>;principal=livy/<livy_host>@<REALM>
    binary with Kerberos and TLS jdbc:hive2://<livy_host>:<port>/<database>;principal=livy/<livy_host>@<REALM>;ssl=true;sslTrustStore=<path_to_truststore>
    http jdbc:hive2://<livy_host>:<port>/<database>;transportMode=http;httpPath=cliservice
    http with Kerberos jdbc:hive2://<livy_host>:<port>/<database>;principal=HTTP/<livy_host>@<REALM>;transportMode=http;httpPath=cliservice
    http with Kerberos and TLS jdbc:hive2://<livy_host>:<port>/<database>;principal=HTTP/<livy_host>@<REALM>;ssl=true;sslTrustStore=<path_to_truststore>;\

    trustStorePassword=<truststore_password>;transportMode=http;httpPath=cliservice

  3. Run the beeline command.
  4. At the Beeline prompt, enter !connect <jdbc_url>
    You can set configuration options by appending ?<configuration_list> to the JDBC connection URL. Separate multiple options with a semi-colon (;). Supported options are:
    • livy.session.driverMemory
    • livy.session.driverCores
    • livy.session.executorMemory
    • livy.session.executorCores
    • livy.session.numExecutors
    • livy.session.queue
    • livy.session.name
    • livy.session.heartbeatTimeoutInSecond
    For example:
    beeline> !connect jdbc:hive2://cdp01.example.com:10090/db01?livy.session.driverMemory=4g;livy.session.driverCores=2

Connecting through HAProxy

If the Livy servers run behind HAProxy, connect to the Livy Thrift Server through the load balancer. The connection differs from a direct connection in the following ways:

  • Use the http transport mode. HAProxy routes requests by cookie, which is not possible in binary mode.
  • Use the load balancer FQDN as the host and the port of the HAProxy frontend for the Thrift Server. That frontend routes by the same SRV_ID cookie values as the REST frontend, to the livy.server.thrift.port of each Livy server.
  • With Kerberos, use the HTTP/load-balancer-fqdn principal. Livy accepts this principal when the Livy Load Balancer Address is set.
  • Add httpHeader.Cookie=SRV_ID=srv-id, where srv-id identifies the backend that owns the session, and the livy.session.name option of the session to connect to.

For example:

jdbc:hive2://livy-lb.example.com:<haproxy_thrift_port>/;principal=HTTP/livy-lb.example.com@<REALM>;ssl=true;sslTrustStore=<path_to_truststore>;trustStorePassword=<truststore_password>;transportMode=http;httpHeader.Cookie=SRV_ID=s1;httpPath=cliservice?livy.session.name=<session_name>

To find the SRV_ID value of a session, see Finding the Livy server that owns a session.