Beginning with HDP 2.6.0, you can use Livy to submit and interact with Spark jobs, through the Livy REST API or from Apache Zeppelin.

When used from Zeppelin, Livy supports security features and user impersonation. When a Zeppelin user submits a Spark job through Livy, user impersonation propagates user information to Livy so that the job runs under the originating user account.

You install both Livy and Zeppelin on the node on which HDP 2.6 and the Spark client are already installed.

For more information about Livy, see the Spark Component Guide.