Enabling Spark 3 engine in Hue

Hue leverages Apache Livy 3 to support Spark SQL queries in Hue on the Apache Spark 3 engine. To enable the Spark 3 engine, install the Spark 3 binaries on your cluster, add Livy for Spark 3 and Spark 3 services, and configure settings for these services using Cloudera Manager.

Log in to Cloudera Manager as an Administrator.
Add the following services by going to Clusters > Actions and clicking Add Service:
- Livy for Spark 3
- Spark 3
important
- While adding the Livy for Spark 3 service to your cluster, you must select Hive as optional dependency on the Select Dependencies page.
- While adding the Spark 3 service to your cluster, you must select an optional dependency containing Kafka on the Select Dependencies page and select the Spark Authentication option on the Review Changes page.

Go to Clusters > HDFS > Configuration and add the following lines in the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml field:

<property>
    <name>hadoop.proxyuser.hue.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hue.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.spark.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.spark.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.livy.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.livy.hosts</name>
    <value>*</value>
</property>

Click Save Changes.
Go to Clusters > Livy for Spark 3 service > Configuration and add the following configurations:
1. Add the hue user in the Admin Users (livy.superusers) field.
2. Go to the HMS Service field and select Hive.
3. Click Save Changes.
Go to Clusters > SPARK_ON_YARN > Configuration > Admin Users, add hue to the list of admin users (spark.history.ui.admin.acls) and click Save Changes.
Go to Clusters > SPARK > Configuration > Admin Users, add hue to the list of admin users (spark.history.ui.admin.acls) and click Save Changes.
Go to Clusters > SPARK 3 > Configuration > Admin Users, add hue to the list of admin users (spark.history.ui.admin.acls) and click Save Changes.
Go to Clusters > Hue > Configuration and enter the following lines in the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini field and click Save Changes:
```
[desktop]
app_blacklist=zookeeper, pig #custom list of blocked apps
[spark]
#This is not a thrift server port
#If this TLS/SSL is enabled then check to see whether the livy url is on https or http and modify the url accordingly.
livy_server_url=http(s)://[***LIVY-FOR-SPARK3-SERVER-HOST***]:[***LIVY-FOR-SPARK3-SERVER-PORT***]   
ssl_cert_ca_verify=false
security_enabled=true
[notebook]
[[interpreters]]
[[[sparksql]]]
name=Spark SQL
interface=livy
```
note
- Ensure that the Spark application is not on the blocked list.
- Set ssl_cert_ca_verify=false if an SSL certificate is not present in Hue's truststore. ssl_cert_ca_verify=true if your environment is a secured environment.
- Set security_enabled=true for Kerberized clusters.
Restart the affected services.

You can now select the Spark SQL dialect on the Hue editor and run Spark queries from Hue.