Configuring external IDE Spark Connect sessions

Learn about how to configure a Spark Connect Session with Cloudera Data Engineering.

Before you create a Spark Connect Session, perform the following steps:

Create a Cloudera Data Engineering Service.
Create a Cloudera Data Engineering Virtual cluster. You must select All Purpose (Tier 2) in the Virtual Cluster option and Spark 3.4.1 as the Spark version.
Initialize the virtual cluster.
Configure Hadoop Authentication.

If you are using an OpenShift cluster, then run the following command:

$ oc -n openshift-ingress-operator annotate ingresscontrollers/default ingress.operator.openshift.io/default-enable-http2=true

Perform the following steps on each user's machine:
1. Create the ~/.cde/config.yaml configuration file and add the vcluster-endpoint and cdp-endpoint parameters.
  This allows the client machine to identify a virtual cluster.
  
  note
  The cdp-endpoint value is the same as the Cloudera console URL. From the Cloudera console URL, copy the protocol (https://), the subdomain (console), the second-level domain, and the top-level domain (.com).
  Example: https://console.[***SECOND-LEVEL-DOMAIN***].com
  
  Figure 1. Getting the Cloudera endpoint URL from the Cloudera console URL
  
  For more information, see vcluster-endpoint and cdp-endpoint.
  For example,
```
cdp-endpoint: https://console.cdp.apps.example.com
credentials-file: /Users/user1/.cde/credentials
vcluster-endpoint: https://ffws6v27.cde-c9b822vr.apps.example.com/dex/api/v1
```
2. Create an access key and update the credentials-file parameter in the ~/.cde/config.yaml configuration file with the path where the credentials file is located. This allows the client machine to acquire the short-lived access tokens.
  note
  Access keys configured with the default profile are supported.
  For example,
```
[default]
cdp_access_key_id=571ff....
cdp_private_key=dvbYd....
```
Create a Spark Connect Session using the UI or the CLI.
- Using the UI: Create a new session as described in Creating Sessions in Cloudera Data Engineering and select Spark Connect (Tech Preview) from the Type drop-down list as the session type.
  Figure 2. Create Session window with Spark Connect selected as session type
  important
  Cloudera Data Engineering 1.5.5 SP3 and higher releases provide the following capabilities:
  - Session timeout extended to up to 90 days for long-running Spark Connect sessions.
  - Dynamic allocation of external IDE Spark Connect session executors.
- Using the CLI: Create a Spark Connect Session by running the following command:
```
cde session create --name [***SPARK-SESSION-NAME***] --type spark-connect
```
  note
  To get all the attributes of a cde session command, run the command with the -h help tag.
On the Cloudera Data Engineering Home page, click Sessions and then select the Spark Connect Session that you have created.
Go to the Connect tab and download the required Cloudera Data Engineering files based on the programming language you want to use to connect.

Figure 3. Connect with Spark Connect tab steps

note
You can copy the URL from the Download link to retrieve the file path and download the client using cURL.
- Python:
  1. Download the required Cloudera Data Engineering TAR file and PySpark 3.4 TAR file.
  2. Create a new Python virtual environment or use your existing one, activate it, and install the TAR files.
```
python3 -m venv cdeconnect
. cdeconnect/bin/activate

pip install [***CDECONNECT TARBALL***]
pip install [***PYSPARK TARBALL***]
```
  3. If you used the self-signed certificates while Initializing the virtual cluster, you must configure the certificates for the Cloudera Data Engineering Virtual Cluster, Spark Connect gRPC server, and the control plane hosts to be trusted. Append all the certificates belonging to those hosts to the Python certifi CA certificates truststore. Usually, the path of the truststore is venv/lib/python3.7/site-packages/certifi/cacert.pem. To trust gRPC connections, export the environment variable:
```
# In bash_profile or terminal
export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=venv/lib/python3.7/site-packages/certifi/cacert.pem

# In a Jupyter notebook use the inbuilt %env magic
%env GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=~/<path-to-cert>
```
    note
    If you do not use self-signed certificates, manually create the cacerts.pem file.
- Java or Scala (Cloudera Data Engineering 1.5.5 SP3 and higher releases):
  1. Download the required Cloudera Data Engineering JVM JAR file. Alternatively, add the JAR file as a Maven dependency in your application pom.xml file using the Cloudera repository.
  2. Include the Cloudera Data Engineering Connect Client JVM JAR file in your application classpath.
    Following is an example command for a Java or Scala JVM application:
```
java --add-opens=java.base/java.nio=ALL-UNNAMED -cp [***PATH-TO-THE-DOWNLOADED-JAR-FILE***] com.cloudera.cde.SparkConnectExample session1
```
  3. If you used the self-signed certificates while Initializing the virtual cluster, add all the certificates to the Java truststore using the keytool utility. For more information, see Importing self-signed certificates to Java truststore.