Configuring external IDE Spark Connect sessions

Learn about how to configure a Spark Connect Session with Cloudera Data Engineering.

Before you create a Spark Connect Session, perform the following steps:

Create a Cloudera Data Engineering Service.
Create a Cloudera Data Engineering Virtual cluster. You must select All Purpose (Tier 2) in the Virtual Cluster option and Spark 3.4.1 as the Spark version.
Initialize the virtual cluster.
Initialize users in virtual clusters.

If you are using an OpenShift cluster, then run the following command:

$ oc -n openshift-ingress-operator annotate ingresscontrollers/default ingress.operator.openshift.io/default-enable-http2=true

Perform the following steps on each user's machine:
1. Create the ~/.cde/config.yaml configuration file and add the vcluster-endpoint and cdp-endpoint parameters. This allows the client machine to identify a virtual cluster. For more information, see vcluster-endpoint and cdp-endpoint.
  For example,
```
cdp-endpoint: https://console-cdp.apps.example.com
credentials-file: /Users/user1/.cde/credentials
vcluster-endpoint: https://ffws6v27.cde-c9b822vr.apps.example.com/dex/api/v1
```
2. Create an access key and update the credentials-file parameter in the ~/.cde/config.yaml configuration file with the path where the credentials file is located. This allows the client machine to acquire the short-lived access tokens.
  note
  Access keys configured with the default profile are supported.
  For example,
```
[default]
cdp_access_key_id=571ff....
cdp_private_key=dvbYd....
```
Create a Spark Connect Session using one of the following methods:
- Using the UI: Create a new session as per Creating Sessions in Cloudera Data Engineering but when you select the session type, select Spark Connect (Tech Preview) from the Type drop-down list.
- Using the CLI: Create a Spark Connect Session by running the following command:
```
cde session create --name [***SPARK-SESSION-NAME***] --type spark-connect
```
  note
  
  To get all the attributes of a cde session command, run the cde session -h command.
On the Cloudera Data Engineering Home page, click Sessions and then select the Spark Connect Session that you have created.
Go to the Connect tab and download the required Cloudera Data Engineering TAR file and PySpark 3.4 TAR file as displayed on the screen.

note
The Copy Link option can be used to retrieve a URL and download the client using cURL.
Create a new Python virtual environment or use your existing one and install the Tar file after activating your Python virtual environment.
```
python3 -m venv cdeconnect
. cdeconnect/bin/activate

pip install [***cdeconnect tarball***]
pip install [***pyspark tarball***]
```
If you have used the self-signed certificates while Initializing the virtual cluster, then you must configure the certificates for the Cloudera Data Engineering Virtual Cluster, Spark Connect gRPC server, and the control plane hosts to be trusted. Append all the certificates belonging to those hosts to the Python "certifi cacerts ca" truststore. Usually, the path of the truststore is venv/lib/python3.7/site-packages/certifi/cacert.pem. For trusting gRPC connections, export the following variable:
note
If you do not have self-signed certificates, manually create cacerts.pem file.
```
# In bash_profile or terminal
export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=venv/lib/python3.7/site-packages/certifi/cacert.pem

# In a Jupyter notebook use the inbuilt %env magic
%env GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=~/<path-to-cert>
```