Configuring external IDE Spark Connect sessions

Learn about how to configure a Spark Connect Session with Cloudera Data Engineering.

Before you create a Spark Connect Session, perform the following steps:

Enable a Cloudera Data Engineering service .
Create a Cloudera Data Engineering Virtual cluster. You must select All Purpose (Tier 2) in the Virtual Cluster option and Spark 3.5.1 as the Spark version.

Perform the following steps on each user's machine:
1. Create the ~/.cde/config.yaml configuration file and add the vcluster-endpoint and cdp-endpoint parameters.
  This allows the client machine to identify a virtual cluster.
  
  note
  The cdp-endpoint value is the same as the Cloudera console URL. From the Cloudera console URL, copy the protocol (https://), the subdomain (console), the second-level domain, and the top-level domain (.com).
  Example: https://console.[***SECOND-LEVEL-DOMAIN***].com
  
  Figure 1. Getting the Cloudera endpoint URL from the Cloudera console URL
  
  For more information, see vcluster-endpoint and cdp-endpoint.
  For example,
```
cdp-endpoint: https://console.cdp.apps.example.com
credentials-file: /Users/user1/.cde/credentials
vcluster-endpoint: https://ffws6v27.cde-c9b822vr.apps.example.com/dex/api/v1
```
2. Create an access key and update the credentials-file parameter in the ~/.cde/config.yaml configuration file with the path where the credentials file is located. This allows the client machine to acquire the short-lived access tokens.
  note
  Access keys configured with the default profile are supported.
  For example,
```
[default]
cdp_access_key_id=571ff....
cdp_private_key=dvbYd....
```
Create a Spark Connect Session using one of the following methods:

note
You can interact with a Spark Connect session that only you have created.
- Using the UI: Create a new session as per Creating Sessions in Cloudera Data Engineering but when you select the session type, select Spark Connect (Tech Preview) from the Type drop-down list.
- Using the CLI: Create a Spark Connect Session by running the following command:
```
cde session create --name [***SPARK-SESSION-NAME***] --type spark-connect
```
  note
  
  To get all the attributes of a cde session command, run the cde session -h command.
On the Cloudera Data Engineering Home page, click Sessions and then select the Spark Connect Session that you have created.
Go to the Connect tab and download the required Cloudera Data Engineering TAR file and PySpark TAR file as displayed on the screen.
note
- The Copy Link option can be used to retrieve a URL and download the client using cURL.
- The PySpark TAR file version must be same as the Virtual Cluster's Spark version.
Create a new Python virtual environment or use your existing one and install the TAR file after activating your Python virtual environment.
```
python3 -m venv cdeconnect
. cdeconnect/bin/activate

pip install [***CDECONNECT TARBALL***]
pip install [***PYSPARK TARBALL***]
```