Configuring external IDE Spark Connect sessions

Learn about how to configure a Spark Connect Session with Cloudera Data Engineering.

Before you create a Spark Connect Session, perform the following steps:

Enable a Cloudera Data Engineering service .
Create a Cloudera Data Engineering Virtual cluster. You must select All Purpose (Tier 2) in the Virtual Cluster option and Spark 3.5.1 as the Spark version.

Perform the following steps on each user's machine:
1. Create the ~/.cde/config.yaml configuration file and add the vcluster-endpoint and cdp-endpoint parameters.
  This allows the client machine to identify a virtual cluster.
  
  note
  The cdp-endpoint value is the same as the Cloudera console URL. From the Cloudera console URL, copy the protocol (https://), the subdomain (console), the second-level domain, and the top-level domain (.com).
  Example: https://console.[***SECOND-LEVEL-DOMAIN***].com
  
  Figure 1. Getting the Cloudera endpoint URL from the Cloudera console URL
  
  For more information, see vcluster-endpoint and cdp-endpoint.
  For example,
```
cdp-endpoint: https://console.cdp.apps.example.com
credentials-file: /Users/user1/.cde/credentials
vcluster-endpoint: https://ffws6v27.cde-c9b822vr.apps.example.com/dex/api/v1
```
2. Create an access key and update the credentials-file parameter in the ~/.cde/config.yaml configuration file with the path where the credentials file is located. This allows the client machine to acquire the short-lived access tokens.
  note
  Access keys configured with the default profile are supported.
  For example,
```
[default]
cdp_access_key_id=571ff....
cdp_private_key=dvbYd....
```
Create a Spark Connect Session using the UI or the CLI.

note
You can interact with a Spark Connect session that only you have created.
- Using the UI: Create a new session as described in Creating Sessions in Cloudera Data Engineering and select Spark Connect (Tech Preview) from the Type drop-down list as the session type.
  Figure 2. Create Session window with Spark Connect selected as session type
  important
  Cloudera Data Engineering provide the following capabilities:
  - Session timeout extended to up to 90 days for long-running Spark Connect sessions.
  - Dynamic allocation of external IDE Spark Connect session executors.
- Using the CLI: Create a Spark Connect Session by running the following command:
```
cde session create --name [***SPARK-SESSION-NAME***] --type spark-connect
```
  note
  To get all the attributes of a cde session command, run the command with the -h help tag.
On the Cloudera Data Engineering Home page, click Sessions and then select the Spark Connect Session that you have created.
Go to the Connect tab and download the required Cloudera Data Engineering files based on the programming language you want to use to connect.

Figure 3. Connect with Spark Connect tab steps
note
- The Copy Link option can be used to retrieve a URL and download the client using cURL.
- The PySpark TAR file version must be same as the Virtual Cluster's Spark version.
- Python:
  1. Download the required Cloudera Data Engineering TAR file and PySpark TAR file.
  2. Create a new Python virtual environment or use your existing one, activate it, and install the TAR files.
```
python3 -m venv cdeconnect
. cdeconnect/bin/activate

pip install [***CDECONNECT TARBALL***]
pip install [***PYSPARK TARBALL***]
```
- Java or Scala:
  1. Download the required Cloudera Data Engineering JVM JAR file. Alternatively, add the JAR file as a Maven dependency in your application pom.xml file using the Cloudera repository.
  2. Include the Cloudera Data Engineering Connect Client JVM JAR file in your application classpath.
    Following is an example command for a Java or Scala JVM application:
```
java --add-opens=java.base/java.nio=ALL-UNNAMED -cp [***PATH-TO-THE-DOWNLOADED-JAR-FILE***] com.cloudera.cde.SparkConnectExample session1
```