Using a workload secret in Spark application code

To use the workload secret credentials, you can read the credentials that are mounted into the Spark drivers and executors as read-only files.

The workload secrets are mounted into the Spark drivers and executors in this path:

/etc/dex/secrets/<workload-credential-name>/<credential-key-1> /etc/dex/secrets/<workload-credential-name>/<credential-key-2>

Example workload credentials to use in the application code:

The workload credential is created with the command below.

./cde credential create --name workload-cred-1 --type workload-credential --workload-cred-key db-pass --workload-cred-key aws-secret

The secrets can be read as local files from the paths below within the Spark drivers and executors:

/etc/dex/secrets/workload-cred-1/aws-secret
/etc/dex/secrets/workload-cred-1/db-pass

Example of a PySpark application code to read a secret:

from pyspark.sql import SparkSession
 
spark = SparkSession \
    .builder \
    .appName("Sample DB Connection") \
    .getOrCreate()
 
# read the password from the local file
dbPass=open("/etc/dex/secrets/workload-cred-1/db-pass").read()
 
# use the password in a jdbc connection
jdbcDF= spark.read \
         .jdbc("jdbc:postgresql:dbserver", "schema.tablename",
         properties={"user": "username", "password": dbPass})