Using a workload secret in Spark application code
To use the workload secret credentials, you can read the credentials that are mounted into the Spark drivers and executors as read-only files.
The workload secrets are mounted into the Spark drivers and executors in this path:
/etc/dex/secrets/<workload-credential-name>/<credential-key-1> /etc/dex/secrets/<workload-credential-name>/<credential-key-2>
Example workload credentials to use in the application code:
The workload credential is created with the command below.
./cde credential create --name workload-cred-1 --type workload-credential --workload-cred-key db-pass --workload-cred-key aws-secret
The secrets can be read as local files from the paths below within the Spark drivers and
executors:
/etc/dex/secrets/workload-cred-1/aws-secret /etc/dex/secrets/workload-cred-1/db-pass
Example of a PySpark application code to read a secret:
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("Sample DB Connection") \
.getOrCreate()
# read the password from the local file
dbPass=open("/etc/dex/secrets/workload-cred-1/db-pass").read()
# use the password in a jdbc connection
jdbcDF= spark.read \
.jdbc("jdbc:postgresql:dbserver", "schema.tablename",
properties={"user": "username", "password": dbPass})
