Cloudera Docs

Configuration options for Spark to work with Ozone File System (ofs)

After setting up ofs, configure Spark for Kerberos and TLS so driver and executor JVMs can access Ozone on secure clusters.

To run Spark jobs with ofs on a Kerberos-enabled cluster, you must grant Spark access to the Ozone filesystem URI and distribute any TLS truststore that Ozone requires. Updating the default Java truststore on cluster nodes is not sufficient for Spark on YARN or Kubernetes: executors run in separate containers and do not inherit node-level cacerts or jssecacerts settings.

Kerberos filesystem access

Configure Spark to access the Ozone paths your application reads or writes:

  • For Spark 2, set spark.yarn.access.hadoopFileSystems.
  • For Spark 3, set spark.kerberos.access.hadoopFileSystems.

You can set the property using the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-defaults.conf in the Cloudera Manager web UI, or pass it at runtime.

Also set spark.hadoop.fs.ofs.impl to org.apache.hadoop.fs.ozone.RootedOzoneFileSystem and include the Ozone filesystem JAR on the Spark classpath (for example with --jars).

Spark 3 client configuration (/etc/spark3/conf/spark-defaults.conf)
spark.kerberos.access.hadoopFileSystems=ofs://<ozone-service-id>/<volume>/<bucket>/
spark.hadoop.fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
spark3-shell
spark3-shell --conf "spark.kerberos.access.hadoopFileSystems=ofs://<ozone-service-id>/<volume>/<bucket>/" --conf spark.hadoop.fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem ...

TLS truststore distribution

When Ozone uses TLS, distribute the cluster truststore to the driver and every executor and point each JVM to that file:

  1. Copy the truststore to a path the Spark client can read. On clusters that use Cloudera Manager Auto-TLS, the global truststore is typically at /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks (see Auto-TLS Agent File Locations).
  2. Add the truststore and Spark keytab to the job with --files.
  3. Set spark.driver.extraJavaOptions and spark.executor.extraJavaOptions with -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword. Use the truststore filename that --files distributes to each container working directory.

Example: spark3-shell on YARN

The following example starts Spark 3 on YARN with Kerberos credentials, distributes the truststore and keytab, and reads from an ofs:// path. Replace the placeholder values with your environment.

spark3-shell \
  --master yarn \
  --jars <path-to-ozone-filesystem-hadoop3-jar> \
  --principal <spark-principal> \
  --keytab <path-to-spark-keytab> \
  --files <path-to-spark-keytab>,<path-to-truststore.jks> \
  --conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=<truststore-filename> -Djavax.net.ssl.trustStorePassword=<truststore-password>" \
  --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=<truststore-filename> -Djavax.net.ssl.trustStorePassword=<truststore-password>" \
  --conf spark.kerberos.access.hadoopFileSystems=ofs://<ozone-service-id>/<volume>/<bucket>/<path> \
  --conf spark.hadoop.fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem

At the Spark shell prompt, read Ozone data with the configured ofs:// URI, for example:

val lines = spark.read.textFile("ofs://<ozone-service-id>/<volume>/<bucket>/<path>")

For S3A access on Kerberized clusters, see Configuring Spark access for Ozone using S3A.