Configuration updates for Spark to work with OzoneFS
After setting up OzoneFS, you can make configuration updates specific to components such as Spark to ensure that they work with Ozone.
Configuration changes for Spark
To run Spark jobs with o3fs
on a secure Kerberos-enabled cluster, ensure
that you assign a valid URI by setting the value of the Spark Client Advanced
Configuration Snippet (Safety Valve) property for the
spark.conf
or the spark-defaults.conf
file through
the Cloudera Manager web UI.
For
example:
spark.yarn.access.hadoopFileSystems=o3fs://bucket1.vol1.securehost1.example.com:9862