Configuration updates for Spark to work with OzoneFS
After setting up OzoneFS, you can make configuration updates specific to components such as Spark to ensure that they work with Ozone.
Configuration changes for Spark
To run Spark jobs with o3fs on a secure Kerberos-enabled cluster, ensure
that you assign a valid URI by setting the value of the Spark Client Advanced
Configuration Snippet (Safety Valve) property for the
spark.conf or the spark-defaults.conf file through
the Cloudera Manager web UI.
For
example:
spark.yarn.access.hadoopFileSystems=o3fs://bucket1.vol1.securehost1.example.com:9862