Configuration options for Spark to work with o3fs
After setting up o3fs
, you can make configuration updates specific
to components such as Spark to ensure that they work with Ozone.
To run Spark jobs with o3fs
on a secure Kerberos-enabled cluster, ensure
that you assign a valid URI by setting the value of the Spark Client Advanced
Configuration Snippet (Safety Valve) property for the
spark.conf
or the spark-defaults.conf
file through
the Cloudera Manager web UI.
For
example:
spark.yarn.access.hadoopFileSystems=o3fs://bucket1.vol1.securehost1.example.com:9862