Configuring Spark access for Ozone using S3A

You must configure specific properties for client applications such as Spark to access the Ozone data store using S3A, including Kerberos and TLS settings on secure clusters.

You must import the CA certificate to run Ozone S3 Gateway from the S3A filesystem. On Kerberized clusters, also distribute the truststore to Spark executors as described in ozone-config-spark-s3a.html#ozone-spark-s3a-kerberos-tls.
Cloudera recommends ofs:// over s3a:// where applicable. For Kerberized Spark access with ofs://, see Configuration options for Spark to work with Ozone File System (ofs).
You must create an ozone-s3.properties file with the following configuration to run the Spark word count program:
```
spark.hadoop.fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.access.key = <access key>
spark.hadoop.fs.s3a.secret.key = <secret>
spark.hadoop.fs.s3a.endpoint = <Ozone S3 endpoint url>
spark.hadoop.fs.s3a.bucket.probe = 0
spark.hadoop.fs.s3a.change.detection.version.required = false
spark.hadoop.fs.s3a.change.detection.mode = none
spark.hadoop.fs.s3a.path.style.access = true
```
note
In the list of configurations, replace the values of access key and secret from the output of ozone s3 getsecret --om-service-id=<ozone service id> and replace the Ozone S3 endpoint URL with the S3 Gateway URL of the Ozone cluster.

The following procedure explains how you can configure Spark access to Ozone using S3A and run a word count program from the Spark shell.

Create an Ozone bucket.
The following example shows how you can create a bucket named sparkbucket:
```
ozone sh bucket create /s3v/sparkbucket
```

Add data to the bucket.

The following example shows how you can add data to the sparkbucket bucket:

hadoop fs  -Dfs.s3a.bucket.probe=0 -Dfs.s3a.change.detection.version.required=false -Dfs.s3a.change.detection.mode=none -Dfs.s3a.access.key=<accesskey> -Dfs.s3a.secret.key=<secret> -Dfs.s3a.endpoint=<s3 endpoint url> -Dfs.s3a.path.style.access=true -Dfs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -mkdir -p s3a://sparkbucket/input

hadoop fs  -Dfs.s3a.bucket.probe=0 -Dfs.s3a.change.detection.version.required=false -Dfs.s3a.change.detection.mode=none -Dfs.s3a.access.key=<accesskey> -Dfs.s3a.secret.key=<secret> -Dfs.s3a.endpoint=<s3 endpoint url> -Dfs.s3a.path.style.access=true -Dfs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -put /tmp/key1 s3a://sparkbucket/input/key1

Start the Spark shell and wait for the prompt to appear.
```
spark-shell --properties-file <ozone-s3.properties>
```
Create a Resilient Distributed Dataset (RDD) from an Ozone file and enter the specified command on the Spark shell.
```
var lines = sc.textFile("s3a://sparkbucket/input/key1")
```
Convert each record in the file to a word.
```
var words = lines.flatMap(_.split(" "))
```
Convert each word to a key-value pair.
```
var wordsKv = words.map((_, 1))
```
Group each key-value pair by key and perform aggregation on each key.
```
var wordCounts = wordsKv.reduceByKey(_ + _ )
```
Save the results of the grouping and aggregation operations to Ozone.
```
wordCounts.saveAsTextFile("s3a://sparkbucket/output")
```

Exit the spark shell and view the results through S3A.

hadoop fs  -Dfs.s3a.bucket.probe=0 -Dfs.s3a.change.detection.version.required=false -Dfs.s3a.change.detection.mode=none -Dfs.s3a.access.key=<accesskey> -Dfs.s3a.secret.key=<secret> -Dfs.s3a.endpoint=<ozone s3 endpoint url> -Dfs.s3a.path.style.access=true -Dfs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -ls -R s3a://sparkbucket/

hadoop fs  -Dfs.s3a.bucket.probe=0 -Dfs.s3a.change.detection.version.required=false -Dfs.s3a.change.detection.mode=none -Dfs.s3a.access.key=<accesskey> -Dfs.s3a.secret.key=<secret> -Dfs.s3a.endpoint=<ozone s3 endpoint url> -Dfs.s3a.path.style.access=true -Dfs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -cat s3a://sparkbucket/output/part-00000

Kerberized cluster with TLS

On a Kerberos-enabled cluster where the Ozone S3 Gateway uses TLS, Spark driver and executor JVMs must trust the Ozone CA certificate. Node-level truststore updates do not reach YARN or Kubernetes executors. Distribute the truststore and Spark keytab with --files, configure S3A properties, grant filesystem access, and set JVM truststore options on the driver and executors.

Complete the steps in Access Ozone S3 Gateway using the S3A filesystem to obtain the CA certificate or locate the cluster truststore (for Auto-TLS clusters, typically /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks; see Auto-TLS Agent File Locations).
Create an ozone-s3.properties file with the S3A settings shown in the prerequisite section.
Start Spark 3 on YARN with Kerberos credentials, distributed files, and TLS JVM options. For Spark 3, set spark.kerberos.access.hadoopFileSystems to the S3A buckets your job accesses. For Spark 2, use spark.yarn.access.hadoopFileSystems instead.

Example spark3-shell invocation:

spark3-shell \
  --master yarn \
  --principal <spark-principal> \
  --keytab <path-to-spark-keytab> \
  --files <path-to-spark-keytab>,<path-to-truststore.jks>,<ozone-s3.properties> \
  --properties-file <ozone-s3.properties> \
  --conf "spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=<truststore-filename> -Djavax.net.ssl.trustStorePassword=<truststore-password>" \
  --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=<truststore-filename> -Djavax.net.ssl.trustStorePassword=<truststore-password>" \
  --conf "spark.kerberos.access.hadoopFileSystems=s3a://<bucket>"

At the Spark shell prompt, read or write S3A paths as in the word-count procedure above, for example:

val lines = spark.read.textFile("s3a://<bucket>/input/<file>")