Shell action for Spark 3

Learn about how to execute Spark 3's spark3-submit through Oozie's Shell action.

Similar to the support for executing Spark 2's spark-submit through Oozie's Shell action, Cloudera also provides full support for executing Spark 3's spark3-submit through Oozie's Shell action.

As Oozie utilizes delegation tokens instead of Kerberos tickets in its YARN applications, it is recommended to unset the HADOOP_TOKEN_FILE_LOCATION environment variable in your Shell script before executing spark3-submit, if you intend to use spark3-submit without relying on Oozie's default delegation tokens. This is because spark3-submit might not function properly with both delegation tokens and Kerberos tickets. However, to ensure the successful completion of your Shell action, please ensure that you reset the HADOOP_TOKEN_FILE_LOCATION environment variable after the execution of your custom Shell script segment. The following example illustrates how you can accomplish this:

#!/usr/bin/env bash

# By executing the commands within brackets,
# we can ensure that the parent environment remains untouched
(
    unset HADOOP_TOKEN_FILE_LOCATION

    kinit -kt /var/keytabs/user.keytab user

    /usr/bin/spark3-submit --master yarn --deploy-mode cluster \
        create_table_with_data_spark3.py tableUsingSpark3FromShellAction
    /usr/bin/spark3-submit --master yarn --deploy-mode cluster \
        read_created_table.py tableUsingSpark3FromShellAction
)