Spark 3 examples with Python or Java application
This section provides you some examples of Spark 3 with Python and Java applications.
Spark3 with a Java application with Hive2 credentials
<workflow-app name="JavaSpark" xmlns="uri:oozie:workflow:1.0">
<credentials>
<credential name="hive-credential" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://...</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>...</value>
</property>
</credential>
</credentials>
<start to="spark-node-javaspark"/>
<action name="spark-node-javaspark" cred="hive-credential">
<spark3 xmlns="uri:oozie:spark3-action:1.0">
<resource-manager>${resourceManager}</resource-manager>
<name-node>${nameNode}</name-node>
<master>${master}</master>
<mode>${mode}</mode>
<name>JavaSpark-Example</name>
<class>com.company.spark.JavaSpark</class>
<jar>${nameNode}/user/${wf:user()}/javaspark_lib/JavaSparkForOozie.jar</jar>
<arg>${inputFile}</arg>
</spark3>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Workflow failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
</message>
</kill>
<end name="end"/>
</workflow-app>
Spark3 with a Python application with Hive2 credentials
<workflow-app name="PySpark" xmlns="uri:oozie:workflow:1.0">
<credentials>
<credential name="hive-credential" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://...</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>...</value>
</property>
</credential>
</credentials>
<start to="spark-node-pyspark"/>
<action name="spark-node-pyspark" cred="hive-credential">
<spark3 xmlns="uri:oozie:spark3-action:1.0">
<resource-manager>${resourceManager}</resource-manager>
<name-node>${nameNode}</name-node>
<master>${master}</master>
<mode>${mode}</mode>
<name>PySpark-Example</name>
<jar>${pythonScript}</jar>
<arg>${inputFile}</arg>
</spark3>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Workflow failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
</message>
</kill>
<end name="end"/>
</workflow-app>