Cloudera Docs

Troubleshoot missing Spark 3 files in ShareLib

Learn about how to troubleshoot missing spark-common-utils_2.12-3.5.4.3.5.7191000.0-XX.jar and spark-sql-api_2.12-3.5.4.3.5.7191000.0-XX.jar files from the Spark 3.5.4 sharelib directory.

  1. Sign in to Cloudera Manager.
  2. Click the Oozie service.
  3. In the Actions drop-down menu, select the Update the Spark3 ShareLib option.
  4. Open the terminal on your device and identify the latest sharelib directory on the HDFS by running the following command: 
    hdfs dfs -ls  /user/oozie/share/lib/
  5. Upload the missing files to the latest Spark 3.5.4 sharelib directory by running the following commands: 
    hdfs dfs -put /opt/cloudera/parcels/SPARK3/lib/spark3/jars/spark-common-utils_2.12-3.5.4.3.5.7191000.0-XX.jar /user/oozie/share/lib/lib_XXXXXXXX/spark3/
hdfs dfs -put /opt/cloudera/parcels/SPARK3/lib/spark3/jars/spark-sql-api_2.12-3.5.4.3.5.7191000.0-XX.jar /user/oozie/share/lib/lib_XXXXXXXX/spark3/
  6. Update the ShareLib by running the following command: 
    oozie admin -sharelibupdate