Cloudera Docs

Exposing Spark metrics with JMX

You can expose Spark job metrics to with JMX.

  1. In Cloudera Manager, go to Spark 3 > Configuration>.
  2. Find the Safety Valve for spark-defaults.conf and add the following:
    • spark.metrics.conf.*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
    • spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource
  3. Save the changes and deploy the client configurations.
  4. Check the /etc/spark3/conf/spark-defaults.conf on all nodes that they contain the added lines.
  5. Check that newly started Spark applications expose their metrics.

For more information on monitoring Spark jobs, refer to and Monitoring and Instrumentation in the Apache Spark documentation.