Enabling Spark configuration for memory efficiency analysis

Memory resource efficiency analysis for Spark jobs is available on Spark engine version 3.3.0 and higher. This feature requires that the spark.executor.processTreeMetrics.enabled configuration be set to true. You can enable this configuration at the individual job level or at the Spark cluster level for all jobs.

1. Log in to Cloudera Manager
2. On the left navigation pane, click Clusters > SPARK3_ON_YARN-1
3. Search for Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf
4. Add spark.executor.processTreeMetrics.enabled=true
5. Click Save Changes
6. Click SPARK3_ON_YARN
7. Click Stale Configuration: Client configuration redeployment needed. The Stale Configurations page opens
8. Click Deploy Client Configuration
9. Click Finish