Cloudera Docs

Setting custom Spark configurations at workbench-level

Administrators can configure custom Spark settings at the Cloudera AI Workbench level. These configurations will then be applied to all projects and newly launched Spark sessions within that workbench. Non-administrator users can view the applied configurations, but cannot modify them at this level.

Understanding Spark configuration layers and precedence
Spark configuration layers applied to workbenches have the following hierarchy and precedence:
  1. Project-level configurations that are set in the spark-defaults.conf file: These are configurations set within specific project files and have the highest precedence, overriding any workbench-level defaults. For details on setting project-level defaults, see Spark configuration files.
  2. Custom workbench level: These are the custom settings you can configure in this topic, applied by administrators at the workbench level. The configurations in the workbench defaults are applied unless overridden in the custom workbench level.
  3. Workbench defaults level: These are the default Spark configurations applied by the Cloudera AI Workbench system to all Spark sessions. Users can view these defaults, which are displayed in an uneditable textbox.
  1. In the Cloudera console, click the Cloudera AI tile.
    The Cloudera AI Workbenches page displays.
  2. Click on the name of the workbench.
    The workbench Home page displays.
  3. Click Site Administration in the left navigation pane.
  4. Select Runtimes tab.
  5. On the Runtimes page, scroll down to find the Spark Configuration section.

    You see two main text areas. The Cloudera AI Workbench Defaults area displays the default Spark configurations applied by Cloudera AI. This section is not editable.

    The Custom workbench level for spark on Kubernetes workloads area is the editable textbox in which you can enter your custom Spark properties.

  6. Enter the desired custom Spark properties in the Custom workbench level for spark on Kubernetes workloads textbox. Each property must be on a new line, typically in the key=value format, similar to a spark-defaults.conf file.
  7. Click Save Spark Configuration located at the bottom right of the section.