Cloudera Docs

Setting custom Spark configurations at workbench-level

Administrators can configure custom Spark settings at the Cloudera AI Workbench level. These configurations will then be applied to all projects and newly launched Spark sessions within that workbench. Non-administrator users can view the applied configurations from Cloudera AI 1.5.5 SP1, but cannot modify them at this level.

Understanding Spark configuration layers and precedence
Spark configuration layers applied to workbenches have the following hierarchy and precedence:
  1. Project-level configurations that are set in the spark-defaults.conf file: These are configurations set within specific project files and have the highest precedence, overriding any workbench-level defaults. For details on setting project-level defaults, see Spark configuration files.
  2. Custom workbench level: These are the custom settings you can configure in this topic, applied by administrators at the workbench level. The configurations in the workbench defaults are applied unless overridden in the custom workbench level.
  3. Workbench defaults level: These are the default Spark configurations applied by the Cloudera AI Workbench system to all Spark sessions. Users can view these defaults, which are displayed in an uneditable textbox.
  4. Base cluster level: If Spark pushdown is enabled the Cloudera Base on premises cluster-level Spark configuration details are visible in an uneditable textbox.
  1. In the Cloudera console, click the Cloudera AI tile.
    The Cloudera AI Workbenches page displays.
  2. Click on the name of the workbench.
    The workbench Home page displays.
  3. Click Site Administration in the left navigation pane.
  4. Select Runtimes tab.
  5. On the Runtimes page, scroll down to find the Spark Configuration section.

    If Spark Pushdown is not enabled, you can see two main text areas under the Spark on Kubernetes tab:

    • The Cloudera AI Workbench Defaults area displays the default Spark configurations applied by Cloudera AI. This section is not editable.

    • The Custom workbench level for spark on Kubernetes workloads area is the editable textbox in which you can enter your custom Spark properties.

    If Spark Pushdown is enabled, select the Spark Pushdown to base cluster tab, where you can see three main text areas:

    • The Base Cluster uneditable textbox, which displays Spark configuration details if Spark pushdown is enabled.

    • The Cloudera AI Workbench Defaults area displays the default Spark configurations applied by Cloudera AI. This section is not editable.

    • The Custom workbench level for spark Pushdown workloads area is the editable textbox in which you can enter your custom Spark properties.

  6. Enter the desired custom Spark properties in the Custom workbench level for spark on Kubernetes workloads textbox. Each property must be on a new line, typically in the key=value format, similar to a spark-defaults.conf file.
  7. Click Save Spark Configuration located at the bottom right of the section.