Cloudera Docs

Enabling Spark pushdown configuration

Administrators must enable the Spark pushdown configuration option to grant users access to Spark pushdown within project configurations.

  1. Enable users to configure projects to use Spark pushdown so that setting up portforwards and configurations for running spark workloads on yarn is an available option.
    1. In the Cloudera console, click the Cloudera AI tile.

      The Cloudera AI Workbenches page displays.

    2. Select Site Administration in the left navigation pane.

      The Site Administration page displays.

    3. Select the Settings tab.
    4. On the Settings page scroll down to the Feature Flags section.
    5. Select the checkbox for the Allow users to enable Spark Pushdown Configuration for Projects option.

      When this option is enabled, users can configure projects to use Spark pushdown that enables setting up portforwards and configurations for running spark workloads on yarn.

      Spark pushdown is enabled.
  2. Enable portforwarding rules and default spark configurations to allow spark job executors to be scheduled in Yarn in the base cluster.

    To enable Spark Pushdown for project users, you must have a Contributor, Operator, or Administrator role in the project.

    This is a project-specific setting to enable Spark pushdown for all newly launched workloads in the project. Each project that intends to use the Cloudera Base on premises cluster Yarn for Spark workloads must enable this setting.

    1. In the Cloudera console, click the Cloudera AI tile.

      The Cloudera AI Workbenches page displays.

    2. Select the required workbench.
    3. Click Projects from the left navigation tree.
    4. From the list of projects, select the one to modify.
    5. Select Project Settings to open the Project Settings dashboard.
    6. Select the General tab.
    7. Select the Enable Spark Pushdown option