Enabling Spark Pushdown configuration for projects

Enable portforwarding rules and default spark configurations to allow spark job executors to be scheduled in Yarn in the base cluster.

To enable Spark Pushdown for project users, you must have a Contributor, Operator, or Administrator role in the project.

This is a project-specific setting to enable Spark pushdown for all newly launched workloads in the project. Each project that intends to use the Cloudera Base on premises cluster Yarn for Spark workloads must enable this setting.

Ensure that the Administrator of your workbench has enabled Spark Pushdown at the Site Administraton level. For more details, see Enabling Spark pushdown configuration.

In the Cloudera console, click the Cloudera AI tile.

The Cloudera AI Workbenches page displays.
Select the required workbench.
Click Projects from the left navigation tree.
From the list of projects, select the one to modify.
Select Project Settings to open the Project Settings dashboard.
Select the General tab.
Select the Enable Spark Pushdown option.