Cloudera Docs

Enabling Spark authentication

Spark authentication here refers to an internal authentication mechanism, and not to Kerberos authentication, which is enabled automatically for all Cloudera Data Platform deployments.

Minimum Required Role: Security Administrator (also provided by Full Administrator)

Spark has an internal mechanism that authenticates executors with the driver controlling a given application. This mechanism is enabled using the Cloudera Manager Admin Console, as detailed below. Cluster administrators can enable the spark.authenticate mechanism to authenticate the various processes that support a Spark application.

To enable this feature on the cluster:

  1. In the Cloudera Data Platform (CDP) Management Console, go to Data Hub Clusters.
  2. Find and select the cluster you want to configure.
  3. Click the link for the Cloudera Manager URL.
  4. Go to Clusters > <Cluster Name> > Spark service > Configuration.
  5. Scroll down to the Spark Authentication setting, or search for spark.authenticate to find it.
  6. In the Spark Authentication setting, click the checkbox next to the Spark (Service-Wide) property to activate the setting.
  7. Enter the reason for the change at the bottom of the screen, and then click Save Changes.
  8. Restart YARN:
    1. Select Clusters > YARN.
    2. Select Restart from the Actions drop-down selector.
  9. Re-deploy the client configurations:
    1. Select Clusters > Cluster_name
    2. Select Deploy Client Configurations from the Actions drop-down selector.
  10. Restart stale services.