Spark Authentication

Spark Authentication can be configured using a shared secret configured by the spark.authenticate configuration parameter. The authentication process is essentially a handshake between Spark and the other party to ensure they have the same shared secret and can be allowed to communicate. If the shared secret does not match, they will not be allowed to communicate.

If you are using Spark on YARN, setting spark.authenticate parameter to true will generate and distribute the shared secret to all applications communicating with Spark. For Cloudera Manager deployments, use the following instructions:
  1. Go to the Spark Service > Configuration tab.
  2. In the Search field, type spark authenticate to find the Spark Authentication settings.
  3. Check the checkbox for the Spark Authentication property.
  4. Click Save Changes.