Cloudera Docs

Encryption in transit for Spark and Livy in Cloudera Runtime 7.3.2.10000 SP1

In service pack 1, Cloudera Manager can enable Spark and Livy encryption settings automatically when you turn on encryption for all ports. Transport Layer Security (TLS) protects HTTP and REST endpoints; Spark network encryption protects RPC between Spark processes.

Overview

Cloudera Runtime 7.3.2.10000 SP1 aligns Spark and Livy with encryption-on-the-wire requirements. Two mechanisms work together:

  • TLS — Encrypts Spark History Server, Spark UI, and Livy Server HTTPS listeners. Configure TLS protocol versions (including TLS 1.3 where supported) through Cloudera Manager global TLS settings or service TLS properties.
  • Spark network encryption — Encrypts communication between Spark drivers, executors, and the external shuffle service when authentication is enabled. This uses AES encryption, not TLS.

When you enable the Encrypt all ports feature in Cloudera Manager, Cloudera Manager sets related Spark properties automatically. For a full list of behavioral changes, see Behavioral Changes in Spark and Behavioral Changes in Livy.

Encrypt all ports

Several Spark security parameters are disabled by default. When Encrypt all ports is active, Cloudera Manager enables them, including:

  • spark.authenticate and spark.network.crypto.enabled for RPC encryption
  • spark.ssl.historyServer.disableHttpPort and spark.ssl.ui.disableHttpPort when TLS is enabled for HTTPS-only Jetty listeners
  • spark.cloudera.auto.disable.insecure.spark.ui.enabled to disable the Spark UI when SSL is not active
  • spark.cloudera.ui.yarn.autoCertificate.clusterMode.enabled for YARN ApplicationMaster certificates in cluster deploy mode (not client mode)

For manual configuration of network encryption, see Configuring Spark for Wire Encryption.

TLS 1.3

Cloudera recommends TLS 1.3 when your JDK and Cloudera Manager TLS policy allow it. Use TLS 1.2 only as an exception with approved cipher suites. For Spark History Server TLS, see Configuring TLS/SSL encryption manually for Spark. For Livy Server TLS, see Configuring TLS/SSL encryption manually for Livy.

Lineage through the Spark Atlas Connector uses TLS on Apache Atlas and Kafka connections; the connector does not provide a separate TLS stack. Configure Atlas TLS as described in Configuring TLS/SSL for Apache Atlas.

FIPS 140-3

FIPS 140-3 support and BCFKS trust store settings are documented separately. See Spark and Livy on FIPS 140-3 clusters.