Cloudera Docs

Spark, Livy, and Spark Atlas Connector encryption in transit (Cloudera Runtime 7.3.2.10000 SP1)

In service pack 1, use Cloudera Manager to enable encryption on the wire for Spark, Livy for Spark 3, and Spark lineage through Apache Atlas and Kafka.

Cloudera Runtime 7.3.2.10000 SP1 extends encryption in transit for Spark ecosystem services. When you enable the Encrypt all ports feature in Cloudera Manager, related TLS and Spark network encryption settings are applied automatically for supported services. You can also configure Transport Layer Security (TLS) manually per service, including TLS 1.3 when your cluster policy allows it.

Spark and Livy

Spark History Server and the Spark UI use TLS on HTTPS listeners. Livy for Spark 3 uses TLS on the Livy Server REST API. Spark RPC between drivers and executors uses Spark network encryption (AES), which is separate from TLS.

Use the following topics in this guide:

For Encrypt all ports behavior, TLS 1.3 recommendations, and links to wire encryption configuration, see Encryption in transit for Spark and Livy in Cloudera Runtime 7.3.2.10000 SP1.

Spark Atlas Connector (lineage)

The Spark Atlas Connector does not provide a separate TLS stack. Lineage metadata is sent to Apache Atlas and Kafka using the TLS and Kerberos settings for those services. Enable TLS on Atlas and Kafka before you rely on encrypted lineage collection.