Transparent Encryption Recommendations for Spark
There are various recommendations to consider when configuring HDFS Transparent Encryption for Spark.
- By default, application event logs are stored at
/user/spark/applicationHistory, which can be made into an encryption zone.
- Spark also optionally caches its JAR file at
/user/spark/share/lib(by default), but encrypting this directory is not required.
KMS ACL Configuration for Spark
In the KMS ACL, grant
DECRYPT_EEK permission for the Spark key to the
spark user and any groups that can submit Spark jobs:
<property> <name>key.acl.spark-key.DECRYPT_EEK</name> <value>spark spark-users</value> </property>