Securing the Key Management Server (KMS)

Cloudera provides two implementations of the Hadoop KMS:
  • Java KeyStore KMS - The default Hadoop KMS included in CDH that uses a file-based Java KeyStore (JKS) for its backing keystore. For parcel-based installations, no additional action is required to install or upgrade the KMS. For package-based installations, you must install additional packages. See Installing and Upgrading Java KeyStore KMS for details. Cloudera strongly recommends not using Java Keystore KMS in production environments.
  • Key Trustee KMS - A custom KMS that uses Cloudera Navigator Key Trustee Server for its backing keystore instead of the file-based Java KeyStore (JKS) used by the default Hadoop KMS. Cloudera strongly recommends using Key Trustee KMS in production environments to improve the security, durability, and scalability of your cryptographic key management. See Cloudera Navigator Data Encryption Overview and Data at Rest Encryption Reference Architecture for more information about the architecture and components involved in encrypting data at rest for production environments. See the following for instructions on installing and upgrading Key Trustee KMS:

This topic contains information on securing the KMS using Kerberos, TLS/SSL communication, and access control lists (ACLs) for operations on encryption keys. Cloudera Manager instructions can be performed for both Key Trustee KMS and Java KeyStore KMS deployments. Command-line instructions apply only to Java KeyStore KMS deployments. Key Trustee KMS is not supported outside of Cloudera Manager. See Installing Key Trustee KMS for more information.

Continue reading:

Enabling Kerberos Authentication for the KMS

Continue reading:

Enabling Kerberos Authentication for the KMS Using Cloudera Manager

Minimum Required Role: Full Administrator

To enable Kerberos for the KMS using Cloudera Manager:
  1. Open the Cloudera Manager Admin Console and go to the KMS service.
  2. Click Configuration.
  3. Set the Authentication Type property to kerberos.
  4. Click Save Changes.
  5. Because Cloudera Manager does not automatically create the principal and keytab file for the KMS, you must run the Generate Credentials command manually. On the top navigation bar, go to Administration > Kerberos > Credentials and click Generate Credentials.
  6. Restart the KMS service.

Kerberos Mapping Rules:

The hadoop.kms.authentication.kerberos.name.rules property can only be changed using the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-site.xml property.

To modify, the value should be copied from the hadoop.security.auth_to_local configuration property, found in the core-site.xml file of the service.

Enabling Kerberos Authentication for the Java KeyStore KMS Using the Command Line

Configure /etc/krb5.conf with information for your KDC server. Create an HTTP principal and keytab file for the KMS.

Configure /etc/hadoop-kms/conf/kms-site.xml with the following properties: 
<property>
    <name>hadoop.kms.authentication.type</name>
    <value>kerberos</value>
</property>

<property>
    <name>hadoop.kms.authentication.kerberos.keytab</name>
    <value>${user.home}/kms.keytab</value>
</property>

<property>
    <name>hadoop.kms.authentication.kerberos.principal</name>
    <value>HTTP/localhost</value>
</property>

<property>
    <name>hadoop.kms.authentication.kerberos.name.rules</name>
    <value>DEFAULT</value>
</property>
Restart the KMS service for the configuration changes to take effect.

Configuring the Java KeyStore KMS Proxyuser Using the Command Line

Each proxyuser must be configured in /etc/hadoop-kms/conf/kms-site.xml using the following properties: 
<property>
    <name>hadoop.kms.proxyuser.#USER#.users</name>
    <value>*</value>
</property>

<property>
    <name>hadoop.kms.proxyuser.#USER#.groups</name>
    <value>*</value>
</property>

<property>
    <name>hadoop.kms.proxyuser.#USER#.hosts</name>
    <value>*</value>
</property>

where #USER# is the username of the proxyuser to be configured.

The hadoop.kms.proxyuser.#USER#.users property indicates the users that can be impersonated. The hadoop.kms.proxyuser.#USER#.groups property indicates the groups to which the users being impersonated must belong. At least one of these properties must be defined. If both are defined, the configured proxyuser can impersonate any user in the users list and any user belonging to a group listed in the groups list.

The hadoop.kms.proxyuser.#USER#.hosts property indicates the host from which the proxyuser can make impersonation requests. "*" means there are no restrictions for the #USER# regarding users, groups, or hosts.

Configuring TLS/SSL for the KMS

Configuring TLS/SSL for the KMS Using Cloudera Manager

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

The steps for configuring and enabling Hadoop TLS/SSL for the KMS are as follows:
  1. Go to the KMS service.
  2. Click Configuration.
  3. In the Search field, type SSL to show the KMS SSL properties (in the Key Management Server Default Group > Security category).
  4. Edit the following SSL properties according to your cluster configuration.
    KMS TLS/SSL Properties
    Property Description
    Enable TLS/SSL for Key Management Server Encrypt communication between clients and Key Management Server using Transport Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).
    Key Management Server TLS/SSL Server JKS Keystore File Location The path to the TLS/SSL keystore file containing the server certificate and private key used for TLS/SSL. Used when Key Management Server is acting as a TLS/SSL server. The keystore must be in JKS format.
    Key Management Server TLS/SSL Server JKS Keystore File Password The password for the Key Management Server JKS keystore file.
    Key Management Server Proxy TLS/SSL Certificate Trust Store File The location on disk of the truststore, in .jks format, used to confirm the authenticity of TLS/SSL servers that Key Management Server Proxy might connect to. This is used when Key Management Server Proxy is the client in a TLS/SSL connection. This truststore must contain the certificates used to sign the services connected to. If this parameter is not provided, the default list of well-known certificate authorities is used instead.
    Key Management Server Proxy TLS/SSL Certificate Trust Store Password The password for the Key Management Server Proxy TLS/SSL Certificate Trust Store File. This password is not required to access the truststore; this field can be left blank. This password provides optional integrity checking of the file. The contents of truststores are certificates, and certificates are public information.
  5. Click Save Changes.
  6. Restart the KMS service.

Configuring TLS/SSL for the Java KeyStore KMS Using the Command Line

To configure KMS to work over HTTPS, set the following properties in the /etc/hadoop-kms/conf/kms_env.sh script:

  • KMS_SSL_KEYSTORE_FILE
  • KMS_SSL_KEYSTORE_PASS
  • KMS_SSL_TRUSTSTORE_FILE
  • KMS_SSL_TRUSTSTORE_PASS

In the /etc/hadoop-kms/tomcat-conf/conf/ directory, replace the server.xml file with the provided ssl-server.xml file.

Create a TLS/SSL certificate for the KMS. As the kms user, use the Java keytool command to create the TLS/SSL certificate: 
$ keytool -genkey -alias tomcat -keyalg RSA

You are asked a series of questions in an interactive prompt. It creates the keystore file, which is named .keystore and located in the kms user home directory. The password you enter for the keystore must match the value of the KMS_SSL_KEYSTORE_PASS environment variable set in the kms-env.sh script in the configuration directory.

The answer to "What is your first and last name?" (CN) must be the hostname of the machine where the KMS will be running.

Configuring KMS Access Control Lists

By default, the KMS Access Control Lists (ACLs) are fully permissive; that is, everyone is allowed to perform all operations. However, restricting HDFS superusers from access to key material is a key design requirement. This prevents a malicious superuser from having access to all the key material and all the encrypted data, and thus being able to decrypt everything.

The KMS supports both whitelist and blacklist ACLs; blacklist entries override whitelist entries. A user or group accessing KMS is first checked for inclusion in the ACL for the requested operation and then checked for exclusion in the blacklist for the operation before access is granted. Add HDFS superusers to the blacklist, and allow everyone else.

The group membership used by ACL entries relies on the configured group mapping mechanism for HDFS. By default, group membership is determined on the local Linux system running the KMS service. If you have configured HDFS to use LDAP for group mapping, the group membership for the ACL entries is determined using the configured LDAP settings. For more information on LDAP-based group membership, see Configuring LDAP Group Mappings.

The ACL syntax for both blacklist and whitelist entries is as follows:
  • Users only: 
    user1,user2,userN
  • Groups only: 
    nobody group1,group2,groupN
  • Users and Groups: 
    user1,user2,userN group1,group2,groupN

Configuring KMS Access Control Lists Using Cloudera Manager

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

  1. Go to the KMS service.
  2. Click Configuration.
  3. In the Search field, type acl to show the Key Management Server Advanced Configuration Snippet (Safety Valve) for kms-acls.xml (in the Key Management Server Default Group category).
  4. Add or edit the ACL properties according to your cluster configuration. See Recommended KMS Access Control List for example ACL entries.
  5. Click Save Changes.
  6. Restart the KMS service.

Configuring Java KeyStore KMS Access Control Lists Using the Command Line

KMS ACLs are defined in the /etc/hadoop-kms/conf/kms-acls.xml configuration file. This file is hot-reloaded when it changes. See Recommended KMS Access Control List for example ACL entries.

Recommended KMS Access Control List

Cloudera recommends the following ACL definition for secure production settings. Replace keyadmin and keyadmingroup with the user and group responsible for maintaining encryption keys.

<configuration>

  <!-- This file is hot-reloaded when it changes -->

  <!--
  KMS ACLs control which users can perform various actions on the KMS.

  This file has 5 sections:
  * Access to specific key operations
  * Blacklists ACLs
  * Default ACLs
  * Whitelist ACLs
  * Key-specific ACLs
-->

<!--
  KMS ACLs that govern access to specific key operations. If access is not
  granted for an operation here, then the operation is forbidden, even if
  a key ACL allows it.

  The ACL value should be either a username or a username and group name
  separated by whitespace.

  An empty value for an ACL indicates that no user is granted access to that
  operation. A value of "*" (for the username or groupname) indicates that
  all users are granted access to that operation. Any operation for which
  there is no ACL will be treated as having an ACL with a value of "*".
-->

<property>
  <name>hadoop.kms.acl.CREATE</name>
  <value>keyadmin keyadmingroup</value>
  <description>
    ACL for create-key operations.
    If the user is not in the GET ACL, the key material is not returned
    as part of the response.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.DELETE</name>
  <value>keyadmin keyadmingroup</value>
  <description>
    ACL for delete-key operations.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.ROLLOVER</name>
  <value>keyadmin keyadmingroup</value>
  <description>
    ACL for rollover-key operations.
    If the user does is not in the GET ACL, the key material is not returned
    as part of the response.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.GET</name>
  <value></value>
  <description>
    ACL for get-key-version and get-current-key operations.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.GET_KEYS</name>
  <value>keyadmin keyadmingroup</value>
  <description>
    ACL for get-keys operations.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.GET_METADATA</name>
  <value>hdfs,keyadmin supergroup,keyadmingroup</value>
  <description>
    ACL for get-key-metadata and get-keys-metadata operations.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.SET_KEY_MATERIAL</name>
  <value></value>
  <description>
    Complementary ACL for CREATE and ROLLOVER operations to allow the client
    to provide the key material when creating or rolling a key.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.GENERATE_EEK</name>
  <value>hdfs supergroup</value>
  <description>
    ACL for generateEncryptedKey CryptoExtension operations.
  </description>
</property>

<property>
  <name>hadoop.kms.acl.DECRYPT_EEK</name>
  <value></value>
  <description>
    ACL for decryptEncryptedKey CryptoExtension operations.
  </description>
</property>

<!--
  KMS blacklists to prevent access to operations. These settings override the
  permissions granted by the KMS ACLs above.

  The blacklist value should be either a username or a username and group name
  separated by whitespace.

  A blank value indicates that no user is blacklisted from the operation. A
  value of "*" (for either the username or groupname) indicates that all users
  are blacklisted from the operation. Any operation for which there is no
  blacklist will be treated as having a blacklist with an empty value.
-->

<!--
  In this template the hdfs user is blacklisted for everything except
  GET_METADATA, GET_KEYS, and GENERATE_EEK. The GET and SET_KEY_MATERIAL
  operations are blacklisted for all users since Hadoop users should not
  need to perform those operations, and access to the key material should
  be as restricted as possible.
-->

<property>
  <name>hadoop.kms.blacklist.CREATE</name>
  <value>hdfs supergroup</value>
</property>

<property>
  <name>hadoop.kms.blacklist.DELETE</name>
  <value>hdfs supergroup</value>
</property>

<property>
  <name>hadoop.kms.blacklist.ROLLOVER</name>
  <value>hdfs supergroup</value>
</property>

<property>
  <name>hadoop.kms.blacklist.GET</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.kms.blacklist.GET_KEYS</name>
  <value></value>
</property>

<property>
  <name>hadoop.kms.blacklist.SET_KEY_MATERIAL</name>
  <value>*</value>
</property>

<property>
  <name>hadoop.kms.blacklist.DECRYPT_EEK</name>
  <value>hdfs supergroup</value>
</property>

<property>
  <name>keytrustee.kms.acl.UNDELETE</name>
  <value></value>
  <description>
    ACL that grants access to the UNDELETE operation on all keys.
    Only used by Key Trustee KMS.
  </description>
</property>

<property>
  <name>keytrustee.kms.acl.PURGE</name>
  <value></value>
  <description>
    ACL that grants access to the PURGE operation on all keys.
    Only used by Key Trustee KMS.
  </description>
</property>

<!--
  Default key ACLs that govern access to key operations for key-operation pairs
  that do not have a specific key ACL already. Specific key ACLs will override
  the default key ACLs

  The ACL value should be either a username or a username and group name
  separated by whitespace.

  An empty value for an ACL indicates that no user is granted access to that
  operation. A value of "*" (for the username or groupname) indicates that
  all users are granted access to that operation. Any operation for which
  there is no ACL will be treated as having an ACL with an empty value.
-->

<property>
  <name>default.key.acl.MANAGEMENT</name>
  <value></value>
  <description>
    Default ACL that grants access to the MANAGEMENT operation on all keys.
  </description>
</property>

<property>
  <name>default.key.acl.GENERATE_EEK</name>
  <value></value>
  <description>
    Default ACL that grants access to the GENERATE_EEK operation on all keys.
  </description>
</property>

<property>
  <name>default.key.acl.DECRYPT_EEK</name>
  <value></value>
  <description>
    Default ACL that grants access to the DECRYPT_EEK operation on all keys.
  </description>
</property>

<property>
  <name>default.key.acl.READ</name>
  <value></value>
  <description>
    Default ACL that grants access to the READ operation on all keys.
  </description>
</property>

<property>
  <name>default.key.acl.MIGRATE</name>
  <value></value>
  <description>
    Default ACL that grants access to the MIGRATE operation on all keys.
  </description>
</property>

<!--
  Whitelist key ACLs that grant access to specific key operations. Any
  permissions granted here will be added to whatever permissions are granted
  by the specific key ACL or the default key ACL. Note that these whitelist
  ACLs grant access to operations on specific keys. If the operations
  themselves are not allowed because of the KMS ACLs/blacklists, then the
  operation will not be permitted, regardless of the whitelist settings.

  The ACL value should be either a username or a username and group name
  separated by whitespace.

  An empty value for an ACL indicates that no user is granted access to that
  operation. A value of "*" (for the username or groupname) indicates that
  all users are granted access to that operation. Any operation for which
  there is no ACL will be treated as having an ACL with an empty value.
-->

<property>
  <name>whitelist.key.acl.MANAGEMENT</name>
  <value>keyadmin keyadmingroup</value>
  <description>
    Whitelist ACL for MANAGEMENT operations for all keys.
  </description>
</property>

<property>
  <name>whitelist.key.acl.READ</name>
  <value>hdfs supergroup</value>
  <description>
    Whitelist ACL for READ operations for all keys.
  </description>
</property>

<property>
  <name>whitelist.key.acl.GENERATE_EEK</name>
  <value>hdfs supergroup</value>
  <description>
    Whitelist ACL for GENERATE_EEK operations for all keys.
  </description>
</property>

<property>
  <name>whitelist.key.acl.DECRYPT_EEK</name>
  <value>keyadmin keyadmingroup</value>
  <description>
    Whitelist ACL for DECRYPT_EEK operations for all keys.
  </description>
</property>

<!--
  Key ACLs that grant access to specific key operations. Any permissions
  granted here are added to whatever permissions are granted by the whitelists.
  The key ACL name should be key.acl.<keyname>.<OPERATION>.

  The ACL value should be either a username or a username and group name
  separated by whitespace.

  An empty value for an ACL indicates that no user is granted access to that
  operation. A value of "*" (for the username or groupname) indicates that
  all users are granted access to that operation. Any key operation for which
  there is no ACL will default to the default ACL for the operation.

  Normally adding users or groups for a specific key and DECRYPT_EEK is
  sufficient to allow access to data protected with HDFS data at rest
  encryption.
-->

<!-- Example acl for a specific key 'mykey'
<property>
  <name>key.acl.mykey.DECRYPT_EEK</name>
  <value>user1 group1</value>
  <description>
    ACL for DECRYPT_EEK operations on key 'mykey'.
  </description>
</property>
-->
</configuration>

Configuring Key Access Control

KMS supports access control for all non-read operations at the key level. All key access operations are classified as:

  • MANAGEMENT - createKey, deleteKey, rolloverNewVersion
  • GENERATE_EEK - generateEncryptedKey, warmUpEncryptedKeys
  • DECRYPT_EEK - decryptEncryptedKey
  • READ - getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata, getCurrentKey
  • ALL - All of the above.
These can be defined in the /etc/hadoop-kms/conf/kms-acls.xml as follows: 
  <property>
    <name>key.acl.testKey1.MANAGEMENT</name>
    <value>*</value>
    <description>
      ACL for create-key, deleteKey and rolloverNewVersion operations.
    </description>
  </property>

  <property>
    <name>key.acl.testKey2.GENERATE_EEK</name>
    <value>*</value>
    <description>
      ACL for generateEncryptedKey operations.
    </description>
  </property>

  <property>
    <name>key.acl.testKey3.DECRYPT_EEK</name>
    <value>*</value>
    <description>
      ACL for decryptEncryptedKey operations.
    </description>
  </property>

  <property>
    <name>key.acl.testKey4.READ</name>
    <value>*</value>
    <description>
      ACL for getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata,
      getCurrentKey operations
    </description>
  </property>

  <property>
    <name>key.acl.testKey5.ALL</name>
    <value>*</value>
    <description>
      ACL for ALL operations.
    </description>
  </property>

  <property>
    <name>default.key.acl.MANAGEMENT</name>
    <value>user1,user2</value>
    <description>
      default ACL for MANAGEMENT operations for all keys that are not
      explicitly defined.
    </description>
  </property>

  <property>
    <name>default.key.acl.GENERATE_EEK</name>
    <value>user1,user2</value>
    <description>
      default ACL for GENERATE_EEK operations for all keys that are not
      explicitly defined.
    </description>
  </property>

  <property>
    <name>default.key.acl.DECRYPT_EEK</name>
    <value>user1,user2</value>
    <description>
      default ACL for DECRYPT_EEK operations for all keys that are not
      explicitly defined.
    </description>
  </property>

  <property>
    <name>default.key.acl.READ</name>
    <value>user1,user2</value>
    <description>
      default ACL for READ operations for all keys that are not
      explicitly defined.
    </description>
  </property>

For all keys for which an ACL has not been explicitly configured, you can configure a default key ACL for a subset of the operation types.

If no ACL is configured for a specific key, and no default ACL is configured for the requested operation, access is denied.

Configuring Java KeyStore KMS Delegation Tokens Using the Command Line

Configure KMS delegation token secret manager using the following properties: 
  <property>
    <name>hadoop.kms.authentication.delegation-token.update-interval.sec</name>
    <value>86400</value>
    <description>
      How often the master key is rotated, in seconds. Default value 1 day.
    </description>
  </property>

  <property>
    <name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
    <value>604800</value>
    <description>
      Maximum lifetime of a delegation token, in seconds. Default value 7 days.
    </description>
  </property>

  <property>
    <name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
    <value>86400</value>
    <description>
      Renewal interval of a delegation token, in seconds. Default value 1 day.
    </description>
  </property>

  <property>
    <name>hadoop.kms.authentication.delegation-token.removal-scan-interval.sec</name>
    <value>3600</value>
    <description>
      Scan interval to remove expired delegation tokens.
    </description>
  </property>