Using EncryptTool for Flink properties

Cloudera Streaming Analytics offers EncryptTool to further protect your user information and configurations when communicating with Flink using the command line. After generating a master key to the user, you need to manually encrypt the parameters and Flink automatically decrypts the protected values. You also must enable EncryptTool protection in the configuration file for Flink.

In addition to the functionality provided by the vanilla version of Apache Flink, CSA includes a solution to protect for sensitive properties in the configuration file and the dynamic properties. This way passwords in clear text to Flink can be avoided.

There are two actions available through the flink-encrypt-tool command line client to use the EncryptTool:
  • generate-key: generating master key per user. The master key is saved to an arbitrary filesystem location specified by the user, by default to the HDFS home folder of the user. It is the responsibility to protect the privileges of the key, so that it is only accessible by them. EncryptTool assumes that all sensitive properties are protected using the same key in a single configuration file.
  • encrypt: encrypting configuration property. The configuration properties have to be manually encrypted and updated in the configuration file or supplied in encrypted format via dynamic properties.

Flink automatically decrypts the values based on the configuration object during runtime with the privileges of the user that has submitted the Flink job, so the visibility of the key has to be set up accordingly.

Users can override the default key location by setting the following property in the flink-conf.yaml:
In order for the flink-encrypt-tool to use the modified configuration file, one can set the following environment variable: export FLINK_CONF_DIR=/path/to/modified-flink-conf-dir
  1. Generate master key using generate-key action.
  2. Define a secure location for the master key.
  3. Use encrypt action to get the encrypted value for each sensitive key.
  4. Update the configuration.
Once the encryption of each property is performed and saved also set the following flag to indicate that the configuration encryption is enabled: security.encrypt-tool.enabled: true