Using an On-disk Encrypted File Channel
Flume supports on-disk encryption of data on the local disk. To implement this:
- Generate an encryption key to use for the Flume Encrypted File Channel
- Configure on-disk encryption by setting parameters in the flume.conf file
Generating Encryption Keys
Use the keytool program included with the Oracle JDK to create the AES encryption keys for use with Flume.
The command to generate a 128-bit key that uses the same password as the key store password is:
keytool -genseckey -alias key-1 -keyalg AES -keysize 128 -validity 9000 \ -keystore test.keystore -storetype jceks \ -storepass keyStorePassword
The command to generate a 128-bit key that uses a different password from that used by the key store is:
keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \ -keysize 128 -validity 9000 -keystore test.keystore \ -storetype jceks -storepass keyStorePassword
The key store and password files can be stored anywhere on the file system; both files should have flume as the owner and 0600 permissions.
Please note that -keysize controls the strength of the AES encryption key, in bits; 128, 192, and 256 are the allowed values.
Configuration
Flume on-disk encryption is enabled by setting parameters in the /etc/flume-ng/conf/flume.conf file.
Basic Configuration
The first example is a basic configuration with an alias called key-0 that uses the same password as the key store:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-0 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0
In the next example, key-0 uses its own password which may be different from the key store password:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-0 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0 agent.channels.ch-0.encryption.keyProvider.keys.key-0.passwordFile = /path/to/key-0.password
Changing Encryption Keys Over Time
To modify the key, modify the configuration as shown below. This example shows how to change the configuration to use key-1 instead of key-0:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-1 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0 key-1
The same scenario except that key-0 and key-1 have their own passwords is shown here:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-1 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0 key-1 agent.channels.ch-0.encryption.keyProvider.keys.key-0.passwordFile = /path/to/key-0.password agent.channels.ch-0.encryption.keyProvider.keys.key-1.passwordFile = /path/to/key-1.password
Troubleshooting
If the unlimited strength JCE policy files are not installed, an error similar to the following is printed in the flume.log:
07 Sep 2012 23:22:42,232 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider.getCipher:137) - Unable to load key using transformation: AES/CTR/NoPadding; Warning: Maximum allowed key length = 128 with the available JCE security policy files. Have you installed the JCE unlimited strength jurisdiction policy files? java.security.InvalidKeyException: Illegal key size at javax.crypto.Cipher.a(DashoA13*..) at javax.crypto.Cipher.a(DashoA13*..) at javax.crypto.Cipher.a(DashoA13*..) at javax.crypto.Cipher.init(DashoA13*..) at javax.crypto.Cipher.init(DashoA13*..) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider.getCipher(AESCTRNoPaddingProvider.java:120) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider.access$200(AESCTRNoPaddingProvider.java:35) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$AESCTRNoPaddingDecryptor.<init>(AESCTRNoPaddingProvider.java:94) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$AESCTRNoPaddingDecryptor.<init>(AESCTRNoPaddingProvider.java:91) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$DecryptorBuilder.build(AESCTRNoPaddingProvider.java:66) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$DecryptorBuilder.build(AESCTRNoPaddingProvider.java:62) at org.apache.flume.channel.file.encryption.CipherProviderFactory.getDecrypter(CipherProviderFactory.java:47) at org.apache.flume.channel.file.LogFileV3$SequentialReader.<init>(LogFileV3.java:257) at org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:110) at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:258) at org.apache.flume.channel.file.Log.replay(Log.java:339) at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:260) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)