Using an On-disk Encrypted File Channel
Flume supports on-disk encryption of data on the local disk. To implement this:
- Generate an encryption key to use for the Flume Encrypted File Channel
- Configure on-disk encryption by setting parameters in the flume.conf file
Flume on-disk encryption operates with a maximum strength of 128-bit AES encryption unless the JCE unlimited encryption cryptography policy files are installed. Please see this Oracle document for information about enabling strong cryptography with JDK 1.6: http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.html
Consult your security organization for guidance on the acceptable strength of your encryption keys. Cloudera has tested with AES-128, AES-192, and AES-256.
Generating Encryption Keys
Use the keytool program included in Oracle JDK 1.6 to create the AES encryption keys for use with Flume.
The command to generate a 128-bit key that uses the same password as the key store password is:
keytool -genseckey -alias key-1 -keyalg AES -keysize 128 -validity 9000 \ -keystore test.keystore -storetype jceks \ -storepass keyStorePassword
The command to generate a 128-bit key that uses a different password from that used by the key store is:
keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \ -keysize 128 -validity 9000 -keystore test.keystore \ -storetype jceks -storepass keyStorePassword
The key store and password files can be stored anywhere on the file system; both files should have flume as the owner and 0600 permissions.
Please note that -keysize controls the strength of the AES encryption key, in bits; 128, 192, and 256 are the allowed values.
Configuration
Flume on-disk encryption is enabled by setting parameters in the /etc/flume-ng/conf/flume.conf file.
Basic Configuration
The first example is a basic configuration with an alias called key-0 that uses the same password as the key store:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-0 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0
In the next example, key-0 uses its own password which may be different from the key store password:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-0 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0 agent.channels.ch-0.encryption.keyProvider.keys.key-0.passwordFile = /path/to/key-0.password
Changing Encryption Keys Over Time
To modify the key, modify the configuration as shown below. This example shows how to change the configuration to use key-1 instead of key-0:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-1 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0 key-1
The same scenario except that key-0 and key-1 have their own passwords is shown here:
agent.channels.ch-0.type = file agent.channels.ch-0.capacity = 10000 agent.channels.ch-0.encryption.cipherProvider = AESCTRNOPADDING agent.channels.ch-0.encryption.activeKey = key-1 agent.channels.ch-0.encryption.keyProvider = JCEKSFILE agent.channels.ch-0.encryption.keyProvider.keyStoreFile = /path/to/my.keystore agent.channels.ch-0.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password agent.channels.ch-0.encryption.keyProvider.keys = key-0 key-1 agent.channels.ch-0.encryption.keyProvider.keys.key-0.passwordFile = /path/to/key-0.password agent.channels.ch-0.encryption.keyProvider.keys.key-1.passwordFile = /path/to/key-1.password
Troubleshooting
If the unlimited strength JCE policy files are not installed, an error similar to the following is printed in the flume.log:
07 Sep 2012 23:22:42,232 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider.getCipher:137) - Unable to load key using transformation: AES/CTR/NoPadding; Warning: Maximum allowed key length = 128 with the available JCE security policy files. Have you installed the JCE unlimited strength jurisdiction policy files? java.security.InvalidKeyException: Illegal key size at javax.crypto.Cipher.a(DashoA13*..) at javax.crypto.Cipher.a(DashoA13*..) at javax.crypto.Cipher.a(DashoA13*..) at javax.crypto.Cipher.init(DashoA13*..) at javax.crypto.Cipher.init(DashoA13*..) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider.getCipher(AESCTRNoPaddingProvider.java:120) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider.access$200(AESCTRNoPaddingProvider.java:35) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$AESCTRNoPaddingDecryptor.<init>(AESCTRNoPaddingProvider.java:94) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$AESCTRNoPaddingDecryptor.<init>(AESCTRNoPaddingProvider.java:91) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$DecryptorBuilder.build(AESCTRNoPaddingProvider.java:66) at org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider$DecryptorBuilder.build(AESCTRNoPaddingProvider.java:62) at org.apache.flume.channel.file.encryption.CipherProviderFactory.getDecrypter(CipherProviderFactory.java:47) at org.apache.flume.channel.file.LogFileV3$SequentialReader.<init>(LogFileV3.java:257) at org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:110) at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:258) at org.apache.flume.channel.file.Log.replay(Log.java:339) at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:260) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
<< Supported Sources, Sinks, and Channels | Viewing the Flume Documentation >> | |