Optimizing Performance for HDFS Transparent Encryption
CDP implements the Advanced Encryption Standard New Instructions (AES-NI),
which provide substantial performance improvements. To get these improvements, you need a recent
version of libcrypto.so
on HDFS and MapReduce client hosts -- that is, any host
from which you originate HDFS or MapReduce requests.
Many OS versions have an older version of the library that does not support AES-NI. The instructions that follow tell you what you need to do for each OS version that CDP supports.
RHEL/CentOS 6.5 or later
The installed version of
libcrypto.so
supports AES-NI,
but you need to install the openssl-devel
package on all
clients:sudo yum install openssl-devel
RHEL/CentOS 6.4 or earlier 6.x versions, or SLES 11
Download and extract a newer version of
libcrypto.so
from a CentOS 6.5 repository and install it on all clients in
/var/lib/hadoop/extra/native/
:- Download the latest version of the
openssl
package. For example:
Thewget http://mirror.centos.org/centos/6/os/x86_64/Packages/openssl-1.0.1e-30.el6.x86_64.rpm
libcrypto.so
file in this package can be used on SLES 11 as well as RHEL/CentOS. - Decompress the files in the package, but do not install
it:
rpm2cpio openssl-1.0.1e-30.el6.x86_64.rpm | cpio -idmv
- If you are using parcels, create the
/var/lib/hadoop/extra/native/
directory:
sudo mkdir -p /var/lib/hadoop/extra/native
- Copy the shared library into
/var/lib/hadoop/extra/native/. Name the target
file
libcrypto.so
, with no suffix at the end, exactly as in the command that follows.sudo cp ./usr/lib64/libcrypto.so.1.0.1e /var/lib/hadoop/extra/native/libcrypto.so
Debian Wheezy
The installed version of
libcrypto.so
supports AES-NI,
but you need to install the libssl-devel
package on all
clients:sudo apt-get install libssl-dev
Ubuntu Precise and Ubuntu Trusty
Install the
libssl-devel
package on all
clients:sudo apt-get install libssl-dev
Testing if encryption optimization works
To verify that a client host is ready to use the AES-NI instruction set
optimization for HDFS encryption at rest, use the following
command:
hadoop checknative
You
should see a response such as the
following:14/12/12 13:48:39 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2
library system-native14/12/12 13:48:39 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
If
you see true
in the openssl
row, Hadoop
has detected the right version of libcrypto.so
and
optimization will work. If you see false
in this row, you
do not have the right version.