Preparing the Environment
HDP supports hardware acceleration with Advanced Encryption Standard New Instructions (AES-NI). Compared with the software implementation of AES, hardware acceleration offers an order of magnitude faster encryption/decryption.
CPU Support for AES NI optimization
To use AES-NI optimization you need CPU and library support, described in the following subsections.AES-NI optimization requires an extended CPU instruction set for AES hardware acceleration.
There are several ways to check for this; for example:
$ cat /proc/cpuinfo | grep aes
Look for output with flags and 'aes'.
Library Support for AES NI optimization
You will need a version of the libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
A version of the libcrypto.so
library with AES-NI support must be
installed on HDFS cluster nodes and MapReduce client hosts -- that is, any host from
which you issue HDFS or MapReduce requests. The following instructions describe how to
install and configure the libcrypto.so
library.
RHEL/CentOS 6.5 or later
On HDP cluster nodes, the installed version of libcrypto.so
supports
AES-NI, but you will need to make sure that the symbolic link exists:
$ sudo ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so
On MapReduce client hosts, install the openssl-devel
package:
$ sudo yum install openssl-devel
Verifying AES NI Support
To verify that a client host is ready to use the AES-NI instruction set optimization for HDFS encryption, use the following command:
hadoop checknative
You should see a response similar to the following:
15/08/12 13:48:39 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
14/12/12 13:48:39 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
If you see true
in the openssl
row, Hadoop has
detected the right version of libcrypto.so
and optimization will
work.
If you see false
in this row, you do not have the correct version.