HDP supports hardware acceleration with Advanced Encryption Standard New Instructions (AES-NI). Compared with the software implementation of AES, hardware acceleration offers an order of magnitude faster encryption/decryption.

To use AES-NI optimization you need CPU and library support, described in the following subsections.

15/08/12 13:48:39 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
14/12/12 13:48:39 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

Create a "master" encryption key for the new encryption zone. Each key will be specific to an encryption zone.

Ranger supports AES/CTR/NoPadding as the cipher suite. (The associated property is listed under HDFS -> Configs in the Advanced hdfs-site list.)

Key size can be 128 or 256 bits.

Recommendation: create a new superuser for key management. In the following examples, superuser encr creates the key. This separates the data access role from the encryption role, strengthening security.

Create an Encryption Key using Ranger KMS (Recommended)

In the Ranger Web UI screen:

To create a new key:

For information about rolling over and deleting keys, see Using the Ranger Key Management Service in the Ranger KMS Administration Guide.

	Warning
	Do not delete an encryption key while it is in use for an encryption zone. This will result in loss of access to data in that zone.

Create an Encryption Key using the CLI

The full syntax of the hadoop key create command is as follows:

[create <keyname> [-cipher <cipher>] 
[-size <size>] 
[-description <description>] 
[-attr <attribute=value>] 
[-provider <provider>] 
[-help]]

Example:

# su - encr

# hadoop key create <key_name> [-size <number-of-bits>]

The default key size is 128 bits. The optional -size parameter supports 256-bit keys, and requires the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File on all hosts in the cluster. For installation information, see the Ambari Security Guide.

Example:

# su - encr

# hadoop key create key1

To verify creation of the key, list the metadata associated with the current user:

# hadoop key list -metadata

For information about rolling over and deleting keys, see Using the Ranger Key Management Service in the Ranger KMS Administration Guide.

	Warning
	Do not delete an encryption key while it is in use for an encryption zone. This will result in loss of access to data in that zone.

Each encryption zone must be defined using an empty directory and an existing encryption key. An encryption zone cannot be created on top of a directory that already contains data.

Recommendation: use one unique key for each encryption zone.

Use the crypto createZone command to create a new encryption zone. The syntax is:

-createZone -keyName <keyName> -path <path>

where:

	Note
	The hdfs service account can create zones, but cannot write data unless the account has sufficient permission. Recommendation: Define a separate user account for the HDFS administrator, and do not provide access to keys for this user in Ranger KMS.

Steps:

To remove an encryption zone, delete the root directory of the zone. For example:

hdfs dfs -rm -R /zone_encr

To copy existing files into an encryption zone, use a tool like distcp.

Note: for separation of administrative roles, do not use the hdfs user to create encryption zones. Instead, designate another administrative account for creating encryption keys and zones. See Creating an HDFS Admin User for more information.

The files will be encrypted using a file-level key generated by the Ranger Key Management Service.

DistCp Considerations

DistCp is commonly used to replicate data between clusters for backup and disaster recovery purposes. This operation is typically performed by the cluster administrator, via an HDFS superuser account.

To retain this workflow when using HDFS encryption, a new virtual path prefix has been introduced, /.reserved/raw/. This virtual path gives super users direct access to the underlying encrypted block data in the file system, allowing super users to distcp data without requiring access to encryption keys. This also avoids the overhead of decrypting and re-encrypting data. The source and destination data will be byte-for-byte identical, which would not be true if the data were re-encrypted with a new EDEK.

Warning

When using /.reserved/raw/ to distcp encrypted data, make sure you preserve extended attributes with the -px flag. This is necessary because encrypted attributes such as the EDEK are exposed through extended attributes; they must be preserved to be able to decrypt the file. For example: sudo -u encr hadoop distcp -px hdfs:/cluster1-namenode:50070/.reserved/raw/apps/enczone hdfs:/cluster2-namenode:50070/.reserved/raw/apps/enczone This means that if the distcp operation is initiated at or above the encryption zone root, it will automatically create a new encryption zone at the destination (if one does not already exist). Recommendation: To avoid potential mishaps, first create identical encryption zones on the destination cluster.

Copying between encrypted and unencrypted locations

By default, distcp compares file system checksums to verify that data was successfully copied to the destination.

When copying between an unencrypted and encrypted location, file system checksums will not match because the underlying block data is different. In this case, specify the -skipcrccheck and -update flags to avoid verifying checksums.

Clients and HDFS applications with sufficient HDFS and Ranger KMS permissions can read and write files from/to an encryption zone.

Overview of the client write process:

Overview of the client read process:

The preceding steps take place through internal interactions between the DFSClient, the NameNode, and Ranger KMS.

In the following example, the /zone_encr directory is an encrypted zone in HDFS.

To verify this, use the crypto -listZones command (as an HDFS administrator). This command lists the root path and the zone key for the encryption zone. For example:

# hdfs crypto -listZones
/zone_encr  key1

Additionally, the /zone_encr directory has been set up for read/write access by the hive user:

# hdfs dfs -ls /
 …
drwxr-x---   - hive   hive            0 2015-01-11 23:12 /zone_encr

The hive user can, therefore, write data to the directory.

The following examples use the copyFromLocal command to move a local file into HDFS.

[hive@blue ~]# hdfs dfs -copyFromLocal web.log /zone_encr
[hive@blue ~]# hdfs dfs -ls /zone_encr
Found 1 items
-rw-r--r--   1 hive hive       1310 2015-01-11 23:28 /zone_encr/web.log

The hive user can read data from the directory, and can verify that the file loaded into HDFS is readable in its unencrypted form.

[hive@blue ~]# hdfs dfs -copyToLocal /zone_encr/web.log read.log
[hive@blue ~]# diff web.log read.log

	Note
	For more information about accessing encrypted files from Hive and other components, see Configuring HDP Services for HDFS Encryption.

Users without access to KMS keys will be able to see file names (via the -ls command), but they will not be able to write data or read from the encrypted zone. For example, the hdfs user lacks sufficient permissions, and cannot access the data in /zone_encr:

[hdfs@blue ~]# hdfs dfs -copyFromLocal install.log /zone_encr
copyFromLocal: Permission denied: user=hdfs, access=EXECUTE, inode="/zone_encr":hive:hive:drwxr-x---

[hdfs@blue ~]# hdfs dfs -copyToLocal /zone_encr/web.log read.log
copyToLocal: Permission denied: user=hdfs, access=EXECUTE, inode="/zone_encr":hive:hive:drwxr-x---

You cannot move data from an Encryption Zone to a global Trash bin outside of the encryption zone.

To delete files from an encryption zone, use one of the following approaches: