Cloudera Docs » 2.3.2 » HDFS Administration Guide
HDFS Administration Guide
Also available as:
PDF
loading table of contents...

​Hive

Recommendation: Store Hive data in an HDFS path called /apps/hive.

​Configuring Hive Tables for HDFS Encryption

Before enabling encryption zones, decide whether to store your Hive tables across one zone or multiple encryption zones.

Single Encryption Zone

To configure a single encryption zone for your entire Hive warehouse:

  1. Rename /apps/hive to /apps/hive-old

  2. Create an encryption zone at /apps/hive

  3. distcp all of the data from /apps/hive-old to /apps/hive.

To configure the Hive scratch directory (hive.exec.scratchdir) so that it resides inside the encryption zone:

  1. Set the directory to /apps/hive/tmp.

  2. Make sure that the permissions for /apps/hive/tmp are set to 1777.

Multiple Encryption Zones

To access encrypted databases and tables with different encryption keys, configure multiple encryption zones.

For example, to configure two encrypted tables, ez1.db and ez2.db, in two different encryption zones:

  1. Create two new encryption zones, /apps/hive/warehouse/ez1.db and /apps/hive/warehouse/ez2.db.

  2. Load data into Hive tables ez1.db and ez2.db as usual, using LOAD statements. (For additional considerations, see "Loading Data into an Encrypted Table.")

​Loading Data into an Encrypted Table

By design, HDFS-encrypted files cannot be moved or loaded from one encryption zone into another encryption zone, or from an encryption zone into an unencrypted directory. Encrypted files can only be copied.

Within an encryption zone, files can be copied, moved, loaded, and renamed.

Recommendations:

  • When loading unencrypted data into encrypted tables (e.g., LOAD DATA INPATH), we recommend placing the source data (to be encrypted) into a landing zone within the destination encryption zone.

  • An attempt to load data from one encryption zone into another will result in a copy operation. Distcp will be used to speed up the process if the size of the files being copied is higher than the value specified by the hive.exec.copyfile.maxsize property. The default limit is 32 MB.

Here are two approaches for loading unencrypted data into an encrypted table:

  • To load unencrypted data into an encrypted table, use the LOAD DATA ... statement.

    If the source data does not reside inside the encryption zone, the LOAD statement will result in a copy. If your data is already inside HDFS, though, you can use distcp to speed up the copying process.

  • If the data is already inside a Hive table, create a new table with a LOCATION inside an encryption zone, as follows:

    CREATE TABLE encrypted_table [STORED AS] LOCATION ... AS SELECT * FROM <unencrypted_table>

    [Note]Note

    The location specified in the CREATE TABLE statement must be within an encryption zone. If you create a table that points LOCATION to an unencrypted directory, your data will not be encrypted. You must copy your data to an encryption zone, and then point LOCATION to that encryption zone.

If your source data is already encrypted, use the CREATE TABLE statement. Point LOCATION to the encrypted source directory where your data resides:

CREATE TABLE encrypted_table [STORED AS] LOCATION ... AS SELECT * FROM <encrypted_source_directory>

This is the fastest way to create encrypted tables.

​Encrypting Other Hive Directories

  • LOCALSCRATCHDIR : The MapJoin optimization in Hive writes HDFS tables to a local directory and then uploads them to distributed cache. To enable encryption, either disable MapJoin (set hive.auto.convert.join to false) or encrypt the local Hive Scratch directory (hive.exec.local.scratchdir). Performance note: disabling MapJoin will result in slower join performance.

  • DOWNLOADED_RESOURCES_DIR: Jars that are added to a user session and stored in HDFS are downloaded to hive.downloaded.resources.dir. If you want these Jar files to be encrypted, configure hive.downloaded.resources.dir to be part of an encryption zone. This directory needs to be accessible to the HiveServer2.

  • NodeManager Local Directory List: Hive stores Jars and MapJoin files in the distributed cache, so if you'd like to use MapJoin or encrypt Jars and other resource files, the YARN configuration property NodeManager Local Directory List (yarn.nodemanager.local-dirs) must be configured to a set of encrypted local directories on all nodes.

    Alternatively, to disable MapJoin, set hive.auto.convert.join to false.

​Additional Changes in Behavior with HDFS-Encrypted Tables

  • Users reading data from read-only encrypted tables must have access to a temp directory that is encrypted with at least as strong encryption as the table.

  • By default, temp datas related to HDFS encryption is written to a staging directory identified by the hive-exec.stagingdir property created in the hive-site.xml file? associated with the table folder.

  • Previously, an INSERT OVERWRITE on a partitioned table inherited permissions for new data from the existing partition directory. With encryption enabled, permissions are inherited from the table.

  • When Trash is enabled, the data file for the table should be moved to the Trash bin -- but if the table is inside an Encryption Zone, this operation is not allowed. For information about deleting data from encryption zones, see Delete Files from an Encryption Zone.

© 2012–2021 by Cloudera, Inc.
Document licensed under the Creative Commons Attribution ShareAlike 4.0 License.
Cloudera.com | Documentation | Support | Community