Hive
Recommendation: Store Hive data in an HDFS path
called /apps/hive
.
Configuring Hive Tables for HDFS Encryption
Before enabling encryption zones, decide whether to store your Hive tables across one zone or multiple encryption zones.
Single Encryption Zone
To configure a single encryption zone for your entire Hive warehouse:
Rename
/apps/hive
to/apps/hive-old
Create an encryption zone at
/apps/hive
distcp
all of the data from/apps/hive-old
to/apps/hive
.
To configure the Hive scratch directory
(hive.exec.scratchdir
) so that it resides inside the encryption
zone:
Set the directory to
/apps/hive/tmp
.Make sure that the permissions for
/apps/hive/tmp
are set to1777
.
Multiple Encryption Zones
To access encrypted databases and tables with different encryption keys, configure multiple encryption zones.
For example, to configure two encrypted tables, ez1.db
and
ez2.db
, in two different encryption zones:
Create two new encryption zones,
/apps/hive/warehouse/ez1.db
and/apps/hive/warehouse/ez2.db
.Load data into Hive tables
ez1.db
andez2.db
as usual, usingLOAD
statements. (For additional considerations, see "Loading Data into an Encrypted Table.")
Loading Data into an Encrypted Table
By design, HDFS-encrypted files cannot be moved or loaded from one encryption zone into another encryption zone, or from an encryption zone into an unencrypted directory. Encrypted files can only be copied.
Within an encryption zone, files can be copied, moved, loaded, and renamed.
Recommendations:
When loading unencrypted data into encrypted tables (e.g.,
LOAD DATA INPATH
), we recommend placing the source data (to be encrypted) into a landing zone within the destination encryption zone.An attempt to load data from one encryption zone into another will result in a copy operation.
Distcp
will be used to speed up the process if the size of the files being copied is higher than the value specified by thehive.exec.copyfile.maxsize
property. The default limit is 32 MB.
Here are two approaches for loading unencrypted data into an encrypted table:
To load unencrypted data into an encrypted table, use the
LOAD DATA ...
statement.If the source data does not reside inside the encryption zone, the
LOAD
statement will result in a copy. If your data is already inside HDFS, though, you can usedistcp
to speed up the copying process.If the data is already inside a Hive table, create a new table with a
LOCATION
inside an encryption zone, as follows:CREATE TABLE encrypted_table [STORED AS] LOCATION ... AS SELECT * FROM <unencrypted_table>
Note The location specified in the
CREATE TABLE
statement must be within an encryption zone. If you create a table that pointsLOCATION
to an unencrypted directory, your data will not be encrypted. You must copy your data to an encryption zone, and then pointLOCATION
to that encryption zone.
If your source data is already encrypted, use the CREATE TABLE
statement. Point LOCATION
to the encrypted source directory where
your data resides:
CREATE TABLE encrypted_table [STORED AS] LOCATION ... AS SELECT
* FROM <encrypted_source_directory>
This is the fastest way to create encrypted tables.
Encrypting Other Hive Directories
LOCALSCRATCHDIR
: The MapJoin optimization in Hive writes HDFS tables to a local directory and then uploads them to distributed cache. To enable encryption, either disable MapJoin (sethive.auto.convert.join
tofalse
) or encrypt thelocal
Hive Scratch directory (hive.exec.local.scratchdir
). Performance note: disabling MapJoin will result in slower join performance.DOWNLOADED_RESOURCES_DIR
: Jars that are added to a user session and stored in HDFS are downloaded tohive.downloaded.resources.dir
. If you want these Jar files to be encrypted, configurehive.downloaded.resources.dir
to be part of an encryption zone. This directory needs to be accessible to the HiveServer2.NodeManager Local Directory List: Hive stores Jars and MapJoin files in the distributed cache, so if you'd like to use MapJoin or encrypt Jars and other resource files, the YARN configuration property NodeManager Local Directory List (
yarn.nodemanager.local-dirs
) must be configured to a set of encrypted local directories on all nodes.Alternatively, to disable MapJoin, set
hive.auto.convert.join
tofalse
.
Additional Changes in Behavior with HDFS-Encrypted Tables
Users reading data from read-only encrypted tables must have access to a temp directory that is encrypted with at least as strong encryption as the table.
By default, temp datas related to HDFS encryption is written to a staging directory identified by the
hive-exec.stagingdir
property created in thehive-site.xml
file? associated with the table folder.Previously, an
INSERT OVERWRITE
on a partitioned table inherited permissions for new data from the existing partition directory. With encryption enabled, permissions are inherited from the table.When using encryption with Trash enabled, table deletion operates differently than the default trash mechanism. For more information see Delete Files from an Encryption Zone.