You see how to set up a staging location for temporarily storing Hive files that
users need to read from Spark.
In the following procedure, you set up a staging location for temporarily storing
Hive files that users read from Spark. First, as the file system administrator, you
grant -wt
access on the parent directory to set up the following access
and restrictions:
- Users can traverse the directories to reach the staging location and create
their own subdirectories.
- Users cannot delete or rename the directories of others.
Next, as administrator, you also set up Ranger policies on files and directories in
the staging location. You set up file system permissions. These policies secure managed,
ACID, as well as external tables. Finally, you launch the spark shell, configure secure
access mode, and create external tables in staging location.
- You obtained privileges to set user permissions on the staging directory and
files.
-
In Cloudera Manager, click , search for
hive.server2.enable.doAs
, and, if
necessary, uncheck to disable.
This step is required by Hive for managing ACID tables.
-
On the file system of your CDP cluster, grant
-wt
access to
users.
For example:
sudo -u hdfs hdfs dfs -chmod 1703 /tmp/staging
sudo -u hdfs hdfs dfs -chmod 1703 /tmp/staging/hwc
sudo -u hdfs hdfs dfs -chmod 1703 /tmp/staging/hwc/*
-
In Ranger, log into the Ranger Admin UI, and in Service Manager locate HDFS,
and then click the service name, for example,
cm_hdfs.cm_hdfs.
-
Click Add New Policy, and specify the deepest path to
files in the staging directory.
For example, in Resource Path, enter the following path:
/tmp/staging/hwc/*/{USER}
-
In Allow Conditions, in Select User, select {USER} and
read, write, and execute permissions.
The Spark session users must have access to the staging location. This action
sets a single Ranger policy for all users.
-
Click Add New Policy again to add another HDFS policy
that grants the hive user additional privileges: read,
write, and execute on the staging path.
For example, in Resource Path, enter the following staging path:
/tmp/staging/hwc
-
In Allow Conditions, in Select User, select hive and
read, write, execute permissions.