In Ambari, go to Services/Hive/Configs, and check the value of hive.metastore.warehouse.dir to determine the location of the Hive warehouse, /apps/hive/warehouse by default.

On any node in the cluster, as the HDFS superuser, enable snapshots. For example:

$ sudo su - hdfs

$ hdfs dfsadmin -allowSnapshot /apps/hive/warehouse

Output is:

Allowing snaphot on /apps/hive/warehouse succeeded

Create a snapshot of the Hive warehouse. For example:

$ hdfs dfs -createSnapshot /apps/hive/warehouse

Output includes the name and location of the snapshot:

Created snapshot /apps/hive/warehouse/.snapshot/s20181204-164645.898

Start Hive as a user who has SELECT privileges on the tables. For example:

$ beeline

beeline> !connect jdbc:hive2://

Enter username for jdbc:hive2://: hive

Enter password for jdbc:hive2://: *********

Output is, for example:

Connected to: Apache Hive (version 1.2.1000.2.6.5.0-292)

Driver: Hive JDBC (version 1.2.1000.2.6.5.0-292)

Identify all tables outside /apps/hive/warehouse/. For example:

hive> USE my_database;

hive> SHOW TABLES;

Determine the location of each table using the DESCRIBE command. For example:

hive> DESCRIBE FORMATTED my_table partition (dt=’20181130);

Create a snapshot of the directory shown in the location section of the output.

Repeat steps 5-7 for each database and its tables outside /apps/hive/warehouse/.

SSH into the host running the Hive Metastore.

Change to the /tmp directory.

Execute the following command to download the pre-upgrade tool JAR:

$ wget http://repo.hortonworks.com/content/repositories/releases/org/apache/hive/hive-pre-upgrade/3.1.0.3.1.4.17-1/hive-pre-upgrade-3.1.0.3.1.4.17-1.jar

Become the Hive service user. For example, run the following su command on Linux:

$ sudo su - hive

In a Kerberized cluster, run kinit to get a Kerberos ticket. For example:

$ kinit -kt /etc/security/keytabs/hive.service.keytab hive/`hostname -f`

Set -Djavax.security.auth.useSubjectCredsOnly=false in a Kerberized environment if, after running kinit, you see the following error:

org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt

Perform the procedure for compacting Hive tables below.

Log in as the hive user.

For example: $ sudo su - hive

Export the JAVA_HOME environment variable if necessary.

For example: $ export JAVA_HOME=[ path to your installed JDK ]

Set STACK_VERSION to the HDP version you are running. For example:

$ export STACK_VERSION=`hdp-select status hive-server2 | awk '{ print $3; }'`

Run the pre-upgrade tool command.

$ $JAVA_HOME/bin/java -cp /usr/hdp/$STACK_VERSION/hive/lib/derby-10.10.2.0.jar:/usr/hdp/$STACK_VERSION/hive/lib/*:/usr/hdp/$STACK_VERSION/hadoop/*:/usr/hdp/$STACK_VERSION/hadoop/lib/*:/usr/hdp/$STACK_VERSION/hadoop-mapreduce/*:/usr/hdp/$STACK_VERSION/hadoop-mapreduce/lib/*:/usr/hdp/$STACK_VERSION/hadoop-hdfs/*:/usr/hdp/$STACK_VERSION/hadoop-hdfs/lib/*:/usr/hdp/$STACK_VERSION/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.1.4.17-1.jar:/usr/hdp/$STACK_VERSION/hive/conf/conf.server:/etc/hadoop/conf/:/etc/hive/conf/ org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool> {hive_log_dir}/pre_upgrade_{target_version}.log

The output indicates whether you need to perform compaction or not:

For more information about the pre-upgrade tool command, see the Pre-upgrade Tool Command Reference below.

Check the following logs on the Hive Metastore host for any errors:

If there are no errors, go to the next step; otherwise, resolve the errors, and repeat this procedure.

On the node where the Hive Metastore resides, log in as a user who has privileges to alter the Hive database.

Start Beeline as the Hive service user. For example:

$ beeline -u 'jdbc:hive2://<Metastore host name>:10000' -n hive

On the Hive command line run the compaction script. For example:

hive> !run /tmp/compacts_nnnnnnnnnnnnn.sql

Output confirms that compaction is queued:

INFO : Compaction enqueued with id 3

…

Proceed to back up the Hive Metastore. This is a mandatory step.

On the node where the database you use for Hive Metastore resides, back up Hive Metastore before upgrading to HDP. For example, in MySQL, dump each database as follows:

mysqldump <hive_db_schema_name> > </path/to/dump_file>

If you use another database for the Hive Metastore, use the equivalent command, such as export for Postgres, to dump the database.

Proceed to upgrade HDP, assuming no Hive update, delete, or merge occurred after compaction; otherwise, repeat the compaction and Hive Metastore backup procedures, and then upgrade HDP.