Preparing Hive for upgrade

The main objective of preparing Hive for upgrade is to run the Hive pre-upgrade tool to detect Hive ACID tables, if there are any, and perform compaction if necessary. The pre-upgrade tool looks for files in an ACID table that contains update or delete events, and generates scripts to compact these tables. Compaction of such tables is mandatory before upgrading to prevent data loss.

Prerequisites

  • Your Hive tables reside in HDP 2.6.5.x. Upgrades from HDP 3.x are not supported in this release of CDP Private Cloud Base.
  • JDK is installed on the node running Hive Metastore.
  • Hive Metastore is running and connected to the node where you will run the Hive pre-upgrade tool.
  • If Hive metastore contains ACID tables, you have enabled ACID operations using Ambari Web or set the following Hive configuration properties to enable ACID. Failure to set these properties before upgrading Hive will result in corrupt or unreadable ACID tables.
    • hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
    • hive.support.concurrency=true
  • Optionally, you shut down HiveServer2. Shutting down HiveServer2 is recommended, but not required, to prevent operations on ACID tables while the tool executes.
  • You ensured the cluster has sufficient capacity to execute any compaction jobs that the upgrade tool might submit by setting hive.compactor.worker.threads to accommodate your data.

  • You obtained permissions below to perform the steps to prepare Hive for upgrade.
    • Hive service user permissions, or all permissions to access Hive that Ranger provides
    • If you use Kerberos, a valid Kerberos ticket for starting Hive as the Hive service user

      The Hive service user is usually the default hive user. If you don’t know who the Hive service user is, go to the Ambari Web UI, and click Cluster Admin > Service Accounts, and then look for Hive User.

    • Capability to login as the HDFS superuser.
    • If you use Kerberos, you have become the HDFS superuser with a valid ticket.