The Hive Metastore Server

Cloudera recommends using a remote Metastore with Hive, especially for CDH 4.2 or later. Since the remote Metastore is recommended, Cloudera Manager treats the Hive Metastore Server as a required role for all Hive services. Here are a couple key reasons why the remote Metastore setup is advantageous, especially in production settings:

• The Hive Metastore database password and JDBC drivers don’t need to be shared with every Hive client; only the Hive Metastore Server does. Sharing passwords with many hosts is a security concern.
• You can control activity on the Hive Metastore database. To stop all activity on the database, just stop the Hive Metastore Server. This makes it easy to perform tasks such as backup and upgrade, which require all Hive activity to stop.

Information about the initial configuration of a remote Hive Metastore Server with Cloudera Manager can be found at Cloudera Manager and Managed Service Databases.

Considerations When Upgrading CDH

Hive has undergone major version changes from CDH 4.0 to 4.1 and between CDH 4.1 and 4.2. (CDH 4.0 had Hive 0.8.0, CDH 4.1 used Hive 0.9.0, and CDH 4.2 or later has 0.10.0). This requires that you manually back up and upgrade the Hive Metastore database when upgrading between major Hive versions.

You should follow the steps in the appropriate in the Cloudera Manager procedure for upgrading CDH to upgrade the Metastore before you restart the Hive service. This applies whether you are upgrading to packages or parcels. The procedure for upgrading CDH using packages is at Upgrading from CDH 4 Packages to CDH 5 Packages. The procedure for upgrading with parcels is at Upgrading from CDH 4 to CDH 5 Parcels.

Considerations When Upgrading Cloudera Manager

Cloudera Manager 4.5 added support for Hive, which includes the Hive Metastore Server role type. This role manages the Metastore process when Hive is configured with a remote Metastore.

When upgrading from Cloudera Manager prior to 4.5, Cloudera Manager automatically creates new Hive service(s) to capture the previous implicit Hive dependency from Hue and Impala. Your previous services will continue to function without impact. If Hue was using a Hive Metastore backed by a Derby database, then the newly created Hive Metastore Server will also use Derby. Since Derby does not allow concurrent connections, Hue will continue to work, but the new Hive Metastore Server will fail to run. The failure is harmless (because nothing uses this new Hive Metastore Server at this point) and intentional, to preserve the set of cluster functionality as it was before upgrade. Cloudera discourages the use of a Derby backed Hive Metastore due to its limitations. You should consider switching to a different supported database.

Cloudera Manager provides a Hive configuration option to bypass the Hive Metastore Server. When this configuration is enabled, Hive clients, Hue, and Impala connect directly to the Hive Metastore database. Prior to Cloudera Manager 4.5, Hue and Impala connected directly to the Hive Metastore database, so the bypass mode is enabled by default when upgrading to Cloudera Manager 4.5 or later. This is to ensure the upgrade doesn't disrupt your existing setup. You should plan to disable the bypass mode, especially when using CDH 4.2 or later. Using the Hive Metastore Server is the recommended configuration and the WebHCat Server role requires the Hive Metastore Server to not be bypassed. To disable bypass mode, see Disabling Bypass Mode.

Cloudera Manager 4.5 or later also supports HiveServer2 with CDH 4.2. In CDH 4 HiveServer2 is not added by default, but can be added as a new role under the Hive service (see Role Instances). In CDH 5, HiveServer2 is a mandatory role.

Disabling Bypass Mode

In bypass mode Hive clients directly access the metastore database instead of using the Hive Metastore Server for metastore information.

1. Go to the Hive service.
2. Select Configuration > View and Edit.
3. Expand the Service-Wide > Advanced category.
4. Uncheck the Bypass Hive Metastore Server checkbox.
5. Click Save Changes.
6. Re-deploy Hive client configurations.
7. Restart Hive and any Hue or Impala services configured to use that Hive service.

Using Hive Gateways

Because the Hive service does not have worker roles, another mechanism is needed to enable the automatic propagation of client configurations to the other hosts in your cluster. Gateway roles fulfill this function. Gateways in fact aren't really roles and do not have state, but they act as indicators for where client configurations should be placed. Hive gateways are created by default when the Hive service is added.