Behavioral changes in Cloudera Runtime 7.1.9 SP1 CHF 15

Hive

HIVE-26976, HIVE-28578, HIVE-29316, HIVE-29030, CDPD-75871: Performance-related issues for HMS

Previous behavior:

Several performance-related issues for HMS occurred, including stale entries or inconsistent metadata in the database and exceptions during the update operation for multiple table column statistics or for multiple partition-level column statistics.

New behavior:

To resolve these issues, a backend database exclusive row lock is now introduced. The statistics importer tasks are coordinated so that only one importer task that obtains the lock runs at a time. The other tasks to import or update the statistics for the same table or partition continue to be in the pending queue until the current importer task completes and then releases the lock. By default, a blocked or pending importer task retries the lock 100 times in the 3 secs ~ 33 secs duration until a failure occurs.

You can now configure the number of retries and the interval between retries using the following parameters that you can set on the Cloudera Manager > Clusters > Hive service > Configuration tab:

• The hive.metastore.s4u.nowait.max.retries parameter allows you to set the number of retries to acquire the row lock immediately without waiting. The default value is 100.
• The hive.metastore.s4u.nowait.retry.sleep.interval parameter allows you to set the interval between the retries to acquire a row lock immediately. The default value is 300ms.