What's New in CDH 5.16.x

Sentry and Hive introduced fine grained privileges to provide object-level privileges to roles.

Fine grained privileges adds the CREATE privilege, which allow users to create databases and tables. See the Sentry Privileges documentation for more information about the new privileges.

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the equivalent of the ALL privilege on the object. See the Object Ownership documentation for information about enabling object ownership.

Because of the new object ownership feature, HMS stores the user that creates a table or database in Hive as the object owner. If object ownership is enabled, Sentry grants the user the OWNER privilege. Whether or not object ownership is enabled, HMS stores the user that creates the object as the object owner. Previously, HMS stored the hive user as the object owner.

The following statements were added to Hive to support object ownership via Sentry:

Sentry and Impala introduced fine grained privileges to provide object-level privileges to roles.

Fine grained privileges include the REFRESH and CREATE privileges, which allow users to create databases and tables, and to execute commands that update metadata information on Impala databases and tables. See Impala Sentry documentation for the new privileges and the scopes of the objects that you can grant the new privileges on.

The following new privileges were added:

If a role has SELECT or INSERT privilege on an object in Impala before upgrading to CDH 5.16.1, that role will automatically get the REFRESH privilege during the upgrade.

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the OWNER privilege which is the equivalent of the ALL privilege on the object. See the Object Ownership documentation for information about enabling object ownership.

If the object ownership feature is enabled, Sentry grants the user the OWNER privilege. Whether or not object ownership is enabled, HMS stores the object creator as the default object owner. Previously, HMS stored the Kerberos user as the object owner.

The following statements were added to Impala to support object ownership via Sentry:

A new query option, MAX_MEM_ESTIMATE_FOR_ADMISSION, was added. Use the new option to set an upper limit on the memory estimates of a query as a workaround for over-estimates precluding a query from being admitted.

A new query option, SHUFFLE_DISTINCT_EXPRS, controls the shuffling behavior when a query has both grouping and distinct expressions.

• Incremental Stats

  The following enhancements improve Impala stability. The features reduce chances of having catalogd and impalad crash due to be out of memory when using incremental stats.

  • Incremental stats are now compressed in memory in catalogd, reducing memory footprint in catalogd.
  • Incremental stats are fetched on demand from catalogd by impalad coordinators. This enhancement reduces memory footprint for impalad coordinators and statestored and also reduces network requirements to broadcast metadata.

  See Loading Incremental Statistics from Catalogd for details.

• Automatic Invalidation of Metadata

  To keep the size of metadata bounded and to reduce the chances of catalogd cache running out of memory, this release introduces an automatic metadata invalidation feature with time-based and memory-based invalidation.

  Automatic invalidation of metadata provides more stability with lower chances of running out of memory, but could potentially cause performance risks. The feature is turned off by default.

  See Startup Options for Automatic Invalidation of Metadata for details.

• Additional separators are supported between date and time in default TIMESTAMP format, specifically, the multi-space separator and the 'T' separator. See TIMESTAMP Data Type for more information on TIMESTAMP format.

• New hint placement is supported for INSERT statements. See Optimizer Hints in Impala for detail.

• The REGEX_ESCAPE() function was implemented for escaping special characters to treat them literally in string literals.

• SHOW CREATE VIEW was implemented with the same functionality as SHOW CREATE TABLE.

• Impala Shell can connect directly to impalad when configured with proxy load balancer and Kerberos. See Configuration Options for the new flag that enables the direct connection.

The CREATE and REFRESH (Impala only) privileges have been introduced to allow users to create databases, tables and functions, and to execute commands that update metadata information on Impala databases and tables.

For more information about the new privileges, see Sentry Privileges.

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the equivalent of the ALL privilege on the object.

In CDH 5.16.1, object ownership is disabled by default. For information about enabling and using object ownership, see Object Ownership.

Sentry no longer normalizes group name characters to be lowercase. Therefore, operating system group names do not need to be treated as case insensitive. In previous versions, Sentry modified capital letters in operating system group names to be lowercase.