New Features in Cloudera Navigator 6.1.0

HDFS metadata extraction can take up significant resources, particularly if the initial build extraction is not completed in a timely interval. Extractors running for long periods then cause the metadata purge operation to be cut short or never run at all. This release addresses these problems by significantly improving metadata extraction for HDFS and HMS and reducing the time required to purge metadata. Together, these improvements will make your Navigator Metadata Server more robust and the resource management more predictable.

• Incremental metadata extraction from HMS

  This release introduces incremental extraction logic for extracting metadata from Hive Metastore (HMS). The CDH 6.1.0 distribution of Hive makes available additional timing information, which Navigator takes advantage of to perform incremental extraction after the first full extraction is complete. Incremental metadata extraction has the potential to substantially improve the HMS extraction performance. With incremental extraction, Navigator can check for HMS metadata changes much more often, substantially reducing the time required for Hive metadata changes to be reflected in Navigator and Hue.

  This feature is disabled by default. Cloudera testing for production data volumes is ongoing.

• HDFS metadata extraction performance improvement

  Previously, Navigator processing for HDFS metadata extraction could take long enough that Navigator could not finish indexing the fsimage before the edit log was checkpointed into a new fsimage. This caused extraction to get stuck in a loop of always parsing the fsimage in bulk extraction mode (slow) rather than moving into incremental extraction from the edit log (fast). This release includes changes that allow Navigator to avoid this "bulk extraction loop," therefore reducing the duration of HDFS extraction by as much as half for the initial extraction. This change improves the performance of subsequent extractions significantly, both by making it much more likely that extraction will be incremental and from additional performance improvements in this fix.

• Improved metadata purge performance

  This release includes changes that significantly reduce the time required to complete purging for HDFS entities and relations. The purge behavior is changed such that HDFS entities are not included in the purge if they are referenced as an endpoint in a data flow lineage relation. The performance improvement will shorten the time required to run the HDFS portion of the purge operation; combined that smaller window with the improved extraction performance and the purge jobs are much more likely to run and run to completion.

This release introduces changes to default HDFS audit filters to better focus the collection of audit events. This change increases the scalability of your Navigator Audit Server system by removing unnecessary audit events.

The following rules were added or updated:

• All HDFS denied access events are accepted. Previously, HDFS denied access events from filtered users and filtered HDFS directories were not recorded.
• HDFS events from the Hive, Spark, and Impala staging directories are discarded. HDFS events from the following job history directories are also discarded:
  • /user/history/done_intermediate
  • /user/spark/applicationHistory
  • /user/spark/spark2ApplicationHistory
• All HDFS delete and rename operations from directories that are not already filtered are accepted.

In addition at the end of the filter list, Cloudera Manager provides a rule that filters events from HDFS getfileinfo operations. By default, this filter rule has no affect. Cloudera recommends that you change this rule to Discard to stop capturing events for this HDFS operation that indicates access to file metadata but does not indicate access to file data. This change alone may provide up to 30% reduction in HDFS audit data size.

When upgrading to this release, you may not see the new filters if you have customized your HDFS audit filters previously:

• If HDFS filters have been changed from the default, the new filters will NOT take effect.
• If HDFS filters have not been changed from the default, the new filters will take effect.

Whether or not the new filters are applied, you'll notice that the Cloudera Manager interface has improved controls for adding the audit filters: you have options for setting the order and for duplicating an existing filter to use it as a starting point on a new filter.

Audit Filter Control in Cloudera Manager Service Configuration

To review the recommended filters and determine further optimizations for your system's requirements, see Recommended Audit Filters and Reviewing Default Audit Filters.

This release includes new checks for Navigator Metadata Server health. If the checks fail, they trigger a health alert in Cloudera Manager for the Navigator Metadata Server. The checks include "Solr Element Count Threshold" and "Solr Relation Count Threshold". These checks trigger an alert if documents in Navigator Metadata Server's embedded Solr collection are threatening to exceed the maximum allowed number for either the element core or the relation core. Typically, if either of these alerts trigger, Navigator has encountered a problem that produces more relations than it should. The triggers let you know that this problem is occurring while it can be addressed efficiently.

For more information, see Navigator Metadata Server Solr Element Count Threshold Test and Navigator Metadata Server Solr Relation Count Threshold Test.

This release includes improvements to the Navigator console, including improvements to the layout and information in the columns list. When you are viewing the detail page for a table or view, columns show inline descriptions and managed properties. A popover menu lets you see additional column metadata. The inline search in the Column box now matches for managed metadata and description text in addition to name and type.

The Navigator console Search field will now auto-suggest names of entities—tables, views, fields and databases. When you select on of the suggested entities, the console opens the entity details page.

This auto-complete feature also can be used with db.table. field search syntax.

Navigator Audit Server receives audit events from Sentry, Hive, and Impala that are produced when an administrator or user changes ownership for a database, table, or view. For more information, see Sentry's Object Ownership and Navigator's list of audit events for Sentry.

Navigator Audit Server receives audit events from HDFS that are produced when an administrator performs a command to manage an HDFS erasure coding policy. For more information, see HDFS's Erasure Coding introduction and Navigator's list of audit events for HDFS.