Issues Fixed in Cloudera Navigator 6.1.0

Navigator Audit Server produced a Null Pointer Exception error and failed to process Impala audits. The error was similar to the following:

2018-09-26 01:53:25,341 ERROR com.cloudera.nav.analytics.dataservices.etl.tasks.audits.AbstractAuditETLTask [pool-12-thread-2]: Error encountered in executing the task New Databases ETL Task for service impala. Will continue with remaining sources. java.lang.NullPointerException

This problem is fixed in this release.

Cloudera Issue: NAV-6889

Audit Analytics in the Navigator console were disabled by default in the previous release; however, this change caused certain common operations to throw exceptions in the Navigator Audit Server log:

java.io.IOException: Unexpected length (too long): 2064261152, max:10485760

This issue is fixed in this release.

Cloudera Issue: NAV-6876

Previously, if the purge API call is made without supplying deleteTimeThresholdMinutes or runtimeCapMinutes, Navigator Metadata Server returns an HTTP error 500. As of this release, Navigator returns an HTTP error 400.

Cloudera Issue: NAV-6818

The entity Description field in the Navigator console now respects newlines and whitespace included in the field text.

Cloudera Issue: NAV-6803

This release of Navigator increases the connection idle time for the connection between Navigator Audit Server and each Cloudera Manager agent that sends audits to Navigator from audited services.

This change will help avoid audit pipeline "broken pipe" errors that can occur when increased load on the audit server node causes Navigator Audit Server not to respond quickly enough.

Cloudera Issue: NAV-6796

The audit event filter in the Navigator console was case sensitive for database or table names when audit events are stored in PostgreSQL databases. This release resolves the problem.

Cloudera Issue: NAV-6141, NAV-6795

This release includes incremental extraction for HMS metadata. See New Features for more information.

This change removes all duplicate Spark data flow relations, extending the fix provided by Navigator Metadata Server has high CPU usage, linking taking too long after Spark extraction.

Cloudera Issue: NAV-6751

Navigator metadata extraction from Hive uses an extractor state to keep track of the Hive entities committed to the Navigator Metadata Server data storage in Solr. In some cases, such as when the extractor state file is deleted or when Navigator Metadata Server is shut down during extraction from Hive, the extractor state no longer represents the state of the Hive extraction. When this happens, there may be entities listed in Navigator that are not accounted for in the extractor state. When this happens, Navigator no longer updates the Hive entities that are in Solr but not in the extractor state.

This release includes a check to make sure that the extractor state is synchronized with the content of Hive entities in Solr. To enable the check set the following property in the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve):

nav.extractor.hive.validate_state=true

Cloudera Issue: NAV-6745

When purging deleted properties from the Navigator console, occasionally the error "Failed to start purge. Reason: API: Service Unavailable" appeared. The error occurred when a purge job is already scheduled and another the purge job could not be added to the queue. In this release, the message is changed to better indicate the problem: "Extractors are running. Waiting for jobs to finish before starting maintenance."

Cloudera Issue: NAV-6739

Using the interactive Navigator API for the /audits endpoint failed with the error:

No serializer found for class sun.net.www.protocol.http.HttpURLConnection$HttpInputStream and
no properties discovered to create BeanSerializer (to avoid exception, disable
SerializationConfig.SerializationFeature.FAIL_ON_EMPTY_BEANS) )"

Also, the documentation accompanying the audit API says a selector is table but selector is actually table_name.

Work-around: Change limit to a value less than 10000.

Cloudera Issue: NAV-6664

Navigator does not have direct dependencies on any third-party libraries that contain security vulnerabilities.

Cloudera Issue: NAV-6622

Navigator Audit Server and Navigator Metadata Server support using MySQL version 10.x for their databases.

Cloudera Issue: NAV-6619

The Cloudera Manager interface for adding Navigator audit filters is improved so that it is easier to control the order of the filters and to duplicate an existing filter.

Audit Filter Controls in Cloudera Manager Service Configuration

Cloudera Issue: NAV-6616

Pre-registration of Hive entities (table, column, or database) fails causing exceptions in the logs such as:

java.lang.ClassCastException: com.cloudera.nav.core.model.GenericEntity cannot be cast to com.cloudera.nav.hive.model.HTable

Cloudera Issue: NAV-6608

In some cases, Navigator Metadata Server processing required an extraordinary amount of memory for longer than expected while extracting and linking metadata when Spark (Spark 1 or Spark 2) extractions are enabled.

A symptom of the problem is a flood of Navigator Metadata Server log messages such as:

Finished linking relation with id: <id>

This change removes duplicate relations from SPARK data flow relations that were causing the system to use the additional resources.

Cloudera Issue: NAV-6591

The Navigator Metadata Server logs included too much logging detail at the INFO level coming from the linking process that runs against Spark entities. Many of the log entries were moved to DEBUG.

Cloudera Issue: NAV-6560

After running a query in Hive, the Hive Server 2 standard error log included the following error:

log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a "com.cloudera.navigator.shaded.log4j.Appender" variable.
log4j:ERROR The class "com.cloudera.navigator.shaded.log4j.Appender" was loaded by
log4j:ERROR [sun.misc.Launcher$AppClassLoader@47d384ee] whereas object of type
log4j:ERROR "org.apache.logj4.ConsoleAppender" was loaded by [sun.misc.Launcher$AppClassLoader@47d384ee].
log4j:ERROR Could not instantiate appender named "out".

This problem occurred because the Navigator plugin for Hive Server 2 expects a different LOG4J integration than what Hive Server 2 is using. The result was that no Navigator audit messages appear in the Hive Server 2 logs. Hive Server 2 auditing was not affected by this problem.

Cloudera Issue: NAV-6523

The Navigator console Search field will now auto-suggest names of entities—tables, views, fields and databases. When you select on of the suggested entities, the console opens the entity details page.

This auto-complete feature also can be used with db.table. field search syntax.

Cloudera Issue: NAV-6496

In large HDFS deployments, the fsimage takes a long time to index. For Navigator, the fallout of this delay was that no HDFS entities deleted in the cluster were marked as deleted in Navigator. This problem is fixed in this release. See also HDFS metadata extraction performance improvement.

Cloudera Issue: NAV-6456

After using Sqoop to transfer data to HDFS, Navigator Metadata Server may have duplicated relations between YARN OPERATION_EXECUTION entities and HDFS directory entities.

Cloudera Issue: NAV-6446

Navigator capped the number of lineage parents that could be returned, however this cap included lineage relations with second-level parents (columns). When reaching the limit, the Navigator Metadata Server log included an error such as Lineage Diagram is limited to 3000 entities.

Work-around: Set nav.capacity.max_nodes_limit=1000 in Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties configuration. Restart Navigator Metadata Server and validate that the lineage appears correctly. This workaround assumes that lineage touches a maximum of 1000 tables; increase the limit if necessary to accommodate lineage relations in your environment.

Cloudera Issue: NAV-6371

This release includes new audit filters, including denied access events. See Improvements to audit extraction filters.

Cloudera Issue: NAV-6290

This release significantly reduces the time required to complete purging for HDFS files. The purge behavior is changed such that HDFS entities are not included in the purge if they are referenced as an endpoint in a data flow lineage relation.

Cloudera Issue: NAV-6284

This release includes improvements to the Navigator console, including improvements to the layout and information in the columns list. When you are viewing the detail page for a table or view, columns show inline descriptions and managed properties. A popover menu lets you see additional column metadata. The inline search in the Column box now matches for managed metadata and description text in addition to name and type.

Cloudera Issue: NAV-6234

When Navigator Metadata Server is started with a fresh storage directory on Cloudera Manager versions 5.12.0 or later, extractors were not working when a single Cloudera Manager managed multiple clusters.

This problem is fixed in this release.

Cloudera Issue: NAV-6145

This release includes new checks for Navigator Metadata Server health. See New Cloudera Manager health alerts for Navigator Metadata Server.

Cloudera Issue: NAV-6116

Navigator processing for HDFS metadata extraction can take long enough that Navigator cannot finish indexing the fsimage before the edit log is checkpointed into a new fsimage. This causes extraction to get stuck in a loop of always parsing the fsimage in bulk extraction mode (which is slow) rather than moving into incremental extraction from the edit log. This release includes changes that allow Navigator to avoid the bulk extraction loop, therefore reducing the duration of HDFS extraction by as much as half for the initial extraction; performance of subsequent extractions can improve significantly, benefiting both by moving to incremental extraction and by improvements in this fix.

Cloudera Issue: NAV-6033

As of this release, Navigator uses jQuery 3.x, which includes security improvements from the 2.x version. This change does not affect the behavior of the Navigator console.

Cloudera Issue: NAV-6003

When a Hive view is created, then dropped, and then subsequently recreated as a table with the same name as that of the original view, the Hive extraction process shows this exception in Navigator Metadata Server logs:

java.lang.ClassCastException: com.cloudera.nav.hive.model.HView cannot be cast to com.cloudera.nav.hive.model.HTable

Cloudera Issue: NAV-5939

Security headers are now included on Navigator console resources (such as JavaScript files) so that Navigator console pages pass through security scanners without additional validation.

Cloudera Issue: NAV-5929

The Navigator console now includes the Content Security Policy (CSP) support, which introduces an addition layer of security for browsers that support it. See Mozilla's Policy Browser Compatibility

Cloudera Issue: NAV-5928

Previously, if Navigator Metadata Server is configured with External Authentication Type "LDAP" and the LDAP server is Active Directory, when the LDAP Group Search Base property is configured with the root of the directory tree, authentication fails in the console with the error "Incorrect username or password."

Navigator Metadata Server logs show the following exception:

2018-10-24 12:21:04,619 ERROR com.cloudera.nav.auth.DelegatingNavAuthProvider [qtp1989132530-59]:
External Authentication Service threw exception during authentication process.
org.springframework.ldap.PartialResultException: Unprocessed Continuation Reference(s);
nested exception is javax.naming.PartialResultException: Unprocessed Continuation Reference(s);
remaining name 'DC=ad,DC=sec,DC=example,DC=com'

There were two common workarounds for this problem: use the Global Catalog or configure a more specific location in the directory tree for the LDAP Group Search Base. In this release, the root-level LDAP Group Search Base is correctly evaluated and no work-around is required. If you have configured one of these workarounds, you can keep the configuration as is with no change of behavior.

Cloudera Issue: NAV-5669

Navigator Audit Server receives audit events from Sentry, Hive, and Impala that are produced when an administrator or user changes ownership for a database, table, or view. For more information, see Sentry's Object Ownership and Navigator's list of audit events for Sentry.

Cloudera Issue: CDH-65161

Navigator Audit Server receives audit events from HDFS that are produced when an administrator performs a command to manage an HDFS erasure coding policy. For more information, see HDFS's Erasure Coding introduction and Navigator's list of audit events for HDFS.

Cloudera Issue: NAV-5266

This problem applies when the Navigator Audit Server database is MySQL configured to use utf8 character set encoding.

When a query includes an Emoji or other Unicode supplementary-plane character that is encoded as four bytes in UTF-8, Navigator Audit Server failed to process the event and any following events from the same service.

For new installations, you can avoid this issue by configuring MySQL v5.5 or later to use the utf8mb4 character set encoding. Converting existing MySQL databases for Navigator Metadata Server and Navigator Audit Server to utf8mb4 requires updating the master tables to meet MySQL restrictions encountered after the change to 4-byte encoding. Contact Cloudera Technical Support for details.

Cloudera Issue: NAV-4845

Navigator Audit Server did not write audit events for activity from one-off processes spawn by Cloudera Manager. The missing audit events could be for HDFS or HBase, with server log messages as follows:

2018-04-07 10:35:09,159 INFO com.cloudera.navigator.analytics.load.AnalyticsStagingService
[AnalyticsStagingService]: Could not load HDFS activity data for analytics java.lang.IllegalStateException:
Number of files must be > 0

or

2018-11-29 12:34:25,616 WARN com.cloudera.navigator.NavigatorServer [653696675@NavigatorServer-1]:
Error persisting events for service hbase java.lang.IllegalArgumentException: No property 'tableName' in
class 'com.cloudera.navigator.model.GenericAuditEvent'.

Cloudera Issue: NAV-4006

The API version was not correct in the Cloudera Navigator download option for audit reports.

Cloudera Issue: NAV-3699

For certain audit events containing SQL, the Navigator console audit entries for SQL now show a preview and an option to expand and format the SQL.

Cloudera Issue: NAV-2331