Issues Fixed in Cloudera Navigator 6.2.0

Navigator Metadata Server purge could produce messages such as "Checking if maintenance is running" and then failed to run during the available window. This problem occurred when a scheduled purge job waits for extraction tasks to finish but while waiting, a policy job starts, preventing the purge job from running. In this release, policy jobs will not be triggered if purge jobs are scheduled and waiting to start.

Cloudera Issue: NAV-7037

Navigator support for compute-only clusters in Cloudera Manager includes audit and metadata extraction from services on the base cluster. No audit events or metadata are extracted from services running on the compute cluster.

For more information, see Virtual Private Cluster Support.

Cloudera Issue: NAV-7028

Navigator tags now support UNICODE characters beyond ASCII. Only ASCII text can be used in the name of a user-defined or managed property. Property values can include international characters.

Cloudera Issue: NAV-7011

This release provides a bulk interface for updating Navigator entity metadata. Use the PUT /entities/bulk/ API to update metadata for many entities in the same call. This API is faster than the single PUT API because it uses a single HTTP request to apply the metadata rather than an HTTP request for each entity.

For more information see the interactive API documentation available from the Help menu in the Navigator console or Updating Metadata for Entities in Bulk: PUT /entities/bulk.

The Apache Thrift client used by Navigator has been upgraded to 0.12.0 to resolve a security vulnerability indicated by CVE-2018-1320.

Cloudera Issue: NAV-6998

This release changes the behavior of the metadata purge operation. When the scheduled purge operation starts, Navigator Metadata Server stops any running extraction and policy tasks so that the purge operation can start immediately. The extraction tasks are automatically rescheduled when the purge completes; the policy tasks are not.

For more information, see Best Practices for Clearing Metadata using Purge.

Cloudera Issue: NAV-6959

This release includes changes that improve how Navigator Metadata Server uses memory, reducing the overall heap required.

Cloudera Issue: NAV-6958

The Navigator console content showing the history of purge jobs is now sorted by time in descending order.

Cloudera Issue: NAV-6916

The Eclipse Jetty web server used by Navigator has been upgraded to 9.4.11 to resolve security vulnerabilities indicated by: CVE-2015-2080, CVE-2016-4800, CVE-2017-7657, CVE-2017-7658, and CVE-2017-9735.

Cloudera Issue: NAV-6901, NAV-6865

Lineage output counts were not displayed correctly when an entity is both an input and output for the current entity. Both the input and output counts are off by one.

In this release, the input and output counts include all leaf nodes that are not operations and not marked deleted.

Cloudera Issue: NAV-6892

The technical metadata for Hive columns now includes the ordinal position of the column as "Field Index", where the index starts at zero. Columns are now listed by their field index when shown in the parent table or view details. Previously, there was no specific order. The additional metadata for existing Hive column entities must be extracted from the Hive sources, so you will not see it immediately after upgrading.

Cloudera Issue: NAV-6815

Spark jobs can use files on the local filesystem as job inputs or outputs. Navigator only supported HFDS, Hive, and S3 assets as job inputs or outputs. As of this release, Navigator now handles local source types when extracting metadata from Spark jobs.

Cloudera Issue: NAV-6811

This release includes Navigator support for file size reporting with the goal of identifying potential performance problems stemming from data stored inefficiently. Navigator collects file size metadata and saves it in HDFS. Telemetry Publisher gets the metadata from HDFS and sends it to Cloudera Workload Experience Manager (WXM).

Not supported for secure clusters.

Enable this feature in Cloudera Manager with Cloudera Management Service properties for "Small Files Reporting:"

Restart Navigator Metadata Server after enabling data collection for small files reporting.

For more information, see File Size Reporting in the Workload Experience Manager documentation.

Cloudera Issue: NAV-6572

The Navigator API interactive documentation for GET /audits had the wrong selector name for databases. The reference to database_name is now correct.

To explore the Swagger interactive API interface, open API Documentation from the help menu in the Navigator console.

Cloudera Issue: NAV-6411

Kite Dataset API was deprecated for Cloudera Manager in 6.0.0. Navigator support for metadata extraction from Kite Datasets is now deprecated and will be removed in a future release. To avoid log messages indicating Kite Dataset issues, you can disable Kite extraction using a Cloudera Manager safety valve. Set the following in the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties:

nav.extractor.hdfs.datasets.enabled

Cloudera Issue: NAV-6368

When an error occurred in the audit pipeline, it was possible that not all components in the Navigator Audit Server recovered fully. One symptom of such a failure is the "CannotSendRequest" error in the Navigator Audit Server log. This problem is fixed in this release.

Cloudera Issue: NAV-6321

Enabling or disabling S3 extraction caused a null pointer exception (NPE) indicating that some of the information needed was null. The error did not affect the S3 extraction functionality. This release improves the interchange to avoid the error.

Cloudera Issue: NAV-6246