Issues Fixed in Cloudera Navigator 6.2.0
The following sections describe the issues fixed in the data management components of Cloudera Navigator 6.2.0:
- Navigator Metadata Server purge jobs might not run if there were policies configured
- Support for Compute-Only Clusters
- International characters in tag names
- Bulk update API
- Security vulnerabilities addressed for Thrift
- Metadata purge priority over other tasks
- Reduction in the amount of memory used by Navigator Metadata Server
- Console display of purge history was not sorted
- Security vulnerabilities addressed for Jetty
- Lineage input and output count corrected
- Index added to Hive column metadata
- Navigator didn't recognize local files in Spark jobs
- File size reporting support
- Swagger interactive documentation fixed for GET /audits API
- Kite Datasets support deprecated
- Navigator Audit Server "CannotSendRequest" error resolved
- Corrected exception produced during S3 extraction
Navigator Metadata Server purge jobs might not run if there were policies configured
Navigator Metadata Server purge could produce messages such as "Checking if maintenance is running" and then failed to run during the available window. This problem occurred when a scheduled purge job waits for extraction tasks to finish but while waiting, a policy job starts, preventing the purge job from running. In this release, policy jobs will not be triggered if purge jobs are scheduled and waiting to start.
Cloudera Issue: NAV-7037
Support for Compute-Only Clusters
Navigator support for compute-only clusters in Cloudera Manager includes audit and metadata extraction from services on the base cluster. No audit events or metadata are extracted from services running on the compute cluster.
For more information, see Virtual Private Cluster Support.
Cloudera Issue: NAV-7028
International characters in tag names
Navigator tags now support UNICODE characters beyond ASCII. Only ASCII text can be used in the name of a user-defined or managed property. Property values can include international characters.
Cloudera Issue: NAV-7011
Bulk update API
This release provides a bulk interface for updating Navigator entity metadata. Use the PUT /entities/bulk/ API to update metadata for many entities in the same call. This API is faster than the single PUT API because it uses a single HTTP request to apply the metadata rather than an HTTP request for each entity.
For more information see the interactive API documentation available from the Help menu in the Navigator console or Updating Metadata for Entities in Bulk: PUT /entities/bulk.
Security vulnerabilities addressed for Thrift
The Apache Thrift client used by Navigator has been upgraded to 0.12.0 to resolve a security vulnerability indicated by CVE-2018-1320.
Cloudera Issue: NAV-6998
Metadata purge priority over other tasks
This release changes the behavior of the metadata purge operation. When the scheduled purge operation starts, Navigator Metadata Server stops any running extraction and policy tasks so that the purge operation can start immediately. The extraction tasks are automatically rescheduled when the purge completes; the policy tasks are not.
For more information, see Best Practices for Clearing Metadata using Purge.
Cloudera Issue: NAV-6959
Reduction in the amount of memory used by Navigator Metadata Server
This release includes changes that improve how Navigator Metadata Server uses memory, reducing the overall heap required.
Cloudera Issue: NAV-6958
Console display of purge history was not sorted
The Navigator console content showing the history of purge jobs is now sorted by time in descending order.
Cloudera Issue: NAV-6916
Security vulnerabilities addressed for Jetty
Lineage input and output count corrected
Lineage output counts were not displayed correctly when an entity is both an input and output for the current entity. Both the input and output counts are off by one.
In this release, the input and output counts include all leaf nodes that are not operations and not marked deleted.
Cloudera Issue: NAV-6892
Index added to Hive column metadata
The technical metadata for Hive columns now includes the ordinal position of the column as "Field Index", where the index starts at zero. Columns are now listed by their field index when shown in the parent table or view details. Previously, there was no specific order. The additional metadata for existing Hive column entities must be extracted from the Hive sources, so you will not see it immediately after upgrading.
Cloudera Issue: NAV-6815
Navigator didn't recognize local files in Spark jobs
Spark jobs can use files on the local filesystem as job inputs or outputs. Navigator only supported HFDS, Hive, and S3 assets as job inputs or outputs. As of this release, Navigator now handles local source types when extracting metadata from Spark jobs.
Cloudera Issue: NAV-6811
Swagger interactive documentation fixed for GET /audits API
The Navigator API interactive documentation for GET /audits had the wrong selector name for databases. The reference to database_name is now correct.
To explore the Swagger interactive API interface, open API Documentation from the help menu in the Navigator console.
Cloudera Issue: NAV-6411
Kite Datasets support deprecated
Kite Dataset API was deprecated for Cloudera Manager in 6.0.0. Navigator support for metadata extraction from Kite Datasets is now deprecated and will be removed in a future release. To avoid log messages indicating Kite Dataset issues, you can disable Kite extraction using a Cloudera Manager safety valve. Set the following in the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties:
Cloudera Issue: NAV-6368
Navigator Audit Server "CannotSendRequest" error resolved
When an error occurred in the audit pipeline, it was possible that not all components in the Navigator Audit Server recovered fully. One symptom of such a failure is the "CannotSendRequest" error in the Navigator Audit Server log. This problem is fixed in this release.
Cloudera Issue: NAV-6321