Fixed Issues in Streams Messaging

Kafka

CDPD-40985: Expose log directory total and usable space through the Kafka API (KAFKA-13958 backport)

This is a backported improvement, see KIP-827 and KAFKA-13958 for more information.

CDPD-35680: Remove the verification of the existence of the JWT "sub" claim to extend compatibility with OAuth providers (KAFKA-13730 backport)

This is a backported improvement, see KAFKA-13730 for more information.

Topics created with the kafka-topics tool are only accessible by the user who created them when the deprecated --zookeeper option is used

The --zookeeper option has been removed from the kafka-topics tool. As a result, encountering this issue is no longer possible. Use the --bootstrap-server option instead.

Certain Kafka command line tools require direct access to ZooKeeper

There are no longer any Cloudera supported Kafka command line tools that require direct ZooKeeper access or require the usage of the --zookeeper option. Use the --bootstrap-server option instead.

CDPD-39422: Consumer polls for coordinator in tight loop (KAFKA-13917 backport)

The consumer uses a backoff period controlled by retry.backoff.ms to wait before polling for the coordinator if the previous poll was unsuccessful. This is a backported improvement, see KAFKA-13917 for more information.

CDPD-39354: Kafka Connect connectors and tasks fail to start

Kafka Connect initializes the Secrets Storage right at startup. As a result, timing issues no longer occur in the configuration resolution of connectors and tasks.

CDPD-45958: Kafka client JAAS override policy validation is incorrect

The JAAS override filter policy now correctly filters based on the specified rules and does not refuse JAAS configurations because of unknown fields.

CDPD-44252: Exception during normal operation in MirrorSourceTask causes the task to fail instead of shutting down gracefully

Stopping the read of offsets in a worker of a MirrorSourceTask will now cause a graceful shutdown and the task can be restarted automatically at a later point.

CDPD-39391: Amazon S3 Sink fails when validating bucket names

Due to an issue with the AWS S3 bucket name validation of the Amazon S3 Sink connector, the connector encountered an exception when validating bucket names. This issue is now fixed.

OPSAPS-64606: Authorization issues if Kafka Connect is not installed

If the Kafka Connect role is not present on the cluster, then a Ranger policy (connect internal - topic) is created with default, non-empty topic names. As a result, the Ranger policy include list cannot be empty and will not have any side effects on other Kafka operations.

OPSAPS-63526: Kafka's Ranger related log4j2 config generation is not idempotent

If Kafka fails to start up due a configuration change, the original cause for the startup failure is no longer replaced with an error related to log4j2 configuration after the first auto-attempted restart.

OPSAPS-63640: Monitoring a high number of Kafka producers might cause Cloudera Manager to slow down and run out of memory

This issue is fixed.

Schema Registry

CDPD-20977: Add RAW Avro JSON Schema API for Hive Integration

Added new endpoints where the client can GET the actual schema text as a JSON document. These endpoints were added as a subresource (.../schemaText) to the existing schema version resource endpoints, corresponding to the "schemaText'' property of those, for example, /api/v1/schemaregistry/schemas/{name}/versions/latest/schemaText.

CDPD-39885: SchemaRegistryResource.uploadFiles fails w/ Timeout waiting for connection from pool

When downloading a file from Schema Registry, the stream was not properly closed which occasionally caused issues. This has been fixed by always closing the stream.

CDPD-40758: Improve upon how SchemaRegistryClient is used in connectors

Fixed the issue when setting the value.converter.serdes.protocol.version in connectors caused the connector to fail on startup because the configuration property was not properly converted to a byte value.

CDPD-41592: Confluent import should handle input without an actual version

Lines will be ignored in Confluent import format where only the first part corresponding to a Cloudera SR "meta" is present and schema version data is missing (or only a "null" string is there).

CDPD-45920: KafkaJsonDeserializer does not support primitive type literals and array types

Fixed JSON deserializer's handling of non-object literal values (simple strings, numeric literals or arrays).

Streams Messaging Manager

CDPD-33699: Remove "adjustTopicOverviewMetrics" from SMM

Removed the logic introduced in 7.1.4/7.2.2.0, where in case the topicMetrics (bytes in/bytes out/messages in) are smaller for a larger time period, the smaller timeperiod's metrics will be displayed. For example, if the metrics are smaller for 30 days and then for 7 days, the 7 day metrics would be used.

CDPD-35136: Data Explorer should allow long keys to be viewed

Fixed the issue with the keys being truncated in the Data Explorer in case they are too long on SMM UI. In case the message's key is too long to render, a "read more" button will appear next to the truncated key, initiating a popup which will show the full content of the key.

CDPD-35374: Incorrect key/value for large numbers

Fixed issue where large numbers were rounded in the Data Explorer in case LongDeserialiser is used for either the key or the value while viewing the content of a partition.

CDPD-35897: Remove partitionMetrics from /api/v1/admin/metrics/consumers/group/{groupId} response

Removed the "topicPartitionMetrics" field from the response of "/api/v1/admin/metrics/consumers/group/{groupId}" endpoint, since the underlying metric has been removed. Additionally, the API path version was bumped to v2.

CDPD-36420: List icons should not appear for the validation errors

Removed bullet point from validation errors on the connector creation form of streams messaging manager UI.

CDPD-39137: Connect topic tracking line is not showing on UI

On the Connectors page, topic lineage lines are not showing up, only the line starts and ends. Now, Topic lineage lines are shown on the Connectors page.

CDPD-39778: Should not suggest flow.snapshot to be a secret

Streams Messaging Manager now does not allow the user to mark the flow.snapshot as sensitive data.

CDPD-39826: The Restart button for the ConnectorTasks is permanently disabled

Issue with the Restart button being permanently inactive on the ConnectorDetails page is fixed.

CDPD-39980: /api/v1/admin/auth/access throws NPE on unsecure environments

Fixed issue "/api/v1/admin/auth/access" throwing InternalServerError (Http code 500) when accessing the endpoint in a non-kerberized environment.

CDPD-40286: When using Oracle database, we can have an NPE during configuration

When using the Oracle database, the customer has the option to provide custom connection properties. Due to a bug, these connection properties were mandatory, which caused an error in case they were not provided.

CDPD-40758: Improve upon how SchemaRegistryClient is used in connectors

Fixed the issue when setting the "value.converter.serdes.protocol.version" in connectors caused the connector to fail on startup because the configuration property wasn't properly converted to a byte value.

CDPD-40871: SMM should fill "partitionMetrics" on "/api/v2/admin/metrics/aggregated/topics/{topicName}" even when partitionMetrics are empty

All the data that is available from the topic partitions are filled regardless whether metrics were fetched from Cloudera Manager or not.

CDPD-41069: Topic can be edited after selecting REPLICATION_STATUS as the alert attribute

Fixed the topic selection dropdown status in the alert editor after various UI events.

CDPD-41420: Schema version is not displayed in SMM when Avro value serializer is chosen

In the Data Explorer now it is possible to see all the schema versions associated with the given topic.

CDPD-41514: SMM displays CPU usage chart instead of CPU load

Fixed issue where SMM displays CPU usage chart instead of CPU load on the broker details page.

CDPD-41542: Consumer instance host field is empty

Fixed issue where consumer instance host field is empty in the SMM UI consumer details page.

CDPD-41553: Replication charts do not render on first visit

Fixed issue where replication charts do not render on first visit in the SMM topic details page.

CDPD-43387: Broker Details page does not show the Cloudera Manager button

The Cloudera Manager buttons that navigate to the broker resource within Cloudera Manager were not visible in previous releases. Now you can navigate from the SMM's broker view to the Cloudera Manager's broker view.

CDPD-43474: KConnect metrics cannot be scraped by prometheus in secure environments

Fixed Prometheus not being able to scrape Connect metrics from secure Kafka clusters. The "connect.prometheus.metrics.port" configuration was removed.

CDPD-43962: Performance improvement on the Broker Details page

The '/api/v2/admin/metrics/aggregated/brokers/{brokerId}' endpoint, called every time while opening the broker details page on the UI, was excessively slow when a large number of topics and partitions were present. This is now fixed by fetching partition metrics in bulk.

Streams Replication Manager

OPSAPS-63992: Rolling restart unavailable for SRM

Streams Replication Manager rolling restart support is added to Cloudera Manager. Streams Replication Manager can be restarted and upgraded without losing service availability.

Cruise Control

There are no fixed issues for Cruise Control in Cloudera DataFlow for Data Hub 7.2.16.