Cloudera Docs

What's New in Streams Messaging

Learn about the new Streams Messaging features in Cloudera DataFlow for Data Hub 7.2.11.

Kafka

New performance related health tests for Kafka
Two new health tests based on the kafka_request_handler_avg_idle_1min_rate and the kafka_network_processor_avg_idle metrics are added for Kafka in Cloudera Manager. The health tests added are the following the following:
  • Request Handler Capacity

    This health test checks the most recent value of the kafka_request_handler_avg_idle_1min_rate metric and sends a warning if less than 30% of request handler capacity is available. Additionally, the warning recommends that users increase the number of I/O threads using the Number of I/O Threads (num.io.threads) property.

  • Network Processor Capacity

    This health test checks the most recent value of the kafka_network_processor_avg_idle metric and sends a warning if less than 30% of network processor capacity is available. Additionally, the warning recommends that users increase the number of network threads using the Number of Network Threads (num.network.threads) property.

For more information, see Cloudera Manager Health Tests Reference.

Schema Registry

Schema Registry Confluent API needs a compatibility endpoint

A new endpoint that checks compatibility between schemas is now available to the Confluent-compatible API.

The endpoint is as follows: 

/api/v1/confluent/compatibility/subjects/[***SCHEMA***]/versions/[***VERSION***]
Where:
  • [***SCHEMA***] is the name of the subject/schema.
  • [***VERSION***] is the version that the new schema text is compared to. The version can be latest or a valid version ID.

When using the endpoint, you must specify the schema, version, and the schema text that you want to compare. The schema text can be sent as a data parameter. For example:

curl -X POST "http://\'hostname\':9090/api/v1/confluent/compatibility/subjects/schemaname/versions/latest" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"schema\":\"{\\\"fields\\\":[{\\\"default\\\": \\\"yellow\\\",\\\"name\\\": \\\"color\\\",\\\"type\\\": [\\\"string\\\"]}],\\\"name\\\": \\\"schemaname\\\",\\\"type\\\": \\\"record\\\"}\"}"

The endpoint responds with a compatibility result. This can either be true or false.

If the schema is compatible, the endpoint returns a true response. For example:

{"compatible":true,"errorMessage":null,"errorLocation":null,"schema":"{\"type\":\"record\",\"name\":\"compatible\",\"fields\":[{\"name\":\"compatible\",\"type\":[\"string\"],\"default\":\"test\"}]}"}

If the schema is not compatible, the endpoint returns a false response. This response contains an error message as well as the location where the error was encountered. For example:

{"compatible":false,"errorMessage":"reader union lacking writer type: STRING","errorLocation":"/fields/0/type/0","schema":"{\"type\":\"record\",\"name\":\"compatible\",\"fields\":[{\"name\":\"compatible\",\"type\":[\"string\"],\"default\":\"test\"}]}"}

Streams Messaging Manager

The SMM API now hides email notifier SMTP passwords in its response
Previously, the /notifiers endpoint returned the full configuration of the notifier. In the case of email notifiers, the configuration included the password of the SMTP server. API responses from now on do not include the password. As result of this change, the PASSWORD field of existing email notifiers is left blank when you edit them. If you decide to edit the notifier you must reenter the password.

Streams Replication Manager

The SRM Service role tries to recover automatically if errors are encountered
The SRM Service role might encounter errors that make metrics processing impossible. An example of this is when the target Kafka cluster is not reachable. If such an error is encountered, the SRM Service role now tries to recover automatically. If recovery is successful, the SRM Service role continues to monitor replications and displays as healthy in Cloudera Manager. However, during recovery, until the recovery is successful, the role displays as unhealthy.
New health tests for the SRM Service role
New health tests are introduced for the SRM Service role. These health tests describe the state of the SRM Service role. If the SRM Service role encounters an error that makes metrics processing impossible, Cloudera Manager now correctly displays the SRM Service role as unhealthy.
The wait time before starting new connectors is now configurable
A new configuration property, connect.start.task.timeout.ms, is added. This property controls the timeout of the tasks executed when starting connectors. The default value of the property is 20000 ms. You can configure the property on a replication level through the Streams Replication Manager's Replication Configs Cloudera Manager property. For example: 
[***ALIAS***]->[***ALIAS***].connect.start.task.timeout.ms=25000
Custom lists of supported/excluded cipher suites and TLS/SSL protocols are configurable for the SRM Service role
A number of new properties related to TLS/SSL are introduced for the SRM Service role. These properties allow users to customize which cipher suites and TLS/SSL protocols should be supported or excluded by the SRM Service role. The properties added are as follows:
  • Supported SSL/TLS Cipher Suites

    (streams.replication.manager.ssl.supportedCipherSuites)

  • Excluded SSL/TLS Cipher Suites

    (streams.replication.manager.ssl.excludedCipherSuites)

  • Supported SSL/TLS Protocols

    (streams.replication.manager.ssl.supportedProtocols)

  • Excluded SSL/TLS Protocols

    (streams.replication.manager.ssl.excludedProtocols)