What's New in Cloudera Manager 7.6.0

Track swap rate vs. total swap used

Added a new health test that tracks swap rate, to reveal information that swap usage alone does not convey. The health test can be enabled by configuring Swap Memory Rate Thresholds.

Zeppelin support for Livy 3 / Spark3 interpreter

Added an additional Livy 3 (for spark3) interpreter. There was a need for an interpreter to run Spark 3 jobs. If a cluster had both Spark2 and Spark3 configured, Zeppelin user could use only one.

Toggle metrics collection without a restart

For most roles, changes to configuration parameters "Enable Metric Collection" and "Metric Filter" can now also be applied with a configuration refresh, without restarting the role.

Support metrics collection from secure endpoints

Custom Service Descriptors can specify that the Cloudera Manager Agent host certificate is to be used for the purpose of TLS client verification when collecting metrics. Existing Custom Service Descriptors are unaffected by the change.

Streams Messaging Manager should authenticate to Streams Replication Manager Service

Streams Messaging Manager now automatically configures Basic Authentication when connecting to Streams Replication Manager and the service dependency based auto-configuration is in use. For manual Streams Replication Manager connectivity configurations, Basic Auth configurations were added (Streams Replication Manager Basic Authentication, Streams Replication Manager Basic Authentication Username, Streams Replication Manager Basic Authentication Password).

Agent should run SS command only over IPv4

Cloudera Manager Agent uses 'ss' command to detect port conflicts. This command fails with a segmentation fault when IPv6 is disabled on the host and causes error messages to flood in the logs. This fix resolves the issue by limiting 'ss' to obtain only IPv4 info. for port detection.

Add support for JSON schema type in the registry configuration template

Schema Registry now supports Avro and JSON schemas.

Make Streams Messaging Manager Cache-Control part of default Streams Messaging Manager REST Server API's responses' headers

New Streams Messaging Manager configuration property has been added: "cache.control.http.response.header.value". This configuration allows you to configure the Cache-Control header's value for certain endpoints. Configure it in the following key-value like fashion:

The key is the path prefix to the endpoints where the Cache-Control header should be added.
The value is the value of Cache-Control header. In order to turn off functionalities provided by the Cache-Control header just delete the entries, or set the value to ""no-store"". To disable caching, set the value of the above mentioned configurations to "no-store"

Co-located Kafka configuration uses full Streams Replication Manager principal name instead of short

When the Streams Replication Manager Co-located Kafka Cluster Alias configuration is used to auto-configure the connection to the co-located Kafka cluster, and Kerberos is enabled, the JAAS configuration is dynamically generated on each host. As a result, you can now use the service dependency method to define a Kerberos enabled co-located cluster.

Create Streams Replication Manager Service Basic Auth service user automatically, export it with a dependency extension

Streams Replication Manager automatically creates a Basic Authentication credential for co-located services (users can change the credentials using Streams Replication Manager Service Co-Located Service Username and Streams Replication Manager Service Co-Located Service User Password). When Basic Authentication is enabled, this user is automatically accepted by Streams Replication Manager Service. For more information, see Configuring Basic Authentication for the Streams Replication Manager Service.

Create default Ranger repo for both DataHub and on-prem

A new service type was added to Ranger: kafka-connect. When Ranger is installed it creates the default policies for all services, so it also needs to create the default cm_kafka_connect policy which grants the default access.

Kafka Connect Ranger plugin setup and integration in CSD

If Ranger is enabled, the kafka-connect ranger plugin will be enabled and authorization will work through Ranger policies.

Add the emit.hearbeats.enabled config to Streams Replication Manager Driver

As a result of the rebase to Kafka 2.8 (KAFKA-10710), an improvement is introduced in connection with heartbeat emission. From now on you can fine tune your deployment and fully deactivate any unnecessary replications that are set up by default by configuring heartbeat emission. This can help with minimizing any performance overhead caused by unnecessary replications. To support this change, an improvement was made for the Streams Replication Manager service in Cloudera Manager. A dedicated configuration property, Enable Heartbeats, is introduced. You can use this property to configure emit.heartbeats.enabled on a global level directly in Cloudera Manager. Replication level overrides are still supported. This can be done by adding emit.heartbeats.enabled with a valid replication prefix to Streams Replication Manager's Replication Configs. For more information on configuring heartbeat emission, see Configuring Streams Replication Manager Driver heartbeat emission.

Implement Update Certificate Screen

A new Update Auto-TLS Truststore Certificate dialog box has been added to the Cloudera Manager Security page. You can use that dialog box to replace certificates in the truststore when Auto-TLS is enabled.

Add "http.metrics.reporter.filter" config to Kafka Csd

"kafka.http.metrics.reporter.exclude.filter": Complies the regex that is provided in the Config, and metrics that match the regex won't be reported by Cloudera Manager, and because of this won't be shown by Streams Messaging Manager either. The upstream-compatible JMX names are used for this filtering. Suggested default: "^kafka.log.Log.*". These metrics are not shown in Cloudera Manager or Streams Messaging Manager by default.

Enable setting offset in Schema Registry DB

Schema Registry offset ranges can be configured via Cloudera Manager: minimum and maximum value can be set.

New UpdateGlobalTruststore command in Cloudera Manager that can replace certificates in the truststore when Auto-TLS is enabled

Once Auto-TLS is enabled, Cloudera Manager lacked a way to let users add, remove, or replace certificates from the truststore. Customers may want to manage the CA certificates across all cluster nodes. You can now run a Cloudera Manager API call to upload certificates.

In order to run this command, call the new UpdateGlobalTruststore Cloudera Manager API command under ClouderaManagerResource and upload the new root certificate authority bundle. The new command updates the global truststore files and distributes them to each agent host. Ensure that you upload the entire truststore that contains all the root certificate authorities you want Cloudera Manager to be aware of.

Restart the service or cluster after running the command. The Cloudera Manager Agents restart after the command finishes running.

Support Cruise Control metric reporter in Kafka

Cruise Control introduces a new metrics reporter in addition to the existing "Cloudera Manager metrics reporter". The "Cruise Control metrics reporter" can be selected using the metric.reporter configuration property of Cruise Control. The upgraded clusters are going to use the "CM metrics reporter" by default, but the newly created ones will have the "Cruise Control metrics reporter". This parameter can be modified manually.

When Ranger authentication is enabled, then the Cruise Control metrics reporter topic and principal names are specific. If any of them changed, the Ranger policy has to be modified too. This is also needed when the policy's details are changed.

Cloudera Manager should display the Knox URL for Oozie UI when Knox is enabled

When the Knox gateway is available on the cluster and its discovery is enabled for Oozie then the Web UI link of Oozie through Knox will appear among the direct links.

Zookeeper SSL/TLS support for Cruise Control

Cruise Control introduced the ability to communicate with ZooKeeper through a secured TLS channel. Cruise Control uses secure communication with ZooKeeper automatically when TLS is enabled on the cluster.

Implement a new checkbox for Oozie to disable the Oozie UI

A new checkbox is implemented on the Oozie configuration page that can be used to turn off the Oozie UI completely. This means that none of these Oozie UI resources will be available. If you are concerned about JQuery vulnerabilities, that cannot be fixed in the short term, you can use this feature to get rid of these by not exposing the Oozie UI.

Cloudera Manager now supports rolling restarts of HA-enabled Schema Registry

The Schema Registry service can now be restarted using rolling restart in Cloudera Manager.

Include HBCK metrics for HBase

The following HBase metrics have been added: - orphan_regions_on_regionserver - orphan_regions_on_filesystem - inconsistent_regions - region_holes - regin_overlaps - unknown_server_regions - empty_region_info_regions

Allow users to access Kafka External Accounts in public cloud

The Limited Cluster Administrator role now has permission to access the External Accounts page, and manage a restricted set of external accounts (limited to the Kafka group).