What's New in Cloudera Manager 7.6.2

Merged keytab with load balancer principal for Kafka service.

Configuration options have been added to use load balancer in front of kafka brokers.

Validate HBase replication setup after first time setup is done

When an HBase replication policy is created then there is a new field available in the API body validateReplicationSetup:
By default, its value is false, which doesn't make any difference: HBase replication policy creation will be the same as before.
When it's set to true, then in the end of the HBase replication policy creation, additional steps will check if replication works properly between the source and destination. HBase policy creation will work the same way as before, no additional action is required by users.
If you want to validate whether HBase replication works properly, use that new field in the API body: "validateReplicationSetup": true

Enable JMX Authentication by default

In Kafka, JMX authentication is enabled by default. Passwords for monitor and control users are generated by default, but if values are set, they will be the values used.

Cloudera Manager client support for hadoop.security.group.mapping.ldap.bind.password in jceks

With this change, the 'ldap.bind.password' parameter will be appended to core-site.xml. This will enable HDFS's clients (like YARN) to access LDAP functionalities.

Automatically refresh metric filter and collection settings

A new parameter 'metric_config_auto_refresh' has been added. By setting this parameter to true, Metric Collection and Metric Filter parameters will be set automatically if they are changed without a role restart or configuration refresh.

Option for logout landing page in the Cloudera Manager Admin Console

With SAML, the logout request re-initiates a new SAML authentication request, therefore the session wasn't terminated unless SAML SLO (Single-LogOut) is enabled. We added a logout success page to break the SAML logout loop which has a Return to Login Page button for new session creation.

If Single-logout (SLO) is supported by the identity provider (IdP), you can enable this feature by turning on SLO in the SAML configuration during SAML set up, .

Configurations for Streams Replication Manager explicit topic creation

Streams Replication Manager now supports user-configured values for partition count and/or replication factor of it's internal topics. The following configuration parameters were introduced: metrics.topic.partition.count, control.topic.replication.factor, streams.replication.manager.service.remote.advertisement.topic.replication.factor, streams.replication.manager.service.streams.replication.factor.

HbaseReplicationFirstTimeSetupReset removes the HDFS files from both sides, and clears the aux entry.

Introduced a new API endpoint: hbaseReplicationFirstTimeSetupCleanAndReset, which cleans up and resets the HBase replication first time setup between the given source and destination:

Removes the jceks file

Clears the HBASE_REPLICATION_AUXILIARY_INFO configuration

Add support for JSON schema type in the registry config template s

Schema Registry now supports Avro and JSON schemas.\

Add an optional "Cluster Alias" field to the existing Kafka external account type

The Kafka External Account name no longer has to match the alias of the Kafka cluster this account is describing. A new "Cluster Alias" field has been introduced for Kafka External Accounts, providing users with freedom to name the accounts as they wish. This field is optional - if it is specified, its value will be used when connecting to the Kafka cluster, otherwise the account name will be used.

Need to update default value for keystore alias in Ranger

Default keystore alias is now configurable for Ranger Admin and is not set to the Hostname by default.

Chive: add an option to ignore certain partitionParameters when comparing and improve location comparison

When customer replicated Hive data, then the replication had poor performance because all partitions were recreated.

This fix introduces a new parameter called HIVE_IGNORED_PARTITION_PARAMETERS for hive_replication_env_safety_valve. The value is a comma separated list of Hive partition parameters that will not be compared during the import stage. This means that even if these partition parameters don't match between the exported and existing partitions, the partition will not be dropped and recreated. Since these parameters can easily differ for metadata only replications, it's safe to ignore them in those cases. By setting these parameters, Hive replication performance can be improved.

OPSAPS-62697 Need to add property javax.security.auth.useSubjectCredsOnly to JVM args

Users having issues with Atlas - Solr communication should add the required argument through the Atlas Service Environment Advanced Configuration Snippet (Safety Valve) configuration parameter. Use the following key and value:

Key: ATLAS_CUSTOM_OPTS

Value: -Djavax.security.auth.useSubjectCredsOnly=false

Kafka topic entity import into Atlas

The Import Kafka Topics Into Atlas command has been introduced. It is available from the actions list of the Kafka service or any of the brokers in Cloudera Manager. The command can be used by users who have permissions to handle service configurations in the system. The Atlas service is required for the command, otherwise the process will fail. When "not Ranger" is preferred as default authorizer, then the "kafka" service user has to be defined in the selected authorizer service.

Add "emit.consumer.metrics" config to SMM CSD, and remove (now) unused SMON host/port configs.

The cm.metrics.service.monitor.host" and "cm.metrics.service.monitor.port" Streams Messaging Manger configuration properties have been removed. These properties are no longer required because Streams Messaging Manager automatically detects the ServiceMonitor's location.

The following new configuration parameter for Streams Messaging Manager has been added: "emit.consumer.metrics": When this is set to false, Streams Messaging Manager does not emit historic ConsumerGroup metrics into the ServiceMonitor, meaning that historic metrics (for group Lag and CommittedOffset) are not available for Groups in Streams Messaging Manager. These metrics are used to populate the charts at the bottom of the ConsumerGroupDetail page, or accessed via the api/v2/admin/metrics/consumers/group/{groupId} REST API endpoint.

Allow HTTP Response Headers to be Configured for Kafka Connect

When a request was made to Kafka connect, the response did not contain a HSTS header. With this, the HSTS header has been added as default to the Kafka Connect REST API response.

Enable setting offset in Schema Registry database

Schema Registry offset ranges can now be configured via Cloudera Manager: minimum and maximum value can be set.

Cloudera Manager diagnostic bundle now includes Cloudera Manager database information

A Cloudera Manager diagnostic bundle will now include an additional file named cm_db_dump_stats.txt, in the cm_db_dump directory of the generated bundle. This file contains statistical data of the Cloudera Manager database dump thread responsible for collecting table data. The purpose of this is to help tally the following:

Number of tables in the database
Total collected tables - Number of records of each table and how many were collected
Size of the records and size of the collection
Status of the collection (in-progress or completed)
Time taken to collect each table

Introduce allowed nexus urls config for Kafka Connect

Users are now able to configure which nexus URLs are allowed for Stateless Nifi connector configurations using the kafka.connect.allowed.nexus.urls property. By default, this is set to empty, which means to "allow everything". During connector creation / modification / validation, the nexus.url property will be validated against this list.

Add custom Kerberos path to agent and Cloudera Manager

Customers are facing the following issues when modifying the default path of krb5.conf in Cloudera Manager :

The credential generation for roles are failing because KDC authentication with Cloudera Manager server fails.
Services are failing to authenticate with Cloudera Manager Agent once manually getting services up by applying hacks (i.e adding relevant JVM arguments or environment variables)
A few services like HDFS, Livy, HiveServer and Knox are reportedly failing as they are unable to locate the new Kerberos path.

To set a custom path, follow the steps in this Knowledge Base article: How to use a custom Kerberos configuration path for a cluster running with Kerberos (MIT)

Configuration property for enabling HTTP Strict Transport Security

Fixed an issue where customers were unable to configure Cruise Control to include Strict Transport Security headers in the responses of the API. A new configuration property, webserver.ssl.sts.enabled has been added for Cruise Control in Cloudera Manager. Setting this value to true configures Cruise Control to include the Strict Transport Security header in the web server responses when SSL is enabled.

Add flags to force Connectors to override the JAAS, and restrict the usage of the Worker principal

Kafka Connect now allows users to force Connectors to override the JAAS configuration of the Kafka connection, and also forbids using the same Kerberos credentials as the Connect Worker is using.

Add OpDB Agent to Knox

Configuration for autodiscovery of the OpDB Agent has been added to Knox. The OpDB Agent is a new service for CDP Operational Database that needs to be discovered by Knox.

Ranger server work directory is now configurable

Ranger Admin / KMS / KMS-KTS server work directory can now be configured through the parameter ranger.tomcat.work.dir.

Ranger RMS server work directory can now be configured through the parameter ranger-rms.tomcat.work.dir.

Ranger Raz server work directory can now be configured through the parameter ranger.raz.tomcat.work.dir.

authzmigrator : Skipping policy item creation for {OWNER}

After the Sentry migration to Ranger there are lots of {OWNER} policies being created, which are very difficult to administer. This occurs during CDH to CDP migration. If you want to skip {OWNER} policies, add the following properties in authorization-migration-site.xml

<property>
    <name>authorization.migration.skip.owner.policy</name>
    <value>true</value>   
</property>