New Features in CDH 6.1.0

There are no notable new features in this release.

There are no notable new features in this release.

There are no notable new features in this release.

The following are some of the notable new features in this release of Hive:

There are no notable new features in this release.

The following is a list of noteworthy improvements made in Impala in resource management and admission control.

A new query option, SHUFFLE_DISTINCT_EXPRS, controls the shuffling behavior when a query has both grouping and distinct expressions.

• Incremental Stats

  The following enhancements improve Impala stability. The features reduce chances of having catalogd and impalad crash due to be out of memory when using incremental stats.

  • Incremental stats are now compressed in memory in catalogd, reducing memory footprint in catalogd.
  • Incremental stats are fetched on demand from catalogd by impalad coordinators. This enhancement reduces memory footprint for impalad coordinators and statestored and also reduces network requirements to broadcast metadata.

  See impala_perf_stats.html#pull_incremental_statistics for details.

• Automatic Invalidation of Metadata

  To keep the size of metadata bounded and to reduce the chances of catalogd cache running out of memory, this release introduces an automatic metadata invalidation feature with time-based and memory-based invalidation.

  Automatic invalidation of metadata provides more stability with lower chances of running out of memory, but could potentially cause performance risks. The feature is turned off by default.

  See impala_config_options.html#auto_invalidate_metadata for details.

• Additional separators are supported between date and time in default TIMESTAMP format, specifically, the multi-space separator and the 'T' separator. See TIMESTAMP Data Type for more information on TIMESTAMP format.

• New hint placement is supported for INSERT statements. See Optimizer Hints in Impala for detail.

• The REGEX_ESCAPE() function was implemented for escaping special characters to treat them literally in string literals.

• SHOW CREATE VIEW was implemented with the same functionality as SHOW CREATE TABLE.

• The SHUTDOWN Statement SQL command was implemented for a graceful shutdown of Impala.
• A query can contain multiple DISTINCT operators.
• Impala Shell can connect directly to impalad when configured with proxy load balancer and Kerberos. See impala-shell Configuration Options for the new flag that enables the direct connection.
• Impala can read and write data in Azure Data Lake Storage Gen2.

  By default, TLS is enabled when ADLS Gen2 is accessed via HTTP and HTTPS.

The Kafka version in CDH 6.1.0 is based on Apache Kafka 2.0.0.

For upstream release notes, see Apache Kafka version 1.0.2, 1.1.0, 1.1.1, and 2.0.0 release notes.

A high number of new metrics are introduced for Kafka. The following list is only a summary, for full list of metrics, see Metrics Reference.

Broker Metrics related to the following:

• Controller State
• Global Partition Count
• Global Topic Count
• Kafka Log Cleaner
• Auto Leader Balance Rate and Time
• Controlled Shutdown Rate and Time
• Controller Change Rate and Time
• ISR Change Rate and Time
• Leader and ISR Response Received Rate and Time
• Log Dir Change Rate and Time
• Manual Leader Balance Rate and Time
• Partition Reassignment Rate and Time
• Topic Change Rate and Time
• Topic Deletion Rate and Time

Broker Topic Metrics related to the following:

• Fetch Message Conversion
• Produce Message Conversion
• Incoming Replication rate
• Outgoing Replication Rate
• Total Fetch Requests per Second
• Total Produce Requests per Second

Replica Metrics related to the following:

• Failed ISR Updates
• Offline Replica Count
• Under Min ISR Partition Count

As of CDH 6.1.0, Cloudera officially supports Kafka clusters with nodes using JBOD configurations.

Dependencies for third-party libraries containing security vulnerabilities are updated. Kafka in CDH 6.1.0 is shipped with third-party libraries that do not contain any known security vulnerabilities.

The properties required for enabling remote JMX authentication on Kafka brokers are available in Cloudera Manager. Users are no longer required to carry out setup through a command line interface.

KAFKA-7050: The default value for request.timeout.ms is decreased to 30 seconds. In addition, a new logic is added that makes the JoinGroup requests ignore this timeout.

• The top-level <highlighting> element in solrconfig.xml is now officially deprecated in favor of the equivalent <searchComponent> syntax. This element has been out of use in default Solr installations for several releases already.
• Shard and cluster metric reporter configuration now require a class attribute.
  • If a reporter configures the group="shard" attribute, also configure the class="org.apache.solr.metrics.reporters.solr.SolrShardReporter" attribute.
  • If a reporter configures the group="cluster" attribute, also configure the class="org.apache.solr.metrics.reporters.solr.SolrClusterReporter" attribute.

  See Shard and Cluster Reporters in the Apache Solr Reference Guide for more information.

• The default value of autoReplicaFailoverWaitAfterExpiration, used with the AutoAddReplicas feature, has increased to 120 seconds from the previous default of 30 seconds. This affects how soon Solr adds new replicas to replace the replicas on nodes that have either crashed or shutdown.
• The default Solr log file size have been raised to 32MB, and the number of backups is now 10. See Configuring Logging for more information on how to change this default logging configuration.
• The eDisMax parser, by default, doesn't allow subqueries that specify a Solr parser using either local parameters, or the older _query_ magic field trick.

  For example, {!prefix f=myfield v=enterp} or _query_:"{!prefix f=myfield v=enterp}" are not supported by default. If you want to allow power-users to do this, set uf=* query or some other value that includes _query_.

  If you need full backwards compatibility for the time being, use luceneMatchVersion=7.1.0 or an earlier version.

• In the XML query parser (defType=xmlparser or {!xmlparser …​ }), the resolving of external entities is now disallowed by default.

• Configuring slowQueryThresholdMillis now logs slow requests to a separate file named solr_slow_requests.log. Earlier, slow requests were logged in the solr.log file.
• In the leader-follower model of scaling Solr, a follower no longer commits an empty index when a completely new index is detected on the leader during replication. To resume earlier behavior, pass false to skipCommitOnMasterVersionZero in the follower section of replication handler configuration, or pass it to the fetchindex command.
• Collections created without specifying a configset name use a copy of the _default configset since Solr 7.0. Before 7.3, the copied configset was named the same as the collection name. From 7.3 onwards, it is named with a new ".AUTOCREATED" suffix to prevent overwriting custom configset names.
• The rq parameter used with Learning to Rank rerank query parsing no longer considers the defType parameter. See Running a Rerank Query for more information about this parameter.
• Replicas that are not up-to-date are no longer allowed to become leader. Use the FORCELEADER command of the Collections API to allow these replicas become leader.
• The behaviour of the autoscaling system now pauses all triggers from execution between the start of actions and the end of a cool down period. The triggers will resume after the cool down period expires. Earlier, the cool down period was a fixed period that started after actions for a trigger event got completed. During this time all triggers continued to run, but any events were rejected and tried later.
• Starting a query string with local parameters {!myparser …​} is used to switch from one query parser to another. It is intended to be used by Solr system developers, not end users doing searches. To reduce negative side-effects of unintended hackability, Solr now limits the cases when local parameters is parsed to contexts in which the default parser is "lucene" or "func".
  • If defType=edismax, q={!myparser …​} doesn’t work. In this example, put the desired query parser into the defType parameter.
  • If deftype=edismax, hl.q= {!myparser …​} doesn’t work. In this example, either put the desired query parser into the hl.qparser parameter or set hl.qparser=lucene.
• The feature to add replicas automatically if a replica goes down was earlier available only when storing indexes in HDFS. It has been ported to the autoscaling framework, and AutoAddReplicas is now available to all users even if their indexes are on local disks.
• Changing the autoAddReplicas property from disabled (false) to enabled (true) using MODIFYCOLLECTION API no longer replaces down replicas for the collection immediately. Instead, replicas are only added if a node containing them went down while AutoAddReplicas was enabled. The parameters autoReplicaFailoverBadNodeExpiration and autoReplicaFailoverWorkLoopDelay are no longer used.
• All Stream Evaluators in solrj.io.eval have been refactored to have a simpler and more robust structure. This simplifies and condenses the code required to implement a new Evaluator and makes it much easier for evaluators to handle different data types (primitives, objects, arrays, lists, and so forth).

The CREATE and REFRESH (Impala only) privileges have been introduced to allow users to create databases, tables and functions, and to execute commands that update metadata information on Impala databases and tables.

For more information about the new privileges, see Sentry Privileges.

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the equivalent of the ALL privilege on the object.

In CDH 6.1.0, object ownership is enabled by default with a new CDH installation. For information about enabling and using object ownership, see Object Ownership.

Sentry no longer normalizes group name characters to be lowercase. Therefore, operating system group names do not need to be treated as case insensitive. In previous versions, Sentry modified capital letters in operating system group names to be lowercase.

This feature implements the --hbase-null-incremental-mode option for the sqoop-import tool, which allows users to specify how NULL column updates are handled during incremental imports. For more information, see Importing Data Into HBase Using Sqoop.

This feature implements the --delete-compile-dir option for the sqoop-import tool, which enables users to automatically delete the generated class and jar files from the disk after the job finishes.

By default all temporary files generated by the ClassWriter are left behind on disk in the /tmp/sqoop-username/compile directory. Because the table schema can be extracted from these files, Cloudera recommends that you use the --delete-compile-dir option to delete these files.

Support for the Hadoop API based implementation for importing data into Parquet has been added. This feature implements a new option, --parquet-configurator-implementation, which allows users to specify which implementation used for importing data into Parquet files. For more information, see Importing Data into Parquet Format Using Sqoop.

This feature implements support for importing data into Hive through HiveServer2.

The feature does not introduce any changes to the default behavior of Hive imports. When the user specifies the --hs2-url option, commands are sent to HiveServer2 through a JDBC connection. The data itself is not transferred via the JDBC connection. It is written directly to HDFS and moved to the Hive warehouse using the LOAD DATA INPATH command just like in the case of the default Hive import.

HiveServer2 provides proper Sentry authorization. As a result, Cloudera recommends importing data into Hive through HiveServer2 instead of the default method. Currently, Sqoop can authenticate to HiveServer2 using Kerberos only.

For more information, see Importing Data into Hive with Sqoop Through HiveServer2.

Sqoop now supports import from RDBMS into Amazon S3 exploiting the capabilities of the Hadoop-Amazon Web Services integration. For more information about the Hadoop-AWS module, see Hadoop-AWS module: Integration with Amazon Web Services.

Support to specify a default precision and scale to be used in the avro schema when a table contains numeric data in Oracle, or numeric or decimal data in Postgres, has been added. This feature implements two new properties, sqoop.avro.logical_types.decimal.default.precision and sqoop.avro.logical_types.decimal.scale to specify the default precision and scale. For more information about Importing Avro in Sqoop, see Importing Avro Data Files in Sqoop.