New Features in CDH 6.2.0

There are no notable new features in this release.

The following are some of the new features in this release of HDFS:

There are no notable new features in this release.

The following are some of the notable new features in this release of YARN:

There are three pre-upgrade tools that help you to validate HBase compatibility when upgrading a CDH 5 cluster to CDH 6:

• hbase pre-upgrade validate-dbe and hbase pre-upgrade validate-hfile: These tools validate that none of your tables or snapshots uses the PREFIX_TREE Data Block Encoding.
• hbase pre-upgrade validate-cp: This tool validates that your co-processors are compatible with the upgrade.

When you are attempting to upgrade from a CDH 5 cluster to a CDH 6 cluster checkboxes appear to ensure that you have performed all the HBase related pre-upgrade migration steps. For more information, see Migrating Apache HBase Before Upgrading to CDH 6.

If Lily HBase NRT Indexer Service is used, Cloudera recommends not to use HBase Serial Replication as it causes additional delays to propagate updates to Solr.

These two engines can be configured only in Cloudera Manager using safety valve. For more information, see Configuring the Off-heap BucketCache.

The following are some of the notable new features in this release of Hive:

There are no notable new features in this release.

Now when you are using Tez as the query execution engine for Hive, jobs are displayed in the Hue Job Browser. The query ID is printed and query progress is displayed.

For more information about this improvement, see the Hue blog.

Graphical displays of Impala SQL query profiles have been enhanced with greater detail. This added information helps you understand where and why query bottlenecks occur and how to optimize your queries to eliminate them. For example, detailed information is now provided about CPU processing time and network or disk I/O time for each node of query execution:

In addition, click the header of the pane to open a details panel:

To access this feature:

1. Run a query in the Hue Impala editor.
2. From the menu on the left, launch the Job Browser.
3. In the Job Browser, select the Queries tab.
4. In the list of queries, click on the query that you just ran to launch the graphical display of the query.

For more information about this new feature, see the Hue blog.

• Remote File Handle Cache

  Impala can now cache remote HDFS file handles when the ‑‑cache_remote_file_handles impalad flag is set to true. This feature does not apply to non-HDFS tables, such as Kudu or HBase tables, and does not apply to the tables that store their data on cloud services such as S3 or ADLS.

  See Scalability Considerations for File Handle Caching for file handle caching in Impala.

• Admission Debug page is available in Impala Daemon web UI at /admission and provides the following information about Impala resource pools.
  • Pool configuration
  • Relevant pool stats
  • Queued queries in order of being queued (local to the coordinator)
  • Running queries (local to this coordinator)
  • Histogram of the distribution of peak memory usage by admitted queries
• A new query option, NUM_ROWS_PRODUCED_LIMIT, was added to limit the number of rows returned from queries.

  Impala will cancel a query if the query produces more rows than the limit specified by this query option. The limit applies only when the results are returned to a client, e.g. for a SELECT query, but not an INSERT query. This query option is a guardrail against users accidentally submitting queries that return a large number of rows.

• Automatic Metadata Sync using Hive Metastore Notification Events

  When enabled, the catalogd polls Hive Metastore (HMS) notifications events at a configurable interval and syncs with HMS. You can use the new web UI pages of the catalogd to check the state of the automatic invalidate event processor.

  Note: This is a preview feature in CDH 6.2, and you should not use the feature without the guidance of Cloudera Support. If you are interested in using the feature, file a support ticket and work through Cloudera Support.

• Impala can now read the TIMESTAMP_MILLIS and TIMESTAMP_MICROS Parquet types.

  See Using the Parquet File Format with Impala Tables for the Parquet support in Impala.

• The LEVENSHTEIN string function is supported.

  The function returns the Levenshtein distance between two input strings, the minimum number of single-character edits required to transform one string to another.

• The IF NOT EXISTS clause is supported in the ALTER TABLE statement.
• Extended or verbose EXPLAIN output includes the following new information for queries:
  • The text of the analyzed query that may have been rewritten to include various optimizations and implicit casts.
  • The implicit casts and literals shown with the actual types.
• CPU resource utilization (user, system, iowait) metrics were added to the Impala profile outputs.

The REFRESH AUTHORIZATION statement was implemented for refreshing authorization data.

The Kafka version in CDH 6.2.0 is based on Apache Kafka 2.1.0.

A number of new metrics are introduced for Kafka. The following list is only a summary, for full list of metrics, see Metrics Reference.

As of CDH 6.2.0, Cloudera supports delegation token based authentication on Kafka clusters. Delegation token based authentication is a lightweight authentication method designed to complement existing SASL authentication. Although Kafka delegation tokens make use of the SCRAM authentication model, SCRAM is not supported. For more information, see Kafka Delegation Tokens.

Broker IDs can now be easily viewed in Cloudera Manager. To view broker IDs select the Kafka service and go to Instances. The broker IDs can be found next to each Kafka broker instance enclosed in brackets.

A new, more flexible logical type API has been introduced in parquet-mr—based on the Thrift field in parquet-format, which has been available for a while. The new API allows storage and retrieval of different type attributes, for example, timestamp semantics and precision.

The new logical types are represented by the LogicalTypeAnnotation class and are entirely forward- and backward-compatible with the previous logical types. Files written using the old API can be read using the new API, and as long as no new types are used, files written using the new API can also be read using the old API.

The Spark service collects Spark driver logs when Spark applications are run in YARN-client mode or with the Spark Shell. This feature is enabled by default, and the logs are persisted to an HDFS directory and included in YARN Diagnostic Bundles.

To disable this feature, uncheck Persist Driver Logs to Dfs on the Configuration page of your Spark service.

For more information, see Collecting Spark Driver Logs.

The Spark structured streaming reference application is a project that includes sample applications that demonstrate an Apache Kafka -> Apache Spark Structured Streaming -> Apache Kudu pipeline for ingestion. The main goal of the project is to aid customers in building a structured streaming application on CDH. For more information, visit Spark Structured Streaming Reference Application for CDH on GitHub.

Sqoop now supports the import and export of DECIMAL type correctly for both HDFS and Hive import. This feature is turned on by default in new CDH 6.2 clusters. It is turned off by default in older (upgraded) clusters.

Learn more at Configuring ADLS Access Using Cloudera Manager.

CDH 6.2 supports using Apache Sqoop with both generations of ADLS. You can use Sqoop to efficiently transfer bulk data between Apache Hadoop and structured datastores such as relational databases. You can use Sqoop to import data from any relational database that has a JDBC adaptor such as SQL Server, MySQL, and others, to the ADLS file system.

For more information, see Importing Data into Microsoft Azure Data Lake Store Using Sqoop.