What's New In CDH 5.4.x

CDH 5.4.0 adds support for RHEL and CentOS 6.6. See CDH and Cloudera Manager Supported Operating Systems.

The following summarizes new security capabilities in CDH 5.4.0:

• Secure Hue impersonation support for the Hue HBase application.
• Redaction of sensitive data from logs, centrally managed by Cloudera Manager, which prevents the WHERE clause in queries from leaking sensitive data into logs and management UIs.
• Cloudera Manager support for custom Kerberos principals.
• Kerberos support for Sqoop 2.
• Kerberos and TLS/SSL support for Flume Thrift source and sink.
• Navigator SAML support (requires Cloudera Manager).
• Navigator Key Trustee can now be installed and monitored by Cloudera Manager.
• Search can be configured to use SSL.
• Search supports protecting Solr and Lily HBase Indexer metadata using ZooKeeper ACLs in a Kerberos-enabled environment.

Developers can use Cell-based APIs in the same way as KeyValue-based APIs if they are not ready to update their code, but will probably have to change code inside DoFns because HBase 0.99 and later APIs deprecated or removed a number of methods from the HBase 0.96 API.

CDH 5.4.0 adds SSL and Kerberos support for the Thrift source and sink, and implements DatasetSink 2.0.

CDH 5.4.0 implements HBase 1.0.

Kite in CDH has been rebased on the 1.0 release upstream. This breaks backward compatibility with existing APIs. The APIs are documented at http://kitesdk.org/docs/1.0.0/apidocs/index.html.

Notable changes are:

• Dataset writers that implement flush and sync now extend interfaces (Flushable and Syncable). Writers that no longer have misleading flush and sync methods.
• DatasetReaderException, DatasetWriterException, and DatasetRepositoryException have been removed and replaced with more specific exceptions, such as IncompatibleSchemaException. Exception classes now indicate what went wrong instead of what threw the exception.
• The partition API is no longer exposed; use the view API instead.
• kite-data-hcatalog is now kite-data-hive.

• Added Spark action which lets you run Spark applications from Oozie workflows. See the Oozie documentation for more details.
• The Hive2 action now collects and reports Hadoop Job IDs for MapReduce jobs launched by Hive Server 2.
• The launcher job now uses YARN uber mode for all but the Shell action; this reduces the overhead (time and resources) of running these Oozie actions.

• Parquet memory manager now changes the row group size if the current size is expected to cause out-of-memory (OOM) errors because too many files are open. This causes a WARN message to be printed in the logs. A new setting, parquet.memory.pool.ratio, controls the percentage of the JVM's heap memory Parquet attempts to use.
• To improve job startup time, footers are no longer read by default for MapReduce jobs (PARQUET-139).
• The Parquet Avro object model can now read lists and maps written by Hive, Avro, and Thrift (similar capabilities were added to Hive in CDH 5.3). This compatibility fix does not change behavior. The extra record layer wrapping the list elements when Avro reads lists written by Hive can now be removed; to do this, set the expected Avro schema or set parquet.avro.add-list-element-records to false.
• Avro's map representation now writes null values correctly.
• The Parquet Thrift object model can now read data written by other object models (such as Hive, Impala, or Parquet-Avro), given a Thrift class for the data; compile a Thrift definition into an object, and supply it when creating the job.

• Solr metadata stored in ZooKeeper can now be protected by Zookeeper ACLs. In a Kerberos-enabled environment, Solr metadata stored in ZooKeeper is owned by the solr user and cannot be modified by other users.

  Administrators who modify Solr ZooKeeper metadata through operations like solrctl init or solrctl cluster --put-solrxml must now supply solrctl with a JAAS configuration using the --jaas configuration parameter. The JAAS configuration must specify the principal, typically solr, that the solr process uses. See solrctl Reference for more information.

  End users, who typically do not need to modify Solr metadata, are unaffected by this change.

• Lily HBase Indexer metadata stored in ZooKeeper can now be protected by Zookeeper ACLs. In a Kerberos-enabled environment, Lily HBase Indexer metadata stored in ZooKeeper is owned by the Solr user and cannot be modified by other users.

  End users, who typically do not manage the Lily HBase Indexer, are unaffected by this change.

• The Lily HBase Indexer supports restricting access using Sentry. For more information, see Configuring Lily HBase Indexer Security.
• Services included with Search for CDH 5.4.0, including Solr, Key-Value Store Indexer, and Flume, now support SSL.
• The Spark Indexer and the Lily HBase Batch Indexer support delegation tokens for mapper-only jobs. For more information, see Spark Indexing and HBaseMapReduceIndexerTool.
• Search for CDH 5.4.0 implements SOLR-5746, which improves solr.xml file parsing. Error checking for duplicated options or unknown option names was added. These checks can help identify mistakes made during manual edits of the solr.xml file. User-modified solr.xml files may cause errors on startup due to these parsing improvements.

• By default, CloudSolrServer now uses multiple threads to add documents.

  To get the old, single-threaded behavior, set parallel updates to false on the CloudSolrServer instance.

  Related JIRA: SOLR-4816.

• Updates are routed directly to the correct shard leader, eliminating document routing at the server. This allows for near linear indexing throughput scalability. Document routing requires that the solrj client must know each document’s unique identifier. The unique identifiers allow the client to route the update directly to the correct shard. For additional information, see Shards and Indexing Data in SolrCloud.

  Related JIRA: SOLR-4816.

• The loadSolr morphline command supports nested documents. For more information, see Morphlines Reference Guide.
• Navigator can be used to audit Cloudera Search activity. For more information on the Solr operations that can be audited, see Exploring Audit Data.

• Search for CDH 5.4 supports logging queries before they are executed. This allows you can identify queries that could increase resource consumption. This also enables improving schemas or filters to meet your performance requirements. To enable this feature, set the SolrCore and SolrCore.Request log level to DEBUG.

  Related JIRA: SOLR-6919

• UniqFieldsUpdateProcessorFactory, which Solr Server implements, has been improved to support all of the FieldMutatingUpdateProcessorFactory selector options. The <lst named="fields"> init param option is deprecated. Replace this option with <arr name="fieldName">.

  If the <lst named="fields"> init param option is used, Solr logs a warning.

  Related JIRA: SOLR-4249.

• Configuration information was previously available using FieldMutatingUpdateProcessorFactory (oneOrMany or getBooleanArg). Those methods are now deprecated. The methods have been moved to NamedList and renamed to removeConfigArgs and removeBooleanArg, respectively.

  If the oneOrMany or getBooleanArg methods of FieldMutatingUpdateProcessorFactory are used, Solr logs a warning.

  Related JIRA: SOLR-5264.

See also Apache Spark Known Issues and Apache Spark Incompatible Changes and Limitations.

• Sqoop 2:
  • Implements Sqoop 2 version 1.99.5.
  • Sqoop 2 supports Kerberos.
  • Sqoop 2 supports PostgreSQL as the repository database.

Problem: Because of a change in the implementation of the NameNode metadata upgrade mechanism, upgrading to CDH 5.4.1 from a version lower than 5.4.0 can take an inordinately long time. In a cluster with NameNode high availability (HA) configured and a large number of edit logs, the upgrade can fail, with errors indicating a timeout in the pre-upgrade step on JournalNodes.

What to do:

To avoid the problem: Do not upgrade to CDH 5.4.1; upgrade to CDH 5.4.2 instead.

If you experience the problem: If you have already started an upgrade and seen it fail, contact Cloudera Support. This problem involves no risk of data loss, and manual recovery is possible.

If you have already completed an upgrade to CDH 5.4.1, or are installing a new cluster: In this case you are not affected and can continue to run CDH 5.4.1.

• Beginning with CDH 5.4.1, Search for CDH supports configurable transaction log replication levels for replication logs stored in HDFS. For more information, see the Transaction Log Replication section in Replication.

• Spark supports submitting python applications in cluster mode.

Problem: Because of a change in the implementation of the NameNode metadata upgrade mechanism, upgrading to CDH 5.4.1 from a version lower than 5.4.0 can take an inordinately long time. In a cluster with NameNode high availability (HA) configured and a large number of edit logs, the upgrade can fail, with errors indicating a timeout in the pre-upgrade step on JournalNodes.

What to do:

To avoid the problem: Do not upgrade to CDH 5.4.1; upgrade to CDH 5.4.2 instead.

If you experience the problem: If you have already started an upgrade and seen it fail, contact Cloudera Support. This problem involves no risk of data loss, and manual recovery is possible.

If you have already completed an upgrade to CDH 5.4.1, or are installing a new cluster: In this case you are not affected and can continue to run CDH 5.4.1.

Problem: During a rolling upgrade to any of the releases listed below, the NameNode may report missing blocks after rolling back multiple DataNodes. This is caused by a race condition with block reporting between the DataNode and the NameNode. No permanent data loss occurs, but data can be unavailable for up to six hours before the problem corrects itself.

Releases affected: CDH 5.0.6, 5.1.5, 5.2.5, 5.3.3, 5.4.1, 5.4.2

What to do:

To avoid the problem: Cloudera advises skipping the affected releases and installing a release containing the fix. For example, do not upgrade to CDH 5.4.2; upgrade to CDH 5.4.3 instead.

The releases containing the fix are: CDH 5.3.4, 5.4.3

If you have already completed an upgrade to an affected release, or are installing a new cluster: You can continue to run the release, or upgrade to a release that is not affected.