HDP-2.4.0 Release Notes
Also available as:

Behavioral Changes

Behavioral changes denote a marked change in behavior from the previously released version to this version of software. In HDP 2.4.0, behavioral changes affect the following Hadoop components.

Table 1.12. Behavioral Changes

Hortonworks Bug IDComponentApache JIRASummaryDetails
   Debian 6 support deprecated.

HDP support for Debian 6 is deprecated with HDP 2.4.0.

Support for Debian 6 will be removed in a future HDP release.

BUG-43020Hue CherryPy is not the default webserver

Scenario: In previous releases of Hue, CherryPy was not the default web-server, but spawning webserver, which is generally not advisable as CherryPy is multithreaded and scales better than spawning webserver.

Previous Behavior: By not commenting out use_cherrpy_server=false in hue.ini, Hue used spawning webserver instead of Hue.

New Behavior: CherryPy is the default webserver for Hue.

Workaround/Expected Customer Action: No action required, unless spawning web server is the desired webserver for Hue. In this case, modify use_cherrypy_server accordingly.

BUG-50292Hue HUE DataTables Warning in partition columns tab.

Scenario: In previous releases of Hue, an error message appeared when Partition Columns tab was clicked in the Beeswax describe table view.

Previous Behavior: When clicking the Partition Columns tab in the Beeswax describe table view, an error message appeared.

New Behavior: No error message appears.

Workaround: None.

BUG-50775YARN Enable collection of debug information by default

Scenario: Container failures during job execution.

Previous Behavior: New feature - no previous behaviour.

New Behavior: This feature(enabled by default) enables generation of additional information which is useful for debugging container failures. The extra information generated is:

  • A copy of the launch_container.sh script.

  • A listing of the contents of the container work directory.

This information is automatically collected if log-aggregation is enabled, or ignored and discarded if log-aggregation is disabled. The feature can be disabled by setting yarn.nodemanager.log-container-debug-info.enabled to false in yarn-site.xml.

BUG-50786Tez Increase the default value for MAX_COUNTERS

The default value of the max counter limit has been increased from 2000 to:

  • tez.counters.max: 10000

  • tez.counters.max.groups: 3000

BUG-51480Spark We should recommend to run spark history server on top of HDFS

In HDP 2.4, the Spark History Service reads data from HDFS. Prior to HDP 2.4, the data was read from the YARN ATS (Application Timeline Service). Refer to the Spark Guide for more info.

BUG-51481Spark Behavior Change for hdp.version for spark configuration from HDP 2.3.4.

Previous Behavior: Before HDP 2.3.4, user need to configure hdp.version variable in spark-defaults.conf:

spark.driver.extraJavaOptions -Dhdp.version=$VERSION$BUILD
spark.yarn.am.extraJavaOptions -Dhdp.version=$VERSION$BUILD

New Behavior: As of HDP 2.4.0: when running HDP, Spark users no longer need to specify hdp.version through the Spark configuration files; it is inferred from the version of HDP installed on the client machine.

There are a few exceptions to this rule:

  • If you are programmatically submitting jobs in a way that spark-env.sh is not executed during the submit step.

  • If you want to specify a different cluster version than what is installed on the client.

Ambari, Spark The Spark shuffle service is enabled by default with this release. It is necessary for Spark Dynamic Executor Allocation to work correctly.