Spark
HDP 2.4.0 provides Spark 1.6.0 and the following Apache patches:
SPARK-11314: Add service API and test service for Yarn Cluster schedulers.
SPARK-11315: Add YARN extension service to publish Spark events to YARN timeline service.
SPARK-11323: Add History Service Provider to service application histories from YARN timeline server.
SPARK-12417: Support to have ORC bloom filters during write code path.
SPARK-12898: Consider having dummyCallSite for HiveTableScan.
SPARK-12925: Improve HiveInspectors.unwrap for StringObjectInspector.getPrimitiveWritableObject.