Spark
HDP 2.4.2 provides Spark 1.6.1 and the following Apache patches:
SPARK-10582: Using dynamic-executor-allocation, if AM failed, the new AM will be started. But the new AM does not allocate executors to driver.
SPARK-11137: Make StreamingContext.stop() exception-safe.
SPARK-11182: HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode.
SPARK-11314: Add service API and test service for Yarn Cluster schedulers.
SPARK-11627: Spark Streaming backpressure mechanism has no initial rate limit, receivers receive data at the maximum speed, it might cause OOM exception.
SPARK-11969: SQL UI does not work with PySpark.
SPARK-12001: StreamingContext cannot be completely stopped if the stop() is interrupted.
SPARK-12009: Avoid re-allocate yarn container while driver want to stop all executors.
SPARK-12142: Can't request executor when container allocator is not ready.
SPARK-12241: Improve failure reporting in Yarn client obtainTokenForHBase().
SPARK-12353: Wrong output for countByValue and countByValueAndWindow.
SPARK-12513: SocketReceiver hang in Netcat example.
SPARK-12523: Support long-running of the Spark on HBase and hive metastore.
SPARK-12920: Fix high CPU usage in Spark thrift server with concurrent users.
SPARK-12925: Improve HiveInspectors.unwrap for StringObjectInspector.getPrimitiveWritableObject.
SPARK-12948: OrcRelation uses HadoopRDD which can broadcast conf objects frequently.
SPARK-12998: Enable OrcRelation when connecting via Spark thrift server.
SPARK-13021: Fail fast when custom RDD's violate RDD.partition's API contract.
SPARK-13117: WebUI should use the local IP not 0.0.0.0.
SPARK-13308: ManagedBuffers passed to OneToOneStreamManager need to be freed in non error cases.
SPARK-13360: Pyspark related environment variable is not propagated to driver in yarn-cluster mode.
SPARK-13468: Fix a corner case where the page UI should show DAG but it doesn't show.
SPARK-13478: Use real user when fetching delegation tokens.
SPARK-13599: Groovy-all ends up in spark-assembly if Hive profile set.
SPARK-13642: Properly handle signal kill in ApplicationMaster.
SPARK-13885: Fix attempt ID regression for Spark running on Yarn.
SPARK-14062: Fix log4j and upload metrics.properties automatically with distributed cache.
SPARK-6847: Stack overflow on updateStateByKey which followed by a stream with checkpoint set.
HDP 2.4.0 provided Spark 1.6.0 and the following Apache patches:
SPARK-11315: Add YARN extension service to publish Spark events to YARN timeline service.
SPARK-11323: Add History Service Provider to service application histories from YARN timeline server.
SPARK-12417: Support to have ORC bloom filters during write code path.
SPARK-12898: Consider having dummyCallSite for HiveTableScan.