Fixed Issues in Apache Spark
Review the list of Spark issues that are resolved in Cloudera Runtime 7.1.6.
- CDPD-372: YARN aggregation job is missing YARN metric folders because of timezone issues
- In this release, the Spark Atlas Connector produces a spark_application entity for each Spark job. Each data flow produced by the job creates a spark_process entity in Atlas, which tracks the actual input and output data sets for that process.
- CDPD-18458: [SPARK-32635] When pyspark
When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result.
Apache patch information
No additional Apache patches.