Fixed Issues in Spark3

Review the list of Spark3 issues that are resolved in Cloudera Runtime 7.2.18.

CDPD-60190: Backport SPARK-39441
[SPARK-39441] Speed up DeduplicateRelations
CDPD-58191: Spark - Upgrade kubernetes library to 5.7.4/5.8.1/5.10.2/5.11.2+ due to CVE-2021-4178
Upgraded kubernetes-client dependency to 5.7.4
CDPD-57535: Revert: CDPD-48171: Temporary workaround pinning snakeyaml to 2.0 not vulnerable to CVE-2022-1471
Reverted back from snakeyaml 2.0. The snakeyaml's Representer constructor has been added back. The other reverted constructors can be found here:
CDPD-56342: Upgrade Parquet to 1.12.3 in Spark
Upgraded Parquet dependency to 1.12.3
CDPD-55116: Fix Spark vulnerability CVE-2023-22946
This fix is blacklisting “spark.submit.deployMode” and “spark.submit.proxyUser.allowCustomClasspathInClusterMode” spark configurations in Livy create session REST API. We have added a new Livy configuration “livy.server.session.allow-custom-classpath” to allow custom class path. In order to disable or rollback this fix, we can add “livy.server.session.allow-custom-classpath” as “true” in Livy configuration via the CM safety valve.

Apache patch information