Fixed Issues in Spark3

Review the list of Spark3 issues that are resolved in Cloudera Runtime 7.2.18.

CDPD-60190: Backport SPARK-39441

[SPARK-39441] Speed up DeduplicateRelations

CDPD-58191: Spark - Upgrade kubernetes library to 5.7.4/5.8.1/5.10.2/5.11.2+ due to CVE-2021-4178

Upgraded kubernetes-client dependency to 5.7.4

CDPD-57535: Revert: CDPD-48171: Temporary workaround pinning snakeyaml to 2.0 not vulnerable to CVE-2022-1471

Reverted back from snakeyaml 2.0. The snakeyaml's Representer constructor has been added back. The other reverted constructors can be found here: https://bitbucket.org/snakeyaml/snakeyaml/commits/3e755d254aeaa902675053047fd53368a175565a/raw

CDPD-56342: Upgrade Parquet to 1.12.3 in Spark

Upgraded Parquet dependency to 1.12.3

CDPD-55116: Fix Spark vulnerability CVE-2023-22946

This fix is blacklisting “spark.submit.deployMode” and “spark.submit.proxyUser.allowCustomClasspathInClusterMode” spark configurations in Livy create session REST API. We have added a new Livy configuration “livy.server.session.allow-custom-classpath” to allow custom class path. In order to disable or rollback this fix, we can add “livy.server.session.allow-custom-classpath” as “true” in Livy configuration via the CM safety valve.