Known Issues in Spark

Known issues and technical limitations for Spark are addressed in Cloudera Runtime 7.3.3.

Known issues identified in Cloudera Runtime 7.3.3

There are no new known issues identified in this release.

Known issues identified before Cloudera Runtime 7.3.3

Known issues identified before Cloudera Runtime 7.3.3 include only unresolved issues from previous releases that continue to affect the Cloudera Runtime 7.3.3 base release.

CDPD-111272: MSCK REPAIR TABLE may perform redundant ALTER TABLE when partitions are already tracked
7.3.3, 7.3.2.100 CHF 1 and higher
When you run MSCK REPAIR TABLE on a Hive table, Spark may issue a redundant ALTER TABLE for partitions that are already tracked, which can increase repair time.
None.
CDPD-67806: Migration of ORC table with timestamp column to Iceberg with spark.sql.timestampType=TIMESTAMP_NTZ fails
7.3.3, 7.3.2 and its SPs and CHFs, 7.3.1.x
The Hive External Catalog converts NTZ timestamp types back to TimeStampType, which Hive does not support, leading to migration failures.
Set spark.sql.iceberg.use-timestamp-without-timezone-in-new-tables and spark.sql.iceberg.handle-timestamp-without-timezone to true to migrate ORC tables to Iceberg with timestamp columns without timezone.
CDPD-90447: Table is stored in HDFS when cluster is enabled with raz and S3 is set to default storage system
7.3.3, 7.3.2 and its SPs and CHFs
Tables created on a RAZ-enabled clusters with S3 set as the default storage system are stored in HDFS instead of the specified S3 bucket.
None. S3 was removed as the default storage.
CDPD-94393: RuntimeWarning Failed to add file" message appears even when Spark successfully loads files
7.3.3, 7.3.2 and its SPs and CHFs, 7.3.1.0 and higher
In both Spark 2 and 3, due to an exception when attempting to add files to the Python path, the RuntimeWarning: Failed to add file message appears even when the Python JAR file is successfully loaded.
None. You can safely ignore the message as the file is loaded successfully and the message does not affect job completion.
Spark 3: RAPIDS Accelerator is not available
7.3.3, 7.3.2 and its SPs and CHFs, 7.3.1.0 and higher
The RAPIDS Accelerator for Apache Spark is not available in Cloudera Runtime.
None.