Compatibility for Cloudera Data Engineering and Cloudera Runtime components

Learn about Cloudera Data Engineering and compatibility for Cloudera Runtime components across different versions.

important

If you switch from the Redhat UBI image to the security hardened image, note that the updated component versions and libraries can impact the Cloudera Data Engineering operation. Cloudera strongly recommends testing the new implementation in a non-production environment first.
For backward compatibility, you can continue using the Redhat UBI image. The security hardened image is the recommended image; however, note that it requires testing and validation.
For more information, see Security hardened Spark image migration guide.
From Cloudera Data Services on premises 1.5.5 onwards, Spark 3.4 is no longer available.
Hive Warehouse Connector (HWC) is supported on all Spark 2.x versions. However, on Spark 3.x versions, HWC is supported on Spark 3.3.2 and higher.

Cloudera Data Engineering version mapping

The following table lists the supported version mappings between Cloudera Data Services on premises and Cloudera Data Engineering:

Table 1. Cloudera Data Engineering version mapping
Cloudera Data Services on premises version	Cloudera Data Engineering version
1.5.5 SP4	1.27.0
1.5.5 SP3 CHF3	1.26.0-h1
1.5.5 SP3	1.26.0
1.5.5 SP2 CHF1	1.25.1-h1
1.5.5 SP2	1.25.1
1.5.5 SP1	1.24.2
1.5.5 CHF1	1.24.0-h1
1.5.5	1.24.0

Table 2. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Security Hardened
Cloudera Runtime Version	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 3.5.4 Scala 2.12.18 Python 3.11.16 Java 17.0.20	Airflow 2.11.2 Python 3.11.16 Java 17.0.20	Iceberg 1.3.0	1.34
7.3.1	Spark 3.5.4 Scala 2.12.18 Python 3.11.16 Java 17.0.20	Airflow 2.11.2 Python 3.11.16 Java 17.0.20	Iceberg 1.5.2	1.34
7.3.2	Spark 3.5.4 Scala 2.12.19 Python 3.11.16 Java 17.0.20	Airflow 2.11.2 Python 3.11.16 Java 17.0.20	Iceberg 1.5.2	1.34

Table 3. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Redhat insecure
Cloudera Runtime Version	Spark 2.4.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_504	Airflow 2.11.2 Python 3.11.16 Java 17.0.20	Iceberg 1.3.0	1.34

Table 4. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Security Hardened
Cloudera Runtime Version	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 3.3.2 Scala 2.12.15 Python 3.10.20 Java 17.0.19	Spark 3.5.4 Scala 2.12.18 Python 3.11.15 Java 17.0.19	Airflow 2.11.2 Python 3.11.15 Java 17.0.19	Iceberg 1.3.0	1.33
7.3.1	NA	Spark 3.5.4 Scala 2.12.18 Python 3.11.15 Java 17.0.19	Airflow 2.11.2 Python 3.11.15 Java 17.0.19	Iceberg 1.5.2	1.33
7.3.2	NA	Spark 3.5.4 Scala 2.12.19 Python 3.11.15 Java 17.0.19	Airflow 2.11.2 Python 3.11.15 Java 17.0.19	Iceberg 1.5.2	1.33

Table 5. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Redhat insecure
Cloudera Runtime Version	Spark 2.4.x	Spark 3.3.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_472	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.11.2 Python 3.11.15 Java 17.0.19	Iceberg 1.3.0	1.33

Table 6. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Security Hardened
Cloudera Runtime Version	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 3.3.2 Scala 2.12.15 Python 3.10.19 Java 17.0.18	Spark 3.5.4 Scala 2.12.18 Python 3.11.14 Java 17.0.17	Airflow 2.11.0 Python 3.11.15 Java 17.0.18	Iceberg 1.3.0	1.32
7.3.1	NA	Spark 3.5.4 Scala 2.12.18 Python 3.11.14 Java 17.0.18	Airflow 2.11.0 Python 3.11.15 Java 17.0.18	Iceberg 1.5.2	1.32
7.3.2	NA	Spark 3.5.4 Scala 2.12.19 Python 3.11.15 Java 17.0.18	Airflow 2.11.0 Python 3.11.15 Java 17.0.18	Iceberg 1.5.2	1.32

Table 7. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Redhat insecure
Cloudera Runtime Version	Spark 2.4.x	Spark 3.3.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_472	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.11.0 Python 3.11.15 Java 17.0.18	Iceberg 1.3.0	1.32

Table 8. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Security Hardened
Cloudera Runtime Version	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 3.3.2 Scala 2.12.15 Python 3.10.19 Java 17.0.17	Spark 3.5.4 Scala 2.12.18 Python 3.11.14 Java 17.0.17	Airflow 2.11.0 Python 3.11.14 Java 17.0.17	Iceberg 1.3.0	1.32
7.3.1	NA	Spark 3.5.4 Scala 2.12.18 Python 3.11.14 Java 17.0.17	Airflow 2.11.0 Python 3.11.14 Java 17.0.17	Iceberg 1.5.2	1.32

Table 9. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Redhat insecure
Cloudera Runtime Version	Spark 2.4.x	Spark 3.3.x	Airflow	Iceberg	Kubernetes
7.1.9 SP1	Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_472	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.11.0 Python 3.11.14 Java 17.0.17	Iceberg 1.3.0	1.32

Cloudera Data Engineering 1.5.5 SP1

Table 10. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Security Hardened
Cloudera Runtime Version	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.7 SP 3	Spark 3.2.3 Scala 2.12.15 Python 3.9.23 Java 11.0.27	NA	NA	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 0.14.1	1.31
7.1.9 SP1	Spark 3.2.3 Scala 2.12.15 Python 3.9.23 Java 11.0.27	Spark 3.3.2 Scala 2.12.15 Python 3.10.18 Java 17.0.16	NA	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 1.3.0	1.31
7.3.1	NA	NA	Spark 3.5.4 Scala 2.12.18 Python 3.11.13 Java 17.0.16	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 1.5.2	1.31

Table 11. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Redhat insecure
Cloudera Runtime Version	Spark 2.4.x	Spark 3.2.x	Spark 3.3.x	Airflow	Iceberg	Kubernetes
7.1.7 SP3	Spark 2.4.7 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	NA	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 0.14.1	1.31
7.1.9 SP1	Spark 2.4.7 Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 1.3.0	1.31

Cloudera Data Engineering 1.5.5 and 1.5.5 CHF1

Table 12. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Security Hardened
Cloudera Runtime Version	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.7 SP 3	Spark 3.2.3 Scala 2.12.15 Python 3.9.22 Java 11.0.27	NA	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 0.14.1	1.30
7.1.9 SP1	Spark 3.2.3 Scala 2.12.15 Python 3.9.22 Java 11.0.27	Spark 3.3.2 Scala 2.12.15 Python 3.10.17 Java 17.0.15	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 1.3.0	1.30
7.3.1	NA	NA	Spark 3.5.1 Scala 2.12.18 Python 3.11.13 Java 17.0.15	Airflow 2.10.4 Python 3.11.13 Java 17.0.15	Iceberg 1.5.2	1.30

Table 13. Cloudera Data Engineering compatibility with Cloudera Runtime component details for Redhat insecure
Cloudera Runtime Version	Spark 2.4.x	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes
7.1.7 SP3	Spark 2.4.7 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	NA	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 0.14.1	1.30
7.1.9 SP1	Spark 2.4.7 Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 1.3.0	1.30
7.3.1	NA	NA	NA	Spark 3.5.1 Scala 2.12.18 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.10.4 Python 3.11.13 Java 17.0.15	Iceberg 1.5.2	1.30