Compatibility for Cloudera AI, Cloudera Runtime components and Data Visualization

Explore Cloudera AI and its compatibility with Cloudera Runtime components and with Cloudera Data Visualization across various versions.

important

If you switch from the Red Hat Universal Base Image (Redhat UBI image) to the security hardened image, note that the updated component versions and libraries can impact the Cloudera AI operation. Cloudera strongly recommends testing the new implementation in a non-production environment first.
For backward compatibility, you can continue using the Redhat UBI image. The security hardened image is the recommended image; however, note that it requires testing and validation.
For more information, see Security hardened Spark image migration guide.
From Cloudera Data Services on premises 1.5.5 onwards, Spark 3.4 is no longer available.
Hive Warehouse Connector (HWC) is supported on all Spark 2.x versions. However, on Spark 3.x versions, HWC is supported on Spark 3.3.2 and higher.

Cloudera AI 1.5.5

Table 1. Cloudera AI compatibility with Cloudera Runtime component details and with Cloudera Data Visualization for Security Hardened image
Cloudera Runtime Version	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes	Cloudera Data Visualization (Data tab)
7.1.7 SP 3	Spark 3.2.3 Scala 2.12.15 Python 3.9.22 Java 11.0.27	NA	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 0.14.1	1.30	8.0.2-b34
7.1.9 SP1	Spark 3.2.3 Scala 2.12.15 Python 3.9.22 Java 11.0.27	Spark 3.3.2 Scala 2.12.15 Python 3.10.17 Java 17.0.15	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 1.3.0	1.30	8.0.2-b34
7.3.1	NA	NA	Spark 3.5.1 Scala 2.12.18 Python 3.11.13 Java 17.0.15	Airflow 2.10.4 Python 3.11.13 Java 17.0.15	Iceberg 1.5.2	1.30	8.0.2-b34

Table 2. Cloudera AI compatibility with Cloudera Runtime component details and with Cloudera Data Visualization for Redhat insecure image
Cloudera Runtime Version	Spark 2.4.x	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes	Cloudera Data Visualization (Data tab)
7.1.7 SP3	Spark 2.4.7 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	NA	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 0.14.1	1.30	8.0.2-b34
7.1.9 SP1	Spark 2.4.7 Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	NA	Airflow 2.10.4 Python 3.11.12 Java 17.0.15	Iceberg 1.3.0	1.30	8.0.2-b34
7.3.1	NA	NA	NA	Spark 3.5.1 Scala 2.12.18 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.10.4 Python 3.11.13 Java 17.0.15	Iceberg 1.5.2	1.30	8.0.2-b34

Table 3. Cloudera AI compatibility with Cloudera Runtime component details and with Cloudera Data Visualization for Security Hardened image
Cloudera Runtime Version	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes	Cloudera Data Visualization (Data tab)
7.1.7 SP 3	Spark 3.2.3 Scala 2.12.15 Python 3.9.23 Java 11.0.27	NA	NA	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 0.14.1	1.31	8.0.6-b93
7.1.9 SP1	Spark 3.2.3 Scala 2.12.15 Python 3.9.23 Java 11.0.27	Spark 3.3.2 Scala 2.12.15 Python 3.10.18 Java 17.0.16	NA	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 1.3.0	1.31	8.0.6-b93
7.3.1	NA	NA	Spark 3.5.4 Scala 2.12.18 Python 3.11.13 Java 17.0.16	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 1.5.2	1.31	8.0.6-b93

Table 4. Cloudera AI compatibility with Cloudera Runtime component details and with Cloudera Data Visualization for Redhat insecure image
Cloudera Runtime Version	Spark 2.4.x	Spark 3.2.x	Spark 3.3.x	Airflow	Iceberg	Kubernetes	Cloudera Data Visualization (Data tab)
7.1.7 SP3	Spark 2.4.7 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	NA	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 0.14.1	1.31	8.0.6-b93
7.1.9 SP1	Spark 2.4.7 Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_452	Spark 3.2.3 Scala 2.12.15 Python 2.7.18 Python 3.6.8 Java 11.0.25	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.11.0 Python 3.11.14 Java 17.0.16	Iceberg 1.3.0	1.31	8.0.6-b93

Table 5. Cloudera AI compatibility with Cloudera Runtime component details and with Cloudera Data Visualization for Security Hardened image
Cloudera Runtime Version	Spark 3.3.x	Spark 3.5.x	Airflow	Iceberg	Kubernetes	Cloudera Data Visualization (Data tab)
7.1.9 SP1	Spark 3.3.2 Scala 2.12.15 Python 3.10.19 Java 17.0.17	Spark 3.5.4 Scala 2.12.18 Python 3.11.14 Java 17.0.17	Airflow 2.11.0 Python 3.11.14 Java 17.0.17	Iceberg 1.3.0	1.32	8.0.8-b39
7.3.1	NA	Spark 3.5.4 Scala 2.12.18 Python 3.11.14 Java 17.0.17	Airflow 2.11.0 Python 3.11.14 Java 17.0.17	Iceberg 1.5.2	1.32	8.0.8-b39

Table 6. Cloudera AI compatibility with Cloudera Runtime component details and with Cloudera Data Visualization for Redhat insecure image
Cloudera Runtime Version	Spark 2.4.x	Spark 3.3.x	Airflow	Iceberg	Kubernetes	Cloudera Data Visualization (Data tab)
7.1.9 SP1	Spark 2.4.8 Scala 2.11.12 Python 2.7.18 Python 3.6.8 Java 1.8.0_472	Spark 3.3.2 Scala 2.12.15 Python 2.7.18 Python 3.8.17 Java 11.0.25	Airflow 2.11.0 Python 3.11.14 Java 17.0.17	Iceberg 1.3.0	1.32	8.0.8-b39