Compatibility for Cloudera Data Engineering and Runtime components

Learn about Cloudera Data Engineering and compatibility for Runtime components across different versions. This document also includes component version compatibility information for AWS Graviton.

Table 1. Cloudera Data Engineering compatibility with Runtime component details
Cloudera Data Engineering Version	Spark 2.4.x	Spark 3.2.x	Spark 3.3.x	Spark 3.5.x	Airflow	Kubernetes
1.24.1 Security hardened	N/A	Spark 3.2.3 Scala 2.12.10 Java 11 Python 3.9 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.3.0 Scala 2.12.15 Java 17 Python 3.10 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.5.1 Scala 2.12.18 Java 17 Python 3.11 Iceberg 1.5.2 Data Lake 7.2.18 and 7.3.1	Airflow 2.10.4 Python 3.11 Java 17	1.31
1.24.1 Redhat UBI (deprecated)	Spark 2.4.8 Scala 2.11.12 Java 1.8.0 Python 2.7 and 3.6 Iceberg N/A Data Lake 7.2.16-7.2.18	Spark 3.2.3 Scala 2.12.10 Java 11 Python 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.3.0 Scala 2.12.15 Java 11 Python 3.8 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.5.1 Scala 2.12.18 Java 11 Python 3.8 Iceberg 1.5.2 Data Lake 7.2.18 and 7.3.1	Airflow 2.9.3 Python 3.11 Java 17	1.31
1.23.1	Spark 2.4.8 Python 2.7 and 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.2.3 Python 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.3.0 Python 3.8 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.5.1 Python 3.8 Iceberg 1.5.2 Data Lake 7.2.18 and 7.3.1	Airflow 2.9.3 Python 3.11 Java 17	1.30
1.23.0	Spark 2.4.8 Python 2.7 and 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.2.3 Python 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.3.0 Python 3.8 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.5.1 Python 3.8 Iceberg 1.5.2 Data Lake 7.2.18	Airflow 2.9.3 Python 3.11	1.30
1.22.0	Spark 2.4.8 Python 2.7 and 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.2.3 Python 3.6 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.3.0 Python 3.8 Iceberg 1.3.1 Data Lake 7.2.16-7.2.18	Spark 3.5.1 Python 3.8 Iceberg 1.4.3 Data Lake 7.2.18	Airflow 2.7 Python 3.8	1.28
1.21	Spark 2.4.8 Python 2.7 and 3.6 Iceberg 1.3.1 Data lake 7.2.16-7.2.18	Spark 3.2.3 Python 3.6 Iceberg 1.3.1 Data lake 7.2.16-7.2.18	Spark 3.3.0 Python 3.8 Iceberg 1.3.1 Data lake 7.2.16-7.2.18	N/A	Airflow 2.7 Python 3.8	1.27
1.20.3	Spark 2.4.8 Python 2.7 Iceberg 1.3.1 Data lake 7.2.15-7.2.17	Spark 3.2.3 Python 3.6 Iceberg 1.3.1 Data lake 7.2.15-7.2.17	Spark 3.3.0 Python 3.8 Iceberg 1.3.1 Data lake 7.2.15-7.2.17	N/A	Airflow 2.6.3 Python 3.8	1.27
1.19.4	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.15-7.2.17 Iceberg 1.1.0	Spark 3.2.3 Scala 2.12.10 Java 11.0.20 Python 3.6 Data lake 7.2.15-7.2.17 Iceberg 1.1.0	Spark 3.3.0 Scala 2.12.15 Java 11.0.20 Python 3.8 Data lake 7.2.15-7.2.17 Iceberg 1.1.0	N/A	Airflow 2.3.4 Python 3.8	1.26
1.19.3	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.15-7.2.17	Spark 3.2.3 Scala 2.12.10 Java 11.0.20 Python 3.6 Data lake 7.2.15-7.2.17	Spark 3.3.0 Scala 2.12.15 Java 11.0.20 Python 3.8 Data lake 7.2.15-7.2.17	N/A	Airflow 2.3.4 Python 3.8	1.25
1.19.2	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.14-7.2.16 Iceberg 1.1.0	Spark 3.2.3 Scala 2.12.10 Java 11.0.20 Python 3.6 Data lake 7.2.14-7.2.16 Iceberg 1.1.0	Spark 3.3.0 Scala 2.12.15 Java 11.0.20 Python 3.8 Data lake: 7.2.16 Iceberg 1.1.0	N/A	Airflow 2.3.4 Python 3.8	1.25
1.19	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.14-7.2.16 Iceberg 1.1.0	Spark 3.2.3 Scala 2.12.10 Java 11.0.20 Python 3.6 Data lake 7.2.14-7.2.16 Iceberg 1.1.0	Spark 3.3.0 Scala 2.12.15 Java 11.0.20 Python 3.8 Data lake: 7.2.16 Iceberg 1.1.0	N/A	Airflow 2.3.4 Python 3.8	1.25
1.18.3	Spark 2.4.8 Scala: 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.14-7.2.16 Iceberg 0.14.1	Spark 3.2.3 Scala 2.12.10 Java 11.0.20 Python 3.6 DL: 7.2.14-7.2.16 Iceberg 0.14.1	N/A	N/A	Airflow 2.3.4 Python 3.8	1.24
1.18.1	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.14-7.2.16 Iceberg 0.14.1	Spark 3.2.0 Scala 2.12.10 Java 11.0.20 Python 3.6 Data lake 7.2.14-7.2.16 Iceberg 0.14.1	N/A	N/A	Airflow 2.3.4 Python 3.8	1.23
1.18	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Data lake 7.2.14-7.2.15 Iceberg 0.14.1	Spark 3.2.0 Data lake 7.2.14-7.2.15 Iceberg 0.14.1	N/A	N/A	Airflow 2.2.5 Python 3.8	1.23
1.17-h1	Spark 2.4.8 Scala 2.11.12 Java 1.8.0_352 Python 2.7 Iceberg 0.14.1	Spark 3.2.1 Iceberg 0.14.1	N/A	N/A	Airflow 2.2.5 Python 3.8	1.23

AWS Graviton component version compatibility

To use AWS Graviton in Cloudera Data Engineering, you need the following runtime component versions:

Spark 3.3 and 3.5
Data Lake 7.2.18 and 7.3.1