What's new in 1.5.5 SP4
Cloudera Data Engineering on premises 1.5.5 SP4 introduces a set of new features for Cloudera Data Engineering.
- In-place upgrade support
- Cloudera Data Engineering on premises now supports In-Place Upgrades and Backup and Restore capabilities. This enhancement allows administrators to upgrade their on premises services seamlessly within the existing environment and provides a robust mechanism for data protection and recovery, ensuring high availability and simplified lifecycle management for Cloudera Data Engineering on premises.
- Expanded Data Connector Support for Kafka, HBase, and Phoenix
-
Cloudera Data Engineering now supports Kafka, HBase, and Phoenix data connectors alongside Ozone. Spark workloads can now connect directly to these sources without manual JAR management or complex custom configurations.
With expanded Data Connector support, Cloudera Data Engineering allows you to:
- Simple UI selection: Choose Ozone, Kafka, HBase, or Phoenix directly in the Cloudera Data Engineering UI when creating or editing Virtual Clusters and Jobs.
- Automatic dependency loading: Cloudera Data Engineering automatically loads required driver files (JARs) into the Spark runtime based on your selected connector.
- Multi-source support: Attach multiple data connectors to a single job simultaneously (for example, combining Ozone and Kafka).
- Smart configuration: Cloudera Data Engineering automatically applies environment settings such as Kafka bootstrap servers or HBase site configurations directly to your workload.
For more information, see Configuring Data Connectors.
- Cloudera Data Engineering integration with Data Recovery Service (DRS)
-
Cloudera Data Engineering now integrates with Data Recovery Service (DRS) on Cloudera Data Services on premises, enabling full backup and restore capabilities for Cloudera Data Engineering services to protect on-premises workloads and ensure business continuity.
With Data Recovery Service (DRS) integration, Cloudera Data Engineering allows you to:
- Perform full backups and restores: Recover CDE services directly to a new on-premises Cloudera Base cluster.
- Maintain log continuity: Retain access to essential operational logs (stdout, stderr, and event logs) throughout backup and recovery workflows.
For more information, see Data Recovery Service.
- Notification Service integration for Cloudera Data Engineering on premises
-
Cloudera Data Engineering now integrates with the Notification Service on Cloudera Data Services on premises, providing automated alerts for your data engineering workflows.
With Notification Service integration, Cloudera Data Engineering allows you to:
- Subscribe to lifecycle events and configuration changes: Configure specific alerts to track system configuration modifications and lifecycle events at both the Service and Virtual Cluster levels.
- Receive real-time notifications: Get automated alerts delivered immediately as events occur to stay updated on your tasks.
For more information, see Notifications.
- Support for resuming and re-running Airflow DAG tasks
- Cloudera Data Engineering now enables Administrators to resume or re-run specific failed tasks within an Airflow DAG using the native Clear Task operation in the Airflow UI. This update eliminates the need for full DAG re-runs or manual workarounds, allowing for more efficient troubleshooting and workflow management while ensuring task-level actions are synchronized within the Cloudera Data Engineering environment. For more information, see Clearing and rerunning failed DAG tasks.
- Export Spark event logs to third party observability tools
- You can now automatically export finalized Spark event logs to your third-party observability tools through the OpenTelemetry (OTel) gateway, with full toggle control directly in the Cloudera Data Engineering service console. To enable Third-Party Observability Export feature, see Adding a Cloudera Data Engineering service.
- Email Alerts tab is renamed in Virtual Cluster configuration
- From Cloudera Data Engineering 1.5.5 SP4 release onwards, the Email Alerts tab in the Virtual Cluster Details page is renamed to Workload Alerts Setup. For enabling Workload Alerts, see Creating virtual clusters.
- Automated job run retention and cleanup
- Cloudera Data Engineering now provides automated log retention and cleanup for job runs. Expired logs stored in shared storage are automatically purged based on your retention settings, preventing storage overflow and performance degradation. For enabling Job Run Retention Policy, see Creating virtual clusters.
- Update TLS Certificates for Cloudera Data Engineering repositories
- Cloudera Data Engineering now supports updating TLS certificates for existing repositories. Previously, users could not modify certificates after a repository was created, which caused issues when certificates expired. For more information, see Updating the Control Plane certificates in Cloudera Data Engineering repositories.
- Enhanced security compliance with single-purpose EKU certificates
- To maintain high security standards and align with modern root certificate program requirements, Cloudera Data Engineering on premises now enforces strict Single-Purpose Extended Key Usage (EKU) for digital certificates. This transition mitigates security risks associated with multi-purpose certificates by ensuring each certificate is restricted to a specific function, such as Server Authentication (serverAuth) or Client Authentication (clientAuth).
- Kubernetes version upgrade to 1.34
- The Kubernetes version that Cloudera Data Engineering uses is upgraded to Kubernetes 1.34. For more information, see Compatibility for Cloudera Data Engineering and Cloudera Runtime components.
- Python version upgrade to 3.11.16
- The Python version that Cloudera Data Engineering uses is upgraded to Python 3.11.16. For more information, see Compatibility for Cloudera Data Engineering and Cloudera Runtime components.
- Java version upgrade to 17.0.20
- The Java version that Cloudera Data Engineering uses is upgraded to Java 17.0.20. For more information, see Compatibility for Cloudera Data Engineering and Cloudera Runtime components.
