Fixed issues in 1.5.5 SP2

You can review the list of reported issues and their fixes in Cloudera Data Engineering on premises 1.5.5 SP2. Fixed issues represent selected issues that were previously logged through Cloudera Support, but are now addressed in the current Cloudera Data Engineering on premises release. These issues might have been reported in previous versions of Cloudera Data Engineering on premises as a known issue; meaning they were reported by customers or identified by Cloudera Quality Engineering teams.

DEX-18727: Spark SQL functions do not work in Spark Connect sessions

Previously, Spark Connect functions required the SPARK_CONNECT_MODE_ENABLED environment variable to be set to true. Otherwise, Spark tried to use the normal Spark context functions, which did not work in Spark Connect. This issue is now fixed. The SPARK_CONNECT_MODE_ENABLED environment variable is automatically set when connecting to a Spark Connect session and the Spark SQL functions work as expected.

DEX-17957: Airflow child jobs used creator privileges instead of submitter privileges in OpenShift Container Platform

Previously, when Airflow jobs were run in OpenShift Container Platform, the privileges of the user who created the job were applied and not the privileges of the user who submitted the job. This caused issues when the job submitter had fewer privileges than the job owner. This issue is now fixed and the Airflow job now launches the child Spark jobs with the privileges of the user who triggers, submits, or schedules the Airflow job.

DEX-18429: Cloudera Data Engineering service fails to start displaying the cluster with same name already exists error

Previously, if a user without the DEAdmin role tried to create a Cloudera Data Engineering service using the API, the request was accepted, but only a database entry was created without generating the corresponding resources. When the user tried to create that service again, it failed displaying the cluster with same name already exists error. This issue is now fixed and only a DEAdmin user can make a request to create a Cloudera Data Engineering service. Requests from users with any other role are now rejected without creating a database entry.