April 7, 2021
This release (1.6) of the Cloudera Data Engineering (CDE) service on CDP Public Cloud introduces the new features and improvements that are described in this topic.
- CDE users can now use API keys, managed using the CDP user management service (UMS), to interact with the CDE Jobs API using the command line.
Raw Scala code
- Users can now submit jobs with raw Scala code, without compiling.
These jobs run
spark-shellto process the application file.
- Admins can now access summary diagnostic logs directly on their local machine.
- A new Diagnostic page has been added to the CDE Service details to generate and download the bundle.
Force TLS certificate renewal
- CDE services older than 90 days will have expired TLS certificates. A new action has been added to the CDE service hamburger menu to renew the certificates and avoid access issues for DE users.
[Technical Preview] GANG scheduling
- GANG is a new resource scheduling policy that overcomes scale-up challenges in situations where high rates of job submission lead to queuing. The new scheduling policy moves jobs off the queue in batches. This clears up the queue, forces scale-up of nodes to process the burst of incoming jobs, and reduces wait and startup time of jobs.
- By default, GANG scheduling is disabled. It can be turned on for specific jobs by adding a new job-level configuration option.