Functional differences between CDSW and Cloudera AI

Consider the following key differences between CDSW and Cloudera AI in some of the functions.

CDSW has a Host Mount feature that allows mounting a local directory on CDSW host into the user sessions. A popular usage of the feature is to push files to users in sessions.

This feature is not available in Cloudera AI but consider using Custom Runtime Addons as a replacement. When you create a new Runtime Addon, Cloudera AI creates a directory on the NFS share directory, for example, (nfs://<nfs-share>/addons/custom-addon-<myaddon>/.../) which gets mounted in all the user sessions. You can then share the files by copying them to the custom addon directory on the NFS.

note

This share is mounted as read-only in the user sessions, that is, users can only read but cannot write to this directory. This is a key difference to host mounts, which are writable in CDSW.
Only the Spark Pushdown feature works with external shuffle service, Spark on Kubernetes supports external shuffle service.
In Cloudera AI the following Spark properties are hardcoded to true value:
- spark.authenticate=true
- spark.io.encryption.enabled=true
- spark.network.crypto.enabled=true
Use the spark-defaults.conf file of the project, if you need to override any of the settings.