Creating a Session using the CDE CLI [Technical Preview]
The cde session create
command allows you to create a new
Session.
Run the following command in the CDE CLI:
cde session create --name <session-name> --type <pyspark/spark-scala>
- You can enable the GPU acceleration using
the
[--enable-gpu-acceleration]
flag during the spark session creation.cde session create --name test-session --type spark-scala \ --enable-gpu-acceleration
- To accelerate session queries on specific hardware, you can use
[--executor-node-selector "nvidia.com/gpu=A100"]
and[--executor-node-toleration "nvidia.com/gpu=true"]
options to configure selectors and tolerations if you want to run the job on specific GPU nodes. You can only provide executor node selectors and tolerations, as GPUs are used by executors only. The selector and tolerations CLI options are optional. You can use either one or both in congestion with--enable-gpu-acceleration
flag.For example:cde session create --name test-session --type spark-scala \ --enable-gpu-acceleration \ --executor-node-selector "nvidia.com/gpu=A100" \ --executor-node-toleration "nvidia.com/gpu=true"
- To accelerate session queries on specific hardware, you can use