Storage requirements

Storage requirements for Data Services.

Storage Requirements

Data Services Storage type Storage required Purpose
CDE Block 500GB per Virtual Cluster in Embedded NFS Stores all information related to virtual clusters
CDW Local 100 GB per executor in LITE mode and 600 GB per executor in FULL mode Used for caching
Control Plane Block 118 GB total if using an External Database, 318 GB total if using the Embedded Database Storage for CDP infrastructure including Fluentd logging, Prometheus monitoring, and Vault. Backing storage for an embedded DB for control plane configuration purpose, if applicable
CML Block 600 GB per node (minimum), 4.5 TB (recommended) Stores all CML workspace information
External NFS or Block 1 TB per Node Stores all user project files. VFS storage can either use Longhorn NFS-provisioner on Longhorn OR directly connect to your NFS.
MonitoringApp Block 30 GB + (Env cnt x 100 GB) Stores metrics collected by Prometheus.