Storage requirements

Storage requirements for Data Services.

Storage Requirements

Data Services Storage type Storage required Purpose
CDE Block 500GB per Virtual Cluster in Embedded NFS Stores all information related to virtual clusters
CDW Local 100 GB per executor in LITE mode and 600 GB per executor in FULL mode Used for caching
Control Plane Block 118 GB total if using an External Database, 318 GB total if using the Embedded Database (SSD support only) Storage for CDP infrastructure including Fluentd logging, Prometheus monitoring, and Vault. Backing storage for an embedded DB for control plane configuration purpose, if applicable
CML Block 600 GB per node (minimum), 4.5 TB (recommended) Stores all CML workspace information
External NFS or Block 1 TB per Node Stores all user project files. VFS storage can either use Longhorn NFS-provisioner on Longhorn OR directly connect to your NFS.
MonitoringApp Block 30 GB + (Env cnt x 100 GB) Stores metrics collected by Prometheus.
Data Catalog Requires Control Plane database and not a dedicated storage space 100 GB extra in Control plane database Stores profiling metadata.