Cloudera Docs

CDE hardware requirements

To launch the Cloudera Data Engineering service in the CDP Private Cloud Data Services host must meet several requirements. Review the requirements to get started with the Cloudera Data Engineering (CDE) service on Red Hat OpenShift 4.7 or 4.8 clusters.

Requirements

  • CDE assumes it has cluster-admin privileges on the OpenShift cluster.
  • Openshift cluster should be configured with route admission policy set to namespaceOwnership: InterNamespaceAllowed. This allows Openshift cluster to run applications in multiple namespaces with the same domain name.
  • Storage Class: CDE assumes a storage class named nfs exists backed by an NFS server.
  • CDE Service requirements:
    Table 1. The following are the CDE Service requirements:
    Component vCPU Memory Block PV or NFS PV Number of pods
    Embedded DB -- -- 100 GB 1
    Config Manager -- -- -- 2
    Dex Downloads -- -- -- 1
    Knox 250m 1 GB -- 1
    Management API 1 2 GB -- 1
    NGINX Ingress Controller 100m 90 MiB -- 1
    FluentD Forwarder 1 1 GB -- 1
    Grafana - - 10 GB 1
  • CDE Virtual Cluster requirements: Overall storage 500 GB per Virtual Cluster and a storage class named nfs exists backed by an NFS server.
    Table 2. The following are the CDE Virtual Cluster requirements:
    Component vCPU Memory Block PV or NFS PV Number of pods
    Airflow API 1,250 m 2 GB, 512 MiB 100 GB 2
    Airflow Scheduler 1 1 GB 100 GB 1
    Airflow Web 1 2 GB 100 GB 1
    Runtime API 1 2 GB 100 GB 1
    Livy 3 12 GB 100 GB 1
    Safari 4 6 GB -- 1
  • Workloads: Depending upon the workload, you must configure resources. Minimal spark/airflow jobs need 1 executor with 1 CPU and 1GB Memory.