Cloudera Docs

CDE hardware requirements

Review the requirements needed to get started with the Cloudera Data Engineering (CDE) service on Red Hat OpenShift.

Requirements

  • CDE assumes it has cluster-admin privileges on the OpenShift cluster.
  • Openshift cluster should be configured with route admission policy set to namespaceOwnership: InterNamespaceAllowed. This allows Openshift cluster to run applications in multiple namespaces with the same domain name.
  • CDE Service requirements:
    Table 1. The following are the CDE Service requirements:
    Component vCPU Memory Block PV or NFS PV Number of pods
    Embedded DB -- -- 100 GB 1
    Config Manager -- -- -- 2
    Dex Downloads -- -- -- 1
    Knox 250m 1 GB -- 1
    Management API 1 2 GB -- 1
    NGINX Ingress Controller 100m 90 MiB -- 1
    FluentD Forwarder 1 1 GB -- 1
    Grafana - - 10 GB 1
  • CDE Virtual Cluster requirements: Overall storage 500 GB per Virtual Cluster and a storage class named nfs exists backed by an NFS server.
    Table 2. The following are the CDE Virtual Cluster requirements:
    Component vCPU Memory Block PV or NFS PV Number of pods
    Airflow API 1,250 m 2 GB, 512 MiB 100 GB 2
    Airflow Scheduler 1 1 GB 100 GB 1
    Airflow Web 1 2 GB 100 GB 1
    Runtime API 1 2 GB 100 GB 1
    Livy 3 12 GB 100 GB 1
    Safari 4 6 GB -- 1
  • Workloads: Depending upon the workload, you must configure resources. Minimal spark/airflow jobs need 1 executor with 1 CPU and 1GB Memory.