CDE hardware requirements

To Launch the Cloudera Data Engineering service in the CDP Private Cloud Experiences host must meet several requirements. Review the requirements to get started with the Cloudera Data Engineering (CDE) service on Red Hat OpenShift 4.6 clusters.

Requirements

  • CDE assumes it has cluster-admin privileges on the OpenShift cluster.
  • Openshift cluster should be configured with route admission policy set to namespaceOwnership: InterNamespaceAllowed. This allows Openshift cluster to run applications in multiple namespaces with the same domain name.
  • Storage Class: CDE assumes a storage class named 'nfs' exists backed by an NFS server.
  • CDE Service requirements:
    Table 1. The following are the CDE Service requirements:
    Component vCPU Memory Local Storage Number of pods
    Embedded DB -- -- 100 GB 1
    Config Manager -- -- -- 2
    Dex Downloads -- -- -- 1
    Knox 250m 1 GB -- 1
    Management API 1 2 GB -- 1
    NGINX Ingress Controller 100m 90 MiB -- 1
    FluentD Forwarder 1 1 GB -- 1
  • CDE Virtual Cluster requirements. Overall storage 100 GB per Virtual Cluster. a storage class named 'nfs' exists backed by an NFS server.
    Table 2. The following are the CDE Virtual Cluster requirements:
    Component vCPU Memory Local Storage Number of pods
    Airflow API 1,250 m 2 GB, 512 MiB 20 GB 2
    Airflow Scheduler 1 1 GB 20 GB 1
    Airflow Web 1 2 GB 20 GB 1
    Runtime API 1 2 GB 20 GB 1
    Livy 3 12 GB 20 GB 1
    Safari 4 6 GB -- 1
  • Workloads: Depending upon the workload, you must configure resources. Minimal spark/airflow jobs need 1 executor with 1 CPU and 1GB Memory.