Creating and managing CDP deployments

A CDP deployment is a set of CDP management services and data services including related cloud provider resources that exist in your AWS, Azure, or GCP account. It is a combination of the cloud infrastructure that may span multiple cloud providers and regions, and the CDP platform that abstracts this underlying cloud provider infrastructure into an integrated, unified, logical data platform layer.

Each CDP deployment consists of CDP services and the underlying cloud provider resources. The cloud provider resources include the resources that you precreate up-front and the resources that CDP creates when provisioning CDP services. The CDP services include a CDP environment and workload services such as Data Hubs and data services (Cloudera Data Warehouse, Cloudera Machine Learning, and so on).

In order for CDP to be deployed, a set of cloud provider prerequisites needs to be provided first, including a virtual network and subnets, storage accounts, and IAM roles and policies. These cloud provider prerequisites are typically customer-managed and exist in the cloud provider account independently of CDP services. As such, they may be shared with other, non-Cloudera cloud services.

Once the cloud provider prerequisites are present, a CDP environment can be deployed in the virtual network. Once your CDP environment is up and running, your core CDP and cloud provider infrastructure is in place and you can start creating Data Hubs and data services in order to run workloads. When these services are created, additional cloud provider resources such as VM instances, security groups, and load balancers are deployed in your cloud account. For each service, you can select which subnets of the underlying virtual network and what storage locations within your specified storage accounts they should use.

These three high-level deployment steps are described in the following diagram:

CDP deployment can be performed by using either CDP web interface or CDP CLI, or Terraform-based CDP deployment patterns. Continue reading to learn about deploying CDP using Terraform.

The Terraform Module for CDP Prerequisites on AWS contains Terraform resource files and example variable definition files for creating the prerequisite cloud provider resources. The module also includes an Ansible playbook for deploying a CDP environment, Data Lake, and FreeIPA, and a set of supporting scripts and template files to interact with the CDP tenant and APIs. The module supports the network deployment patterns described in CDP deployment pattern definitions.

To simplify the task of defining and creating CDP deployments, we provide and describe a set of predefined target architectures recommended by Cloudera. These target architectures are called deployment patterns.

In Cloudera’s Terraform framework, each pattern is represented by a deployment template that allows you to quickly instantiate one of the reference deployments. The templates can be used as a starting point and modified according to your needs. You can learn more about the recommended configurations of CDP Public Cloud from the documentation of our end-to-end deployment patterns as well as our network reference architectures.

Deployment patterns are predefined architectures recommended by Cloudera that simplify the task of defining and creating CDP deployments. There are many options available for deploying CDP, but as a best practice, Cloudera recommends that you use one of the following three deployment patterns: private, semi-private, or public.

These patterns are based on the identically named network reference architectures and extend them, by incorporating Cloudera’s recommended configuration for deploying CDP in multiple availability zones, selecting the Data Lake scale, configuring storage access policies and setting up fine-grained access control.

As can be expected, each of these deployment patterns brings a unique trade-off among various aspects, such as ease of setup, security provided, workloads supported, and so on. Read the following content to understand what specific networking, IAM, and storage cloud provider configurations, and CDP configurations are applied as part of the supported deployment patterns.