Deploy CDP using Terraform
This guide demonstrates how to deploy CDP on AWS or Azure by using one of the CDP deployment templates.
The templates use Terraform, an open source Infrastructure as Code (IaC) software tool for defining and managing cloud or data center infrastructure. You interface the templates via a simple configuration file residing in a GitHub repository.
For an overview of best practices for deploying CDP, refer to Creating and managing CDP deployments.Prerequisites
Prior to deploying CDP, you should make sure that your cloud account meets the basic requirements and that you've installed a few prerequisites.
Next, you can follow the instructions below for deploying CDP.
Deploy CDP
Setting up a CDP deployment involves cloning a GitHub repository, editing the configuration, and running Terraform commands.
Step 1: Clone the repository
The cdp-tf-quickstarts repository contains Terraform resource files to quickly deploy Cloudera Data Platform (CDP) Public Cloud and associated pre-requisite cloud resources. It uses the CDP Terraform Modules provided by Cloudera to do this.
Clone this repository and navigate to the directory with the cloned repository:
git clone https://github.com/cloudera-labs/cdp-tf-quickstarts.git
cd cdp-tf-quickstarts
Step 2: Edit the configuration file for the required cloud provider
In the cloned repository, change to the required cloud provider directory. Currently AWS and Azure are available.
Next, edit the input variables in the configuration file as required:
cd aws
mv terraform.tfvars.template terraform.tfvars
vi terraform.tfvars
cd azure
mv terraform.tfvars.template terraform.tfvars
vi terraform.tfvars
Sample content of this file, with indicators of values to change are shown below. The variables are explained below the sample. You should review and update all the variables.
# ------- Global settings -------
env_prefix = "<ENTER_VALUE>" # Required name prefix for cloud and CDP resources, e.g. cldr1
# ------- Cloud Settings -------
aws_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eu-west-1
# ------- CDP Environment Deployment -------
deployment_template = "<ENTER_VALUE>" # Specify the deployment pattern below. Options are public, semi-private or private
# ------- Global settings -------
env_prefix = "<ENTER_VALUE>" # Required name prefix for cloud and CDP resources, e.g. cldr1
# ------- Cloud Settings -------
azure_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eastus
# ------- CDP Environment Deployment -------
deployment_template = "<ENTER_VALUE>" # Specify the deployment pattern below. Options are public, semi-private or private
As an outcome of this step, your configuration file should look similar to the following:
# ------- Global settings -------
env_prefix = "test-env" # Required name prefix for cloud and CDP resources, e.g. cldr1
# ------- Cloud Settings -------
aws_region = "eu-west-1" # Change this to specify Cloud Provider region, e.g. eu-west-1
# ------- CDP Environment Deployment -------
deployment_template = "public" # Specify the deployment pattern below. Options are public, semi-private or private
# ------- Global settings -------
env_prefix = "test-env" # Required name prefix for cloud and CDP resources, e.g. cldr1
# ------- Cloud Settings -------
azure_region = "westeurope" # Change this to specify Cloud Provider region, e.g. eastus
# ------- CDP Environment Deployment -------
deployment_template = "public" # Specify the deployment pattern below. Options are public, semi-private or private
The following tables explain the mandatory inputs that need to be provided in the configuration file.
Table 1: Mandatory inputs
Input | Description | Default value |
env_prefix |
A string prefix that will be used to name the cloud provider and CDP resources created. | Not set |
aws_region |
The AWS region in which the cloud prerequisites and CDP will be deployed. For example, eu-west-1. For a list of supported AWS regions, see Supported AWS regions. | Not set |
deployment_template |
The selected deployment pattern. Values allowed:
|
public |
Input | Description | Default value |
azure_region |
The Azure region in which the cloud prerequisites and CDP will be deployed. For example, eastus. For a list of supported Azure regions, see Supported Azure regions. | Not set |
env_prefix |
A string prefix that will be used to name the cloud provider and CDP resources created. | Not set |
deployment_template |
The selected deployment pattern. Values allowed:
|
public |
The following tables explain the optional inputs. The optional inputs can optionally be added to the configuration file. While the mandatory inputs are present in the configuration file and only their values need to be provided, the optional inputs should be added manually.
Table 2: Optional inputs
Input | Description | Default value |
aws_key_pair |
The name of an AWS keypair that exists in your account in the selected region. | Not set |
ingress_extra_cidrs_and_ports |
Inbound access to the UI and API endpoints of your deployment will be
allowed from the Enter your machine’s public IP here, with ports 443 and 22. If unsure, you can check your public IP address here. |
CIDRs are not set. Ports are set to 443, 22 by default. |
create_vpc |
Flag to specify if the VPC should be created | true |
cdp_vpc_id |
VPC ID for CDP environment. Required if create_vpc is
false |
Empty string |
cdp_public_subnet_ids |
List of public subnet ids. Required if create_vpc is false |
Empty list |
cdp_private_subnet_ids |
List of private subnet ids. Required if create_vpc is false |
Empty list |
private_network_extensions |
Enable creation of resources for connectivity to CDP Control Plane (public subnet and NAT Gateway) for Private Deployment. Only relevant for private deployment template | true |
Input | Description | Default value |
public_key_text |
An SSH public key string to be used for the nodes of the CDP environment. | Not set |
ingress_extra_cidrs_and_ports |
Inbound access to the UI and API endpoints of your deployment will be
allowed from the Enter your machine’s public IP here, with ports 443 and 22. If unsure, you can check your public IP address here. |
CIDRs are not set. Ports are set to 443, 22 by default. |
create_vnet |
Flag to specify if the VNet should be created | true |
cdp_resourcegroup_name |
Preexisting Azure resource group for CDP environment. Required if
create_vnet is false |
Empty string |
cdp_vnet_name |
VNet name for CDP environment. Required if create_vnet is
false |
Empty string |
cdp_subnet_names |
List of subnet names for CDP resources. Required if create_vnet is
false |
Empty list |
cdp_gw_subnet_ids |
List of subnet names for CDP Gateway. Required if create_vnet is
false |
Empty list |
Step 3: Launch the deployment
terraform init
terraform apply
Terraform will show a plan with the list of cloud provider and CDP resources that will be created.
When you are prompted, type yes
to tell Terraform to perform the
deployment. Typically, this will take about 60 minutes. Once the deployment is complete, CDP
will print output similar to the following:
Apply complete! Resources: 46 added, 0 changed, 0 destroyed.
You can navigate to the CDP web interface at https://cdp.cloudera.com/ and see your deployment progressing. Once the deployment completes, you can create Data Hubs and data services.
Clean up the CDP environment and infrastructure
If you no longer need the infrastructure that’s provisioned by Terraform, run the following command to remove the deployment infrastructure and terminate all resources:
terraform destroy