Configuring VPC on AWS with private IPs and CCM

When you configure a VPC with private IPs and CCM, you have two alternatives: Have CDP set up your private network and security groups or set up the VPC with your private IPs and security groups.

Have CDP set up your private network and security groups

If you choose to have CDP create your private network and security groups when you are testing or sandboxing CCM, when you enable CCM, CDP performs the following:

  • Creates 3 public and 3 private subnets
  • Creates the Data Lake and Data Hub clusters in the private subnet.

The public subnets have an internet gateway attached. The private subnets have a NAT gateway attached. When CDP creates the security group, it opens two ports to the CIDR range that you specify, port 22 and port 443. Use these ports only to access these clusters.

No ports are opened for the control plane. Because the SSH tunnel service does not have a predictive IP address range for the NLB, all outbound traffic is open.

Set up VPC with your own private IPs and security groups

If you choose to configure your own VPCs with private IPs, you will need the following:

  • At least three private subnets for hosts that will use CCM.
  • At least three availability zones (AZs).
  • Outbound traffic via the SSH (secure shell) tunnel initiated by CCM allowed to the Cloudera hosted NLBs on workload nodes.

In the AWS console, configure the following:

  1. Create one public subnet and place the NAT gateway there to allow outbound connectivity in the private subnet:

    1. Assign an internet gateway to the public subnet.

    2. All inbound traffic must be on private subnets.

      In the case of a datahub, that is the Knox proxy. In the case of free IPA and a data lake, which doesn’t have Knox, use the Nginx proxy.
  2. Create three private subnets:

    1. Route the private subnet to the NAT gateway.

    2. You must configure outbound traffic for CDP resources.

    3. The workload clusters containing CCM (Knox, master, or CM for Classic Cluster) must be able to reach the Network Load Balancers (NLBs).

    4. Currently you can use ports 6000-6049 to connect to the NLBs.

      The private subnets must be in different availability zones (AZs).
  3. Create your security groups.

    Security groups do not need any external facing connections. You can choose what they open. The only requirement is to have outbound connectivity. At this point there is no defined network CIDR range needed. The SSH tunnel service will assign a random IP to the NLB.

In the CDP user interface, configure the following:

  1. After you have created your subnets and security groups, follow the steps for creating an Environment.

    See Register an AWS environment.
  2. Click the button to Enable Cluster Connectivity Manager on the Region, Networking, Security and Storage page.

    This will install CCM on the gateway nodes for the private subnets, establishing the reverse SSH tunnel.