Configuring Cloudera VPC to deploy Lambda Functions

Before you begin

Cloudera DataFlow Functions running on top of AWS Lambda require access to the Cloudera DataFlow Catalog deployed on the Cloudera Control Plane. This means that Lambda requires outbound internet access to download the flow that is going to be run, and it also requires access to some AWS services like Security Token Service (STS) and AWS Secrets Manager.

This article explains how you can allow a managed, serverless service like AWS Lambda to have internet access. AWS Lambda cannot be deployed on a public subnet in a VPC as it is not possible to grant a Lambda function a public IP address. This means that to have internet access, you need to have at least one private subnet for each availability zone where the Lambda function is going to run and you need to configure a Network Address Translation (NAT) gateway on the public subnets to route its outbound traffic.

Step 1 - Creating VPC private subnets and route tables

A VPC can have multiple subnets, which can be public or private. Depending on the connectivity that you need, you might also have to add gateways and route tables. The Internet Gateway (IG) component enables your instances to connect to the internet through the Amazon EC2 network edge. For more information on basic AWS networking concepts, see How Amazon VPC works.

A public subnet has a direct route to an internet gateway. Resources in a public subnet can access the public internet. By default, the AWS subnets are public, so the instances have public and private IPs and they have a default route table with a route to an internet gateway.

A private subnet does not have a direct route to an internet gateway. Resources in a private subnet require a NAT device to access the public internet, so it cannot have a route to an IG. An EC2 instance in a non-default subnet only has private IPs and does not have internet access.

To allow a private subnet to have internet access you need to use a NAT device, which maps multiple private IPs into a single public IP. So, if you configure the NAT device in a public subnet with a public IP, it can use the internet gateway to route the traffic and get access to the internet.

Use the following procedure to create subnets for your VPC and to create route tables that will be associated with the subnets.

1. Create one private subnet per availability domain of your VPC where you will deploy the Lambda function.
   1. Open the Amazon VPC console.
   2. Click Subnets in the left navigation pane.
   3. Click Create subnet.
   4. For the VPC ID, select the VPC where you want to add the new subnet.
   5. (Optional) For Name, add a name for your subnet.
   6. Define the Availability Zone and the CIDR.
   7. Click Create subnet.
2. Create a route table that will be associated with the subnet.
   1. Click Route tables in the left navigation pane.
   2. Click Create route table.
   3. (Optional) For Name, add a name for your route table.
   4. For VPC, choose your VPC.
   5. (Optional) To add a tag, click Add new tag and add the tag key and tag value.
   6. Click Create route table.

Step 2 - Creating NAT gateways and configuring route tables

Now that you have your subnets, you need to create and configure the Network Address Translation (NAT) gateways that map the private IPs to a public IP and allow the instances in your private subnets to connect to services outside your VPC.

1. Create NAT gateways.
   1. Open the Amazon VPC console.
   2. Click NAT gateways in the left navigation pane.
   3. Click Create NAT gateway.
   4. (Optional) Add a name for the NAT gateway.
   5. Define the subnet where you want to create the NAT gateway.
   6. Define the connectivity type.
   7. Allocate an Elastic IP. For more information, see Elastic IP addresses.
2. Configure the route tables of the private subnets created in the previous procedure.
   1. Click Route tables.
   2. Search for the route table.
   3. Click Routes > Edit routes.

      There are two routes that need to be defined.

      Route 1

      • Destination: <VPC CIDR>
      • Target: local
      • Status: Active
      Route 2

      • Destination: 0.0.0.0/0
      • Target: <Search for one of the NAT Gateways created>
      • Status: Active

   4. Click Save Changes.

3. Associate each route table with each private subnet.
   1. Click Subnet Associations.
   2. Select the specific subnet.
   3. Save the associations.
   4. Validate that subnets have the route table with it.

Result

With this, the private subnet components are able to communicate with other components within the same VPC and have internet access.