Register an AWS environment
Once you’ve met the cloud provider requirements, register your AWS environment.
Before you begin
This assumes that you have already fulfilled the environment prerequisites described in AWS requirements.
Steps - CDP web interface
- Navigate to the Management Console > Environments > Register environment:
- On the Register Environment page, provide the following information:
Parameter Description General Information Environment Name Enter a name for your environment. The name:
- Must be between 5 and 28 characters long.
- Can only include lowercase letters, numbers, and hyphens.
- Must start with a lowercase letter.
Description (Optional) Enter a description for your environment. Select Cloud Provider Select Amazon. Credential Select Credential Select an existing credential or select Create new credential.
For instructions on how to create a credential, refer to Creating a role-based credential.
- Click Next.
- On the Data Access and Data Lake Scaling page, provide the following information:
Parameter Description Data Lake Settings Data Lake Cluster Name Enter a name for the Data Lake cluster that will be created for this environment. The name:
- Must be between 5 and 100 characters long
- Must contain lowercase letters
- Cannot contain uppercase letters
- Must start with a letter
- Can only include the following accepted characters are: a-z, 0-9, -.
Data Lake Version Select Cloudera Runtime version that should be deployed for your Data Lake. The latest stable version is used by default. All Data Hub clusters provisioned within this Data Lake will be using the same Runtime version. Data Access and Audit Select an Instance Profile Select the instance profile created in Minimal setup for cloud storage. Storage Location Base Provide the S3 location created for data storage in Minimal setup for cloud storage. Data Access Role Select the IAM role created in Minimal setup for cloud storage. Ranger Audit Role Select the IAM role created in Minimal setup for cloud storage. IDBroker Mappings We recommend that you leave this out and set it up after registering your environment as part of Onboarding CDP users and groups for cloud storage. Scale Scale Select Data Lake scale. By default, “Light Duty” is used.
For more information on Data Lake scale, refer to Data Lake scale.
- Click Next.
- On the Region, Networking and Security page, provide the following information:
Parameter Description Region Select Region Select the region that you would like to use for CDP.
If you would like to use a specific existing virtual network, the virtual network must be located in the selected region.
Network Select VPC You have two options:
Select the existing virtual network where you would like to provision all CDP resources. Refer to VPC and subnet.
Select Create new network to have a new network with three subnets created. One subnet is created for each availability zone assuming three AZs per region; If a region has two AZs instead of three, then still three subnets are created, two in the same AZ.
Select Subnets This option is only available if you choose to use an existing network. Multiple subnets must be selected and CDP distributes resources evenly within the subnets. Network CIDR This option is only available if you select to create a new network.
If you selected to create a new network, provide Network CIDR that determines the range of private IPs that EC2 instances will use. This must be a valid private IP CIDR IP in IPv4 range.
For example 10.10.0.0/16 are valid IPs. /16 is required to allow for enough IP addresses.
Create Private Subnets This option is only available if you select to have a new network and subnets created. Is is turned on by default so that private subnets are created in addition to public subnets. If you disable it, only public subnets will be created. Create Private Endpoints This option is only available if you select to have a new network and subnets created. It is disabled by default. Enable this option to use private endpoints instead of public endpoints. Enable Cluster Connectivity Manager This option is enabled by default. You can disable it if you do not want to use CCM. You can use Cluster Connectivity Manager (CCM) for communication with Data Lake and Data Hub workload clusters that are on private subnets. For more information about the required setup, refer to Cluster Connectivity Manager documentation. Enable Public Endpoint Access Gateway When CCM is enabled, you can optionally enable Public Endpoint Access Gateway to provide secure connectivity to UIs and APIs in Data Lake and Data Hub clusters deployed using private networking.
If you are using your existing VPC, under Select Endpoint Access Gateway Subnets, select the public subnets for which you would like to use the gateway. The number of subnets must be the same as under Select Subnets and the availability zones must match. For more information, refer to Public Endpoint Access Gateway documentation.
Proxies Select Proxy Configuration Select a proxy configuration if previously registered. For more information refer to Setting up a proxy server. Security Access Settings Select Security Access Type This determines inbound security group settings that allow connections to the Data Lake and Data Hub clusters from your organization’s computers. You have two options:
Create new security groups - Allows you to provide custom CIDR IP range for all new security groups that will be created for the Data Lake and Data Hub clusters so that users from your organization can access cluster UIs and SSH to the nodes.
This must be a valid CIDR IP in IPv4 range. For example: 192.168.27.0/24 allows access from 192.168.27.0 through 192.168.27.255. You can specify multiple CIDR IP ranges separated with a comma. For example: 192.168.27.0/24,192.168.28.0/24If you use this setting, several security groups will get created: one for each Data Lake host group the Data Lake and one for each host group), one for each FreeIPA host group, and one for RDS; Furthermore, the security group settings specified will be automatically used for Data Hub, Data Warehouse, and Machine Learning clusters created as part of the environment.
Provide existing security groups (Only available for an existing VPC) - Allows you to select two existing security groups, one for Knox-installed nodes and another for all other nodes. If you select this option, refer to Security groups to ensure that you open all ports required for your users to access environment resources.
SSH Settings New or existing SSH public key You have two options for providing a public SSH key:
- Select a key that already exists on your AWS account within the specific region that you would like to use.
- Upload a public key directly from your computer.
Add tags You can optionally add tags to be created for your resources on AWS. Refer to Defining custom tags. Enable S3Guard DynamoDB Table Name
(Optional) This option only appears when Runtime version older than 7.2.2 is selected. S3Guard is not used by newer Runtime versions.Provide a name for a DynamoDB table that will be created automatically, or select an existing DynamoDB table that meets the requirements described in DynamoDB table. The table is used by S3Guard to provide consistent view of the S3 object store.
If you choose to specify a table name, ensure that a table with the specified name does not already exist within the selected region on your AWS account.
If you choose to create your own table, ensure that no other environment is currently using this table.
- Click Next.
- On the Storage page, provide the following information:
Parameter Description Logs Select an Instance Profile Select the instance profile created in Minimal setup for cloud storage. Logs Location Base Provide the S3 location created for log storage in Minimal setup for cloud storage. (Optional) Provide the S3 location created for FreeIPA backups in Minimal setup for cloud storage. If not provided, the default Backup Location Base uses the Logs Location Base. Telemetry Enable Workload Analytics Enables Workload Manager support for workload clusters created within this environment. When this setting is enabled, diagnostic information about job and query execution is sent to Workload Manager. For more information, refer to Enabling workload analytics and logs collection. Enable Cluster Logs Collection When this option is enabled. the logs generated during deployments will be automatically sent to Cloudera. For more information, refer to Enabling workload analytics and logs collection.
- Click on Register Environment to trigger environment registration.
- The environment creation takes about 60 minutes. The creation of the FreeIPA server and Data Lake cluster is triggered. You can monitor the progress from the web UI. Once the environment creation has been completed, its status will change to “Running”.
Steps - CDP CLI
Unlike in the CDP web interface, in CDP CLI environment creation is a three-step process with environment creation, setting IDBroker mappings and Data Lake creation being three separate steps. The easiest way to obtain the correct commands is to provide all parameters in CDP web interface and then generate the CDP CLI commands on the last page of the wizard. For detailed steps, refer to Obtain CLI commands for registering an environment.
After you finish
- You must assign roles to specific users and groups for the environment so that selected users or user groups can access the environment. Next, you need to perform user sync. For steps, refer to Enabling admin and user access to environments.
- You must onboard your users and/or groups for cloud storage. For steps, refer to Onboarding CDP users and groups for cloud storage.
- You must create Ranger policies for your users. For instructions on how to access your Data Lake, refer to Accessing Data Lake services. Once you've accessed Ranger, create Ranger policies to determine which users have access to which databases and tables.