Deploying Cloudera Manager and CDH on AWS

To deploy Cloudera Manager and CDH on an AWS EC2 instance, begin by creating an environment. The environment defines common settings, like region and key pair, that Cloudera Director uses with AWS. While creating an environment, you are also prompted to deploy its first cluster.

To create an environment:

Open a web browser and go to the private IP address of the instance you created in Launching an EC2 Instance for Cloudera Director. Include port 7189 in the address. For example:
```
http://192.0.2.0:7189
```
In the Cloudera Director login screen, enter admin in both the Username and the Password fields.
In the Cloudera Director Welcome screen, click Let's get started.
This opens a wizard for adding an environment, Cloudera Manager, and a CDH cluster.
In the Add Environment screen:
1. Enter a name in the Environment Name field.
2. Select Amazon Web Services (AWS) from the Cloud provider field.
3. Enter your AWS credentials in the Access key ID and Secret access key fields.
  Note: Leave the Access key ID and Secret access key fields blank if you are using AWS Identity and Access Management (IAM) for authentication and authorization.
4. In the EC2 region field, select the same region in which your Cloudera Director instance was created.
5. In the SSH Credentials section:
  1. For a RHEL AMI, enter ec2-user in the Username field.
    Note: Depending on the operating system, different AMIs may require different SSH logins. To check the SSH login name for your EC2 instance, go to the Instances page in the AWS Console and click the instance name to display details for your instance and click the Usage Instructions tab. In this example, centos is used with a Centos AMI, not ec2-user:
  2. Copy the SSH private key you created in Launching an EC2 Instance for Cloudera Director in the Private key field.
Click Continue to add Cloudera Manager.
In the Add Cloudera Manager screen:
1. Enter a name for this deployment of Cloudera Manager in the Cloudera Manager name field.
2. In the Instance Template field, click Select a Template if you already have one that you want to use, otherwise, click Create New Instance Template.
  The Create New Instance Template modal screen displays.
In the Create New Instance Template modal screen:
1. In the Instance Template name field, enter a name for the template.
2. In the Instance type field, select m4.large or m4.xlarge.
3. In the Image (AMI) ID field, enter the ID for the Amazon machine image (AMI) you chose in Launching an EC2 Instance for Cloudera Director, or find another AMI with a supported operating system.
  Note: To reduce cluster bootstrap times, you can preload the AMIs you use with Cloudera Manager and CDH. For more information, see Using Custom Repositories with Cloudera Manager and CDH.
4. In the Tags field, add one or more tags to associate with the instance.
  Note: Cloudera Director uses the Name field on AWS. By default, Cloudera Director puts the following values in the Name field: Cloudera Director id, Cloudera Director template name. If you want to create a custom key value for this field in place of "Name," see Configuring Cloudera Director to Use Custom Tag Names on AWS.
5. In the Security group IDs field, enter the security group ID you set up in Creating a New Security Group.
6. In the VPC subnet ID field, enter the ID of the VPC subnet that was created during VPC setup.
7. Click Advanced Options if you want to specify additional values for optional features, such as EBS, IAM, Spot instances, and AWS user data.
  Note: With Cloudera Director 2.4 and higher, you can enter opaque data or scripts in the User data field of the instance template, and this data will be passed to the instances as they are launched, using the EC2 user data mechanism. The data will be base64-encoded automatically. For more information, including examples, see Instance Metadata and User Data in the AWS documentation.
8. Click Save changes.
In the Desired License Type field, select one of the following license types:
- Cloudera Enterprise: includes the core CDH services (HDFS, Hive, Hue, MapReduce, Oozie, Sqoop, YARN, and ZooKeeper) and, depending on the license edition, one or more additional services (Accumulo, HBase, Impala, Navigator, Solr, Spark). For more information on Cloudera Enterprise licenses, see Managing Licenses in the Cloudera Manager documentation.
- Cloudera Enterprise Trial: a 60-day trial license that includes all CDH services.
- Cloudera Express: no license required.
To enable usage-based billing, you must have a Cloudera Enterprise license and a billing ID provided by Cloudera. Perform these steps in the Add Cloudera Manager screen:
1. In the Desired License Type field, select Cloudera Enterprise.
2. In the License Key field, either select a Cloudera Enterprise license file to upload or select Direct Input and input the license file text directly into the text area.
3. To enable usage-based billing, enter the billing ID provided to you by Cloudera in the Billing ID field.
In the Database Server section, the default selection is Embedded Database. This installs an embedded PostgreSQL database for Cloudera Manager. The embedded PostgreSQL database should be used only when creating a demonstration or proof-of-concept deployment. It is not recommended for production. If desired, select Create Database Server Instance or Register Existing Database Server from the dropdown list, instead of Embedded Database:

For information about using an external database for Cloudera Director server, see the following pages:
- Using MySQL for Cloudera Director Server
- Using MariaDB for Cloudera Director Server
In the Add Cloudera Manager screen, click Cloudera Manager Configurations.
In the Cloudera Manager Configurations modal screen, set the heap size:
1. In the Scope field, select Host Monitor and add firehose_heapsize and 1073741824 in the respective Name and Value fields.
2. Click +.
3. In the Scope field, select Service Monitor and add firehose_heapsize and 1073741824 in the respective Name and Value fields.
4. Click Save Changes.

By default, the version of Cloudera Manager installed depends on the version of Cloudera Director you are using:

Cloudera Director version	Cloudera Manager version installed
Cloudera Director 2.0	Latest released version of Cloudera Manager 5.5
Cloudera Director 2.1	Latest released version of Cloudera Manager 5.7
Cloudera Director 2.2	Latest released version of Cloudera Manager 5.8
Cloudera Director 2.3	Latest released version of Cloudera Manager 5.10
Cloudera Director 2.4	Latest released version of Cloudera Manager 5.11
Cloudera Director 2.5	Latest released version of Cloudera Manager 5.12

To install a version of Cloudera Manager higher or lower than the default version, perform the following steps:

In the Configurations section, check Override default Cloudera Manager repository.
In the Repository URL field, enter the repository URL for the version of Cloudera Manager to install. Repository URLs for versions of Cloudera Manager 5 have the form http://archive.cloudera.com/cm5/ followed by the operating system, operating system major version, processor architecture, cm (for Cloudera Manager), and the Cloudera Manager major, minor, and (if applicable) maintenance release number. For example, the repository URL for Cloudera Manager 5.5.4 on any supported version of RHEL 7 is http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.5.4/.
Note: The Cloudera Manager minor version must the same as or higher than the CDH minor version. For example, Cloudera Manager 5.5 cannot be used to launch or manage a CDH 5.7 cluster, but Cloudera Manager 5.7 can be used with a CDH 5.7 (or lower) cluster.
In the Repository Key URL field, enter the URL for the repository key. Repository key URLs have the same form as repository URLs except they end with the name of the key file instead of the Cloudera Manager version. For example, the repository key URL for any version of Cloudera Manager 5 on any supported version of RHEL 7 is http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera.

In the Add Cloudera Manager screen, click Continue.
At the Confirmation prompt, click OK to begin adding a cluster.

On the Add Cluster screen:

Enter a name for the cluster in the Cluster name field.

Enter the version of CDH to deploy in the Version field or leave the default value. By default, the version of CDH installed depends on the version of Cloudera Director you are using:

Cloudera Director version	CDH version installed
Cloudera Director 2.0	Latest released version of CDH 5.5
Cloudera Director 2.1	Latest released version of CDH 5.7
Cloudera Director 2.2	Latest released version of CDH 5.9
Cloudera Director 2.3	Latest released version of CDH 5.10
Cloudera Director 2.4	Latest released version of CDH 5.11
Cloudera Director 2.5	Latest released version of CDH 5.12

To install a version of CDH higher or lower than the default version, perform the following steps:

Enter the desired CDH version in the Version field of the Products section. For example, for CDH 5.4.8 enter 5.4.8.
Scroll down to Configurations (optional) and expand the section.
Click Override default parcel repositories.
Enter the repository parcel URL for the version of CDH you want to install. Parcel URLs for versions of CDH 5 have the form http://archive.cloudera.com/cdh5/parcels/, followed by the major, minor, and (if applicable) maintenance release number. For example, the URL for CDH 5.4.8 is http://archive.cloudera.com/cdh5/parcels/5.4.8.
Note: The CDH minor version must not be higher than the Cloudera Manager minor version. For example, CDH 5.7 will not work with Cloudera Manager 5.5, but CDH 5.7 (or lower) will work with Cloudera Manager 5.7.

In the Services section, select the services you want to install.
In the Instance groups area, choose an existing instance template or create a new one, either for the all instance groups in the cluster, or for each group. For each instance group, indicate the number of instances you want.
Note: To reduce cluster bootstrap times, you can preload the AMIs you use for your cluster instance groups. For more information, see Using Custom Repositories with Cloudera Manager and CDH.
If you want to use Spot instances for your workers group:
1. In the Create New Instance Template modal screen, click Advanced Options.
2. In the Spot bid (USD/hr) field, enter your Spot bid price.
3. Click the Use Spot instances checkbox.
4. Click Save Changes.
  For more information about using Spot instances with Cloudera Director, see Using Spot Instances.

Click Continue.
At the Confirmation prompt, click OK to deploy the cluster. Cloudera Director displays a status screen.
When the cluster is ready, click Continue.
You are finished with the deployment tasks.

Configuring a SOCKS Proxy for Amazon EC2

Pausing a Cluster in AWS