Cloudera Enterprise Reference Architecture for AWS Deployments

Refer to CDH and Cloudera Manager Supported JDK Versions for a list of supported JDK versions.

Cloudera recommends deploying three or four machine types into production:

• Master Node. Runs the Hadoop master daemons: NameNode, Standby NameNode, YARN Resource Manager and History Server, the HBase Master daemon, Sentry server, and the Impala StateStore Server and Catalog Server. Master nodes are also the location where Zookeeper and JournalNodes are installed. The daemons can often share single pool of servers. Depending on the cluster size, the roles can instead each be run on a dedicated server. Kudu Master Servers should also be deployed on master nodes.
• Worker Node. Runs the HDFS DataNode, YARN NodeManager, HBase RegionServer, Impala impalad, Search worker daemons and Kudu Tablet Servers.
• Utility Node. Runs Cloudera Manager and the Cloudera Management Services. It can also host a MySQL (or another supported) database instance, which is used by Cloudera Manager, Hive, Sentry and other Hadoop-related projects.
• Edge Node. Contains all client-facing configurations and services, including gateway configurations for HDFS, YARN, Impala, Hive, and HBase. The edge node is also a good place for Hue, Oozie, HiveServer2, and Impala HAProxy. HiveServer2 and Impala HAProxy serve as a gateway to external applications such as Business Intelligence (BI) tools.

For more information refer to Recommended Cluster Hosts and Role Distribution.

A public subnet in this context is a subnet with a route to the Internet gateway. Instances provisioned in public subnets inside VPC can have direct access to the Internet as well as to other external services such as AWS services in another region. If your cluster requires high-bandwidth access to data sources on the Internet or outside of the VPC, your cluster should be deployed in a public subnet. This gives each instance full bandwidth access to the Internet and other external services. Unless it’s a requirement, we don’t recommend opening full access to your cluster from the Internet. Using security groups (discussed later), you can configure your cluster to have access to other external services but not to the Internet, and you can limit external access to nodes in the public subnet.

Deployment in the public subnet looks like this:

The public subnet deployment with edge nodes looks like this:

Instances provisioned in private subnets inside VPC don’t have direct access to the Internet or to other AWS services, except when a VPC endpoint is configured for that service. To access the Internet, they must go through a NAT gateway or NAT instance in the public subnet; NAT gateways provide better availability, higher bandwidth, and require less administrative effort. If your cluster does not require full bandwidth access to the Internet or to external services, you should deploy in a private subnet. VPC endpoint interfaces or gateways should be used for high-bandwidth access to AWS services.

Deployment in the private subnet looks like this:

Deployment in private subnet with edge nodes looks like this:

The edge nodes in a private subnet deployment could be in the public subnet, depending on how they must be accessed. The figure above shows them in the private subnet as one deployment option.

Management nodes for a Cloudera Enterprise deployment run the master daemons and coordination services, which may include:

• ResourceManager
• NameNode
• Standby NameNode
• JournalNodes
• ZooKeeper

Allocate a vCPU for each master service. The more master services you are running, the larger the instance will need to be. For example, if running YARN, Spark, and HDFS, an instance with eight vCPUs is sufficient (two for the OS plus one for each YARN, Spark, and HDFS is five total and the next smallest instance vCPU count is eight). If you add HBase, Kafka, and Impala, you would pick an instance type with more vCPU and memory. The memory footprint of the master services tend to increase linearly with overall cluster size, capacity, and activity.

Cloudera supports running master nodes on both ephemeral- and EBS-backed instances.

Utility nodes for a Cloudera Enterprise deployment run management, coordination, and utility services, which may include:

• Cloudera Manager
• JournalNode
• ZooKeeper
• Oozie
• Hive Server
• Impala Catalog Server
• Impala State Store
• Job History Server
• Cloudera Management Services

Refer to Master node requirements.

Worker nodes for a Cloudera Enterprise deployment run worker services, which may include:

• HDFS DataNode
• YARN NodeManager
• HBase RegionServer
• Impala Daemons
• Solr Servers

Allocate a vCPU for each worker service. For example an HDFS DataNode, YARN NodeManager, and HBase Region Server would each be allocated a vCPU. You will need to consider the memory requirements of each service. Some services like YARN and Impala can take advantage of additional vCPUs to perform work in parallel. Consider your cluster workload and storage requirements, determine the vCPU and memory resources you wish to allocate to each service, then select an instance type that’s capable of satisfying the requirements.

DFS is supported on both ephemeral and EBS storage, so there are a variety of instances that can be utilized for Worker nodes.

Hadoop client services run on edge nodes. They are also known as gateway services. Some example services include:

• Third-party tools
• Hadoop command-line client
• Hive command-line client
• Impala command-line client
• Flume agents
• Hue Server
• HBase REST proxy
• HBase Thrift proxy

Edge node services are typically deployed to the same type of hardware as those responsible for master node services, however any instance type can be used for an edge node so long as it has sufficient resources for your use. Depending on the size of the cluster, there may be numerous systems designated as edge nodes.

A detailed list of configurations for the different instance types is available on the EC2 instance types page.

Amazon EC2 provides enhanced networking capacities on supported instance types, resulting in higher performance, lower latency, and lower jitter. In order to take advantage of enhanced networking, you should launch an HVM (Hardware Virtual Machine) AMI in VPC and install the appropriate driver. More details can be found in the Enhanced Networking documentation.

VPC has several different configuration options. See the VPC documentation for detailed explanation of the options and choose based on your networking requirements. You can deploy Cloudera Enterprise clusters in either public or private subnets. In both cases, the instances forming the cluster should not be assigned a publicly addressable IP unless they must be accessible from the Internet. If you assign public IP addresses to the instances and want to block incoming traffic, you can use security groups.

For private subnet deployments, connectivity between your cluster and other AWS services in the same region such as S3 or RDS should be configured to make use of VPC endpoints. VPC endpoints allow configurable, secure, and scalable communication without requiring the use of public IP addresses, NAT or Gateway instances. See the VPC Endpoint documentation for specific configuration options and limitations.

For public subnet deployments, there is no difference between using a VPC endpoint and just using the public Internet-accessible endpoint.

Clusters that do not need heavy data transfer between the Internet or services outside of the VPC and HDFS should be launched in the private subnet. These clusters still might need access to services like software repositories for updates or other low-volume outside data sources. Do this by provisioning a NAT instance or NAT gateway in the public subnet, allowing access outside the private subnet into the public domain. Cloudera does not recommend using NAT instances or NAT gateways for large-scale data movement.

If cluster instances require high-volume data transfer outside of the VPC or to the Internet, they can be deployed in the public subnet with public IP addresses assigned so that they can directly transfer data to and from those services. Configure the security group for the cluster nodes to block incoming connections to the cluster instances.

If you completely disconnect the cluster from the Internet, you block access for software updates as well as to other AWS services that are not configured via VPC Endpoint, which makes maintenance difficult. If you are required to completely lock down any external access because you don’t want to keep the NAT instance running all the time, Cloudera recommends starting a NAT instance or gateway when external access is required and stopping it when activities are complete.

You can establish connectivity between your data center and the VPC hosting your Cloudera Enterprise cluster by using a VPN or Direct Connect. We recommend using Direct Connect so that there is a dedicated link between the two networks with lower latency, higher bandwidth, security and encryption via IPSec. If you don’t need high bandwidth and low latency connectivity between your data center and AWS, connecting to EC2 through the Internet is sufficient and Direct Connect may not be required.

Security Groups are analogous to host firewalls. You can define rules for EC2 instances and define allowable traffic, IP addresses, and port ranges. Instances can belong to multiple security groups. Cloudera Enterprise deployments require the following security groups:

• Cluster
• Flume Nodes
• Edge Nodes

A full deployment in a private subnet using a NAT gateway looks like the following:

Data is ingested by Flume from source systems on the corporate servers. You’ll have flume sources deployed on those machines.

End users are the end clients that interact with the applications running on the edge nodes that can interact with the Cloudera Enterprise cluster.

As described in the AWS documentation, Placement Groups are a logical grouping of EC2 instances that determine how instances are placed on underlying hardware. Cluster Placement Groups are within a single availability zone, provisioned such that the network between them has higher throughput and lower latency. AWS accomplishes this by provisioning instances as close to each other as possible. This limits the pool of instances available for provisioning but guarantees uniform network performance. Spread Placement Groups ensure that each instance is placed on distinct underlying hardware; you can have a maximum of seven running instances per AZ per group.

Cloudera recommends provisioning the worker nodes of the cluster within a cluster placement group. Edge nodes can be outside the placement group unless you need high throughput and low latency between those and the cluster—for example, if you are moving large amounts of data or expect low-latency responses between the edge nodes and the cluster. Master nodes should be placed within a spread placement group to prevent master metadata loss.

EC2 instances have storage attached at the instance level, similar to disks on a physical server. The storage is virtualized and is referred to as ephemeral storage because the lifetime of the storage is the same as the lifetime of your EC2 instance. If you stop or terminate the EC2 instance, the storage is lost. The storage is not lost on restarts, however. Different EC2 instances have different amounts of instance storage, as highlighted above. For long-running Cloudera Enterprise clusters, the HDFS data directories should use instance storage, which provide all the benefits of shipping compute close to the storage and not reading remotely over the network.

When using instance storage for HDFS data directories, special consideration should be given to backup planning. Since the ephemeral instance storage will not persist through machine shutdown or failure, you should ensure that HDFS data is persisted on durable storage before any planned multi-instance shutdown and to protect against multi-VM datacenter events. You can set up a scheduled distcp operation to persist data to AWS S3 (see the examples in the distcp documentation) or leverage Cloudera Manager’s Backup and Data Recovery (BDR) features to backup data on another running cluster.

Amazon Elastic Block Store (EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. HDFS data directories can be configured to use EBS volumes. Encrypted EBS volumes can be used to protect data in-transit and at-rest, with negligible impact to latency or throughput.

There are different types of volumes with differing performance characteristics: the Throughput Optimized HDD (st1) and Cold HDD (sc1) volume types are well suited for DFS storage. From the Amazon ST1/SC1 release announcement:

These magnetic volumes provide baseline performance, burst performance, and a burst credit bucket. While [GP2] volumes define performance in terms of IOPS (Input/Output Operations Per Second), [these] volumes define it in terms of throughput (MB/s).

Baseline and burst performance both increase with the size of the provisioned EBS volume. For example, a 500 GB ST1 volume has a baseline throughput of 20 MB/s whereas a 1000 GB ST1 volume has a baseline throughput of 40 MB/s.

A few considerations when using EBS volumes for DFS:

We strongly recommend using S3 to keep a copy of the data you have in HDFS for disaster recovery. The durability and availability guarantees make it ideal for a cold backup that you can restore in case the primary HDFS cluster goes down. For a hot backup, you need a second HDFS cluster holding a copy of your data.

You can also directly make use of data in S3 for query operations using Hive and Spark. Standard data operations can read from and write to S3. Hive does not currently support use of reference scripts or JAR files located in S3 or LOAD DATA INPATH operations between different filesystems (example: HDFS to S3).

For more information, see Configuring the Amazon S3 Connector.

We require using EBS volumes as root devices for the EC2 instances. When instantiating the instances, you can define the root device size. The root device size for Cloudera Enterprise clusters should be at least 500 GB to allow parcels and logs to be stored. You should not use any instance storage for the root device.

For use cases with lower storage requirements, using r3.8xlarge or c4.8xlarge is recommended. They provide a lower amount of storage per instance but a high amount of compute and memory resources to go with it. For more storage, consider h1.8xlarge.

For use cases with higher storage requirements, using d2.8xlarge is recommended. These provide a high amount of storage per instance, but less compute than the r3 or c4 instances. The d2.8xlarge instances have 24 x 2 TB instance storage. h1.8xlarge and h1.16xlarge also offer a good amount of local storage with ample processing capability (4 x 2TB and 8 x 2TB respectively).

AWS offers the ability to reserve EC2 instances up front and pay a lower per-hour price. This is beneficial for users that are using EC2 instances for the foreseeable future and will keep them on a majority of the time. Reserving instances can drive down the TCO significantly of long-running Cloudera Enterprise clusters. There are different options for reserving instances in terms of the time period of the reservation and the utilization of each instance. See the AWS documentation to plan instance reservation.

We recommend running at least three ZooKeeper servers for availability and durability.

For durability in Flume agents, use memory channel or file channel. Flume’s memory channel offers increased performance at the cost of no data durability guarantees. File channels offer a higher level of durability guarantee because the data is persisted on disk in the form of files. Cloudera supports file channels on ephemeral storage as well as EBS. If the EC2 instance goes down, the data on the ephemeral storage is lost. For guaranteed data delivery, use EBS-backed storage for the Flume file channel.