Generic Reference Architecture for Cloudera Enterprise Running in a Private Cloud

RC = UC x 4²

NN = (RC / ( PND x PDC)) + 3³

NBW = PND x PDT x 8 x 2⁴

CT = PND x NN x PDT

SPN = 2 (assumption)⁵

NS = NN x SPN

CPS = 10 (assumption)

NC = NS x CPS

Working through an example —

Following requirements are available —

• Usable HDFS capacity required is 400TB
• Standard hardware model in datacenter calls for 2-socket boxes with 10 Cores each.
• Standard 4TB SATA drives are used, with 12 drives per node.
• Minimum 128GB RAM per node.

UC = 400TB

CPS = 10 cores

PDC = 4 TB

PND = 12 drives

PDT =100MB/s (good estimate for SATA drives)

RC = UC x 4 = 1600TB

NN = (RC/(PND x PDC)) + 3 = (1600TB/(12x4TB)) + 3 = 37 nodes (rounded)

NBW = PND x PDT x 8 x 2 = 12 x 100MB/s x 8 x 2 = 19200Mbps (~ 20 Gbps)

NS = NN x NS = 37 nodes x 2 sockets = 74 sockets

NC = NS x CPS = 74 x 10 cores = 740 cores

CT = PND x NN x PDT = 12 drives x 37 nodes x 100MB/s = 44,400MB/s

This gives us the following blanket size to start a configuration with —

• 37 nodes - 34 Workers + 3 Masters
• Each node with 2 x 10-Core sockets and 128GB RAM
• Each node with 20Gbps NICs (pair of 10GbE NICs bonded)
• This cluster would theoretically provide about 1.6PB of raw capacity, or ~ 400TB of usable HDFS capacity. It would also generate ~ 44GB/s of sequential IO throughput.
• This cluster would have ~ 740 Physical CPU cores or 1480 Hyper-threaded CPU cores.

²This assumes the standard HDFS replication factor of 3. Add to that the 25% raw storage for intermediate storage, gives us the number 4.

³Here the number 3 represents the minimum number of Master nodes

⁴Here 8 is the factor that converts B/s to b/s (Bytes to bits) and 2 is factoring in 2x network bandwidth recommended for best performance

⁵Here assuming standard 2-socket servers are being considered. This value will change if a more compute heavy node is selected.

NC = RT/PCT

CPS = 10 (assumption)

NS = NC/CPS

SPN = 2 (assumption)

NN = NS/SPN

NBW = (RT x 8 x 2) / NN

STPN = RT / NN

Working through an example with this --

PCT = 50MB/s(assumption)

RT = 20GB/s (based on requirements)

NC = RT/PCT = 20GBps/50MBps = 400

NS = NC/CPS = 400/10 = 40

NN = NS/SPN = 40/2 = 20

NBW = (RT x 8 x 2) / NN = (20GBps x 8 x 2) / 20 = 16Gbps

STPN = RT / NN = 20GBps / 20 = 1 GBps

In this exercise, we start with the requirement that the cluster needs to provide 20GB/s of throughput, and expected per core throughput is 50MB/s.

Certain other assumptions are made, such as each node would have two sockets each with ten cores.

We arrive at a basic envelope size —

Certain other assumptions are made, such as each node would have two sockets each with ten cores.

We arrive at a basic envelope size --

• Require 400 cores or 40 sockets, which results in 20 physical nodes.
• The network bandwidth required to achieve the 50MB/s per core throughput is 16Gbps, or two 10GbE NICs bonded together (or a single 25Gbps NIC).
• This doesn’t do a good job of HDFS capacity sizing, but if we follow the assumption that we would require enough storage spindles to achieve 20GBps of IO throughput, we would require at least 1 GBps of throughput per node.
  • This can help derive the number of spindles per node (either Direct attached or Remote) provided we know the throughput per spindle.
    • For example, if we use Direct attached SATA drives and each drive can provide 100MB/s of throughput⁶, we would need 10 such spindles to provide 1GBps throughput.
    • If we use remote block storage, with each spindle providing say 40MB/s of throughput, we would need 25 such spindles (LUNs). That could call for more VMs to be deployed if a per-VM throughput limitation is in place.
    • This would also allow the derivation of infrastructure details such as, the type of SAN HBA (Host Bus Adapter) that would be needed. 1GBps is 8 gbps, which implies 2 x 8 gbps FC Adapters could be used for fault tolerance, better load balancing across multiple paths and so on (two SAN fabrics are the norm in most shops).
    • The capacity of the storage would have to be calculated based on HDFS capacity required for the cluster.

⁶This is typically on the lower end for SATA spindles. It is not unusual to get up to 120MB/s of sequential IO.

⁷This point was considering mainly the backend spindle count. In shared storage arrays, the L1 and L2 caches can easily be overrun by hadoop workloads. Typically in SAN topologies, network bandwidth is not the concern. Due to the need for multiple (typically 2 for fault tolerance) fabrics, there is sufficient capacity built in. In case of Fiber Channel, 2 x 8 gbps is standard, 2 x 16 gbps more prevalent these days and if using iSCSI, 2 x 10 gbps is the norm.

This section defines two types of hypervisors and what is involved in their selection.

¹⁰“As Sized” here is predicated on a sizing exercise undertaken, or at least based on the recommendations provided in Cloudera’s Hardware Requirements Guide.

¹¹The SDN and Non-SDN demarcation has to do with leveraging the physical networking with minimal virtualization overhead. With SDN features enabled, such as using VXLAN, there is a encapsulation-decapsulation penalty to be paid at the network layer, and that may negatively impact performance.

¹²“As Sized” here is predicated on a sizing exercise undertaken, or at least based on the recommendations provided in Cloudera’s Hardware Requirements Guide.

DAS Nodes

The diagram above illustrates how the storage subsystem of a worker-type node should be designed. The best practice is to obtain hardware with directly attached storage, and pass those devices through to the virtual machines running in this environment with a physical to virtual ratio of 1:1. So each physical spindle at the hypervisor level, maps to a virtual disk spindle at the VM level.

Master Nodes

The diagram above illustrates how a master-type hypervisor system should be designed. The virtual disks provisioned on these VMs can either be Direct Device Pass-throughs as in the storage nodes, or can be a Virtual disk carved out of a RAID5/6¹³ backed disk group (at hypervisor layer).

¹³The RAID level used here should be well thought through, depending on the number of spindles available at the hypervisor layer. If 6-8 disks are being used, then RAID5 or RAID6 would make sense, as they will provide both data protection as well as leverage the spindle count to provide higher throughput.

In SAN-based deployments there are typically two SAN fabrics, which are fault tolerant and most modern SAN HBAs can be effectively multipathed to provide load-balancing and higher aggregate bandwidth, depending on the SAN Storage Array controller (whether it is Active/Active or Active/Passive). The hypervisor layer needs to be aware of this topology and be configured to leverage best performance from the storage backend.

It is also possible to leverage more modern, distributed storage platforms, where-in drivers specific to the Software Defined Storage (SDS) system might need to be installed at the hypervisor layer. The storage then is presented to the VM layer over Ethernet/iSCSI (or similar protocols).

A typical SAN-based topology would resemble the diagram shown above.

With remote block storage devices (RBS) , there may not be a need for as many spindles at the VM level as we would require with locally attached storage. However, the RBS device should be able to provide adequate IO throughput to meet the minimum requirements already articulated for each type of node (masters and workers). And also, it would be operationally prudent to maintain a 1:1¹⁴ mapping of RBS device to virtual storage device.

Moreover, attention must be paid to running stress tests on the infrastructure as articulated in the Storage Acceptance Guide, such that a baseline be established showing that the minimum performance characteristics are being achieved when running workloads in a distributed manner. Such tests should be run at regular intervals to ensure that acceptable SLAs are being maintained by the underlying infrastructure as clusters evolve in their compute needs or as data volumes and data velocities grow.

¹⁴This makes troubleshooting storage hotspotting etc easier to track. Maintaining a 1:1 relationship between LUN and Virtual device ensures that device mappings don’t span Guest VM boundaries.

• No duplicated replicas are on the same node or nodes under the same node group.
• First replica is on the local node or local node group of the writer.
• Second replica is on a remote rack of the first replica.
• Third replica is on the same rack as the second replica.
• The remaining replicas are located randomly across rack and node group for minimum restriction.

The HDFS client obtains a list of replicas for a specific block sorted by distance, from nearest to farthest: local node, local node group, local rack, off rack.

• At the node level, the target and source for balancing follows this sequence: local node group, local rack, off rack.
• At the block level, a replica block is not a good candidate for balancing between source and target node if another replica is on the target node or on the same node group of the target node.

HVE typically supports failure and locality topologies defined from the perspective of virtualization. However, you can use the new extensions to support other failure and locality changes, such as those relating to power supplies, arbitrary sets of physical servers, or collections of servers from the same hardware purchase cycle.

Using Cloudera Manager, configure the following in safety valves:

• HDFS
  • hdfs core-site.xml (Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml/core_site_safety_valve):

    <property>
         <name>net.topology.impl</name>
    <value>org.apache.hadoop.net.NetworkTopologyWithNodeGroup</value>
    </property>
    <property>
         <name>net.topology.nodegroup.aware</name>
         <value>true</value>
    </property>
    <property><name>dfs.block.replicator.classname</name><value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup</value></property>
    

• In mapred-site.xml, add the following properties and values (this is set using the HDFS Replication Advanced configuration snippet (safety valve) mapred-site.xml (mapreduce_service_replication_config_safety_valve)):

  <property>
   <name>mapred.jobtracker.nodegroup.aware</name>
   <value>true</value>
  </property>
  <property><name>mapred.task.cache.levels </name><value>3</value></property>
  

Establish the Topology:

Follow the instructions to set rack location of hosts here — Specifying Racks for Hosts.

Select all multiple hosts from the Hosts page and then assign rack.

Alternately, In Cloudera manager, you can specify the topology by going into the Hosts/Status page and editing the Rack assignment from /default to /default/nodegroup<id>.

The following safety valves need to be applied --

1. HDFS -- Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml
2. YARN - YARN Service MapReduce Advanced Configuration Snippet (Safety Valve) - mapred.xml

Follow this sequence of actions to enable HVE —

• Apply the safety valves
• Assign the rack topology to the nodes
• Stop the cluster
• Deploy client config
• Start ZooKeeper
• Start HDFS

Start all other services.