Reference Architecture for Deploying Cloudera Enterprise 5.x on Red Hat OpenStack Platform 11 with Red Hat Ceph Storage 2.x

In order to validate that the Ceph backend is adequately performant as well as scalable, we strongly advise that the following test methodology be applied in parallel from multiple client nodes to ensure that multiple clients accessing the Ceph cluster can experience deterministic and reliable throughput and latency responses.

We also strongly recommend re-validating backend performance each time additional Ceph nodes and/or client nodes are added to the environment.

The methodology provided by RedHat will be able to test whether the Ceph backend can provide the required throughput.

Once this is ascertained, then, a CDH cluster built on VMs, that have sufficient storage allocated and appropriate number of volumes provided, can be instantiated and standard MapReduce tests such as TeraGen, TeraSort and TeraValidate run against the cluster, to ascertain whether the CDH cluster is able to yield the required throughput as well.

Ceph Backend Logical Topology Diagram 4

The performance of the Ceph RBD backend is predicated on the number of SSDs and types of SSDs being used for the journal write acceleration. For instance, if you have a SATA SSD rated to do sequential writes of ~ 380MB/s you can use that to back 3-4 SATA OSD drives, considering we want to get 100-120MB/s sequential write throughput per SATA drive. So in a node with 24 SATA drives, you will need 5-6 SATA SSDs to provide best overall throughput.

If you use NVMe SSDs that are rated for much higher sequential write throughput of say ~ 900MB/s, you can use it to back 8-9 SATA OSD drives.

The following parameters should be considered in the Ceph configuration to allow for larger journal devices --

• Using block devices instead of directories implies that the entire block device gets used as a journal device. A single SSD can be partitioned, for example a 500GB drive can be partitioned into five 100GB partitions and each partition allocated as journal device for an OSD spinning drive.
• Adjust the OSD op threads parameter from default of 2 to larger value. This parameter helps the Ceph OSD daemon multi-thread servicing requests. For instance, in /etc/ceph/ceph.conf file, set : osd op threads = 30
• The Ceph journal filestore max sync interval determines the interval at which the journal quiesces writes and synchronizes the filesystem, which creates a consistent commit point. It can then free journal entries up to the commit point. It defaults to 5 seconds and increasing this can improve write and metadata performance. For example, in /etc/ceph/ceph.conf, set : filestore max sync interval = 15. For a more detailed understanding on this subject, read the RedHat Ceph Storage Configuration Reference.
• The effective throughput one can achieve from the VM’s perspective is also dependent on the network throughput available between the compute nodes and the Ceph storage nodes. Ideally we should have no over-subscription between the OSP Compute and the Ceph Storage nodes.

Consider using SSDs similar to Intel’s DC P4600 or DC P4800 series NVMe SSDs for best performance for Ceph Storage nodes.

Following rules of thumb should be considered while designing the Ceph backend for CDH --

• Definitely consider using 3x replication on the Ceph RBD pools. It is to be noted however, that we should still strongly recommend keeping at least 2x replication in place in HDFS for higher availability. With Ceph, 3x replication allows inconsistencies found during integrity checking (scrubbing) to be resolved.
• Separate network for storage
• Ceph storage nodes should have 2x bandwidth as compared to compute nodes. So if compute nodes have 1 x 10gbe NIC for storage, Ceph nodes should have 2 x 10gbe NICs, bonded in link-aggregation mode (and not failover mode).
• For production workloads, have at least 8 storage nodes.
• RAID Controller considerations for the storage nodes - follow best practices. There is no RAID being used at the physical disk layer - just use disks as JBOD. Ceph’s RADOS architecture will provide the fault-tolerance and redundancy required.
• Have nodes with 12-36 disks (higher density drives - 4TB at least).
• Consider having flash-based pools for local and intermediate data (OS, YARN local-dirs, impalad scratch dirs)
• Each OSD (disk) should should have 1 GHz of 1 core of CPU and 2GB of RAM (so if node has 24 disks, there should be 24 GHz of Compute (so 24 HT CPUs) and 48GB of RAM (so consider at least a server with 12 physical cores and 64GB of RAM for example).
• 1 NVMe SSD as a journal for every 9-8 OSD spindles or 1 SATA SSD for every 3-4 OSD spindles.
• Recommended filesystem for ceph storage devices is XFS. Follow RedHat’s best practices for tuning.

The Ceph cluster can be set up either as an independent cluster, or following Red Hat’s instructions of setting up a Ceph cluster via OSP director. Setting it up independently gives us better control over the configuration.

Some key steps towards that are to configure the Ansible playbooks for Monitors and OSDs and then executing ansible-playbook run to set the cluster up.

Red Hat provides a PG (Placement Group) Calculator to help properly identify the number of placement groups as well as download the commands that should be run to create the storage pools. This tool also provides a good explanation on the relationship between OSDs and PGs per OSD.

After running numbers through the calculator, a commands text file is emitted, that can be then executed as root on the ceph cluster.

## Note:
## The 'while' loops below pause between pools to allow all PGs to be created.
## This is a safety mechanism to prevent saturating the Monitor nodes.
##-------------------------------------------------------------------

ceph osd pool create cinder-backup 2048
ceph osd pool set cinder-backup size 3
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create cinder-volumes 4096
ceph osd pool set cinder-volumes size 3
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create ephemeral-vms 2048
ceph osd pool set ephemeral-vms size 3
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create glance-images 512
ceph osd pool set glance-images size 3
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

Once the ceph backend is configured, the OpenStack controller(s) and the Compute hosts need to be configured to work with the ceph backend. The references section of this document has (links to) instructions on how glance and cinder on the controller node needs to be configured to support ceph. Also provided are details on how to configure the controller and nova compute nodes as ceph clients to handle ceph.

Towards that end, the following things need to be highlighted --

1. Packages needed to set up the ceph clients - python-rbd and ceph-common.
   1. yum install -y python-rbd ceph-common
2. Configuration files/keyring files needed for ceph client configuration
   1. /etc/ceph/ceph.conf
   2. /etc/ceph/*.keyring files (that were configured for cephx authentication to work for specific user ids, or the admin id as per documentation)
   3. the secret.xml and the ceph.client.<user>.key files that will be needed to set up libvirt security on the compute nodes.
   4. /etc/nova/nova.conf file updated with support for rbd.

      The file will contain similar entries under the [libvirt] section --

      images_type = rbd
      images_rbd_pool = vmpool
      images_rbd_ceph_conf = /etc/ceph/ceph.conf
      rbd_user = cinder
      rbd_secret_uuid = 15d2e1d8-502e-4c4e-87db-c0b8930473cf

   5. /etc/glance/glance-api.conf

      [glance_store]
      stores = rbd,file
      rbd_store_pool = imgpool
      rbd_store_user = glance
      rbd_store_ceph_conf = /etc/ceph/ceph.conf
      rbd_store_chunk_size = 8

   6. /etc/cinder/cinder.conf

      [default]
      enabled_backends=lvm,rbd
      backup_driver = cinder.backup.drivers.ceph
      backup_ceph_conf = /etc/ceph/ceph.conf
      # This ideally is cinder-backup
      backup_ceph_user = cinder-backup
      backup_ceph_chunk_size = 134217728
      backup_ceph_pool = bkuppool
      backup_ceph_stripe_unit = 0
      backup_ceph_stripe_count = 0
      restore_discard_excess_bytes = true
      

      Create a new section called [rbd]

      [rbd]
      volume_driver = cinder.volume.drivers.rbd.RBDDriver
      volume_backend_name=rbd
      rbd_pool = volumes
      rbd_ceph_conf = /etc/ceph/ceph.conf
      rbd_flatten_volume_from_snapshot = false
      rbd_max_clone_depth = 5
      rbd_store_chunk_size = 4
      rados_connect_timeout = -1
      glance_api_version = 2
      # this ideally is cinder
      rbd_user = cinder
      rbd_secret_uuid = 15d2e1d8-502e-4c4e-87db-c0b8930473cf

After making the configuration updates, the respective OpenStack services need to be restarted. In addition to the configuration file changes listed above, run the following from the Controller node --

# source ~/keystonerc_admin

[root@vb0324 ~(keystone_admin)]# cinder type-create rbd
[root@vb0324 ~(keystone_admin)]# cinder type-key rbd set volume_backend_name=rbd
[root@vb0324 ~(keystone_admin)]# cinder type-list
+--------------------------------------+-------+
|                  ID                  |  Name |
+--------------------------------------+-------+
| 525f3c08-0180-4612-bf91-6c94aef6da67 | iscsi |
| aa5627ec-bd28-4c13-8ce9-185921523219 |  lvm  |
| b853fc32-bab8-4d67-9d7f-c855c9d19161 |  rbd  |
+--------------------------------------+-------+

Once this is done, you should be able to create rbd volumes via cinder.

This will allow you to attach the volumes to your VMs, format them in an FS format of your choosing, mount them and run various performance benchmarking tests.

After your VMs have been instantiated, you can set up CDH 5.x just like you would a bare-metal cluster. The Cloudera section below has relevant details and caveats for virtualized deployment of the platform.

The standard benchmarking tool - teragen/terasort can be used to baseline performance of your virtualized CDH cluster and help optimize both at the OS level as well at the OpenStack and infrastructure level.

Optimization details are out of scope of the current iteration of this document, but similar considerations as performance tuning for bare-metal clusters may be employed.

Setting up concrete QOS rules ensure that IO patterns are evenly distributed among volumes presented to the various VMs in a given cluster.

Set the QOS rules as follows (this is an example, actual values will have to be set experimentally, depending on the actual workload) --

$ openstack volume qos create --property read_bytes_sec=209715200 \
--property write_bytes_sec=209715200 --property read_iops_sec=800 \
--property write_iops_sec=400 --consumer front-end datanode

This sets up the QOS rule. The rule is then associated with a cinder volume type.

openstack volume qos associate datanode rbd

This tells cinder that each volume of type “rbd” needs to be set up with the QOS properties defined in the QOS type “datanode”.

The QOS rules show up in the VM xml config as follows (you can generate that by running virsh dumpxml <domain id>) --

<disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <auth username='xxxx'>
        <secret type='ceph' uuid='6f03c880-4ca6-4d6b-997b-e82ad6ed8055'/>
      </auth>
      <source protocol='rbd' name='rbd/volume-dfb3a0d6-a150-4b7f-86ce-776280da47d2'>
        <host name='10.17.204.23' port='6789'/>
        <host name='10.17.204.24' port='6789'/>
        <host name='10.17.206.11' port='6789'/>
      </source>
      <backingStore/>
      <target dev='sdc' bus='scsi'/>
      <iotune>
        <read_bytes_sec>209715200</read_bytes_sec>
        <write_bytes_sec>209715200</write_bytes_sec>
        <read_iops_sec>800</read_iops_sec>
        <write_iops_sec>400</write_iops_sec>
      </iotune>
      <serial>dfb3a0d6-a150-4b7f-86ce-776280da47d2</serial>
      <alias name='scsi0-0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>