Deploying CDP in multiple Azure availability zones

Use cases

A multi-AZ Data Lake and FreeIPA constitute a resilient environment that provides a solid basis for multi-AZ Data Hubs and CDP data services. Data Hubs and CDP data services depend on the FreeIPA instance in the Data Lake to provide DNS resolution. Deploying FreeIPA across multiple availability zones ensures that critical DNS resolution is available in the event of an availability zone outage. Furthermore, a medium duty or enterprise Data Lake provides high availability, and additional compute and memory resources for key SDX services and is recommended for production workloads.

Deploying your Data Hubs across multiple availability zones is key if your mission-critical applications depend on HBase and Kafka. Multiple availability zone deployment for operational workloads is considered best practice by the cloud vendors. It ensures that your applications can continue to run in the event of an availability zone outage.

When an entire availability zone fails, HBase automatically rebalances regions among the remaining instances in the cluster to maintain availability. The write-ahead log (WAL), which is replicated across the three availability zones is automatically replayed by the newly assigned region servers in other availability zones to ensure writes to the database are not lost.

When using the multi availability zone feature, CDP ensures that Kafka replicates partitions across brokers in different availability zones. During an availability zone failure this ensures that no data is lost and applications can continue to access the data they need. Cruise Control, which is deployed alongside every Kafka cluster in CDP Public Cloud, detects that topics need to be rebalanced to the remaining brokers. Once the availability zone is back online, you can repair your Kafka cluster, restoring the initial broker distribution across availability zones. Afterwards Cruise Control kicks in and ensures that all topic partitions are balanced across the cluster.

Limitations

The following limitations apply when deploying a multi-AZ CDP:

• When an AZ is down, you cannot create a new Data Hub, and create or activate CDP data services within the environment. Existing workloads will continue to work.

• When an AZ is down, you cannot resize, stop, or restart Data Hubs.

• Non-AZ environments or clusters cannot be converted to multi-AZ.

Azure requirements

In order to use multi-AZ, you should meet the following Azure requirements:

1. The Azure region that you select should support setting up Azure PostgreSQL Flexible Server in Zone-Redundant HA mode and also the instance types to be used. See Flexible Server Azure Regions.

2. The ADLS Gen2 storage account should be created as zone-redundant storage (ZRS). To specify ZRS via Azure CLI during storage account creation, the --sku option should be set to Standard_ZRS. Below is a sample Azure CLI command:

   azure % az storage account create \
    --name test-storage \
    --resource-group rg-test-rg \
    --access-tier Cool \
    --allow-blob-public-access false \
    --allow-cross-tenant-replication false \
    --allow-shared-key-access true \
    --enable-hierarchical-namespace true \
    --skuStandard_ZRS

Register a multi-AZ environment

You can register a multi-AZ AWS environment via CDP UI or CDP CLI. You may choose to enable multi-AZ for Data Lake only or for FreeIPA only. There is no requirement to enable both.

Steps

Create a multi-AZ Data Hub

You can create multi-AZ Data Hubs within any existing environment. Detailed steps are provided below.

Prerequisites

You can create a multi-AZ Data Hub in a multi-AZ environment only. If you are trying to create a multi-AZ Data Hub in an environment that uses the default AZ distribution, you need to first edit that environment and add AZs to it.

Steps

cdp datahub create-azure-cluster \
 --cluster-name test-cluster1 \
 --environment-name test-env \
 --cluster-template-name "7.2.17 - Data Engineering: Apache Spark, Apache Hive, Apache Oozie" \
 --multi-az \

cdp datahub create-azure-cluster \
 --cluster-name test-cluster1 \
 --environment-name test-env \
 --cluster-template-name "7.2.17 - Data Engineering: Apache Spark, Apache Hive, Apache Oozie" \
 --multi-az \
 --instance-groups
          nodeCount=1,instanceGroupName=compute,instanceGroupType=CORE,instanceType=Standard_D5_v2,rootVolumeSize=100,attachedVolumeConfiguration=\[\{volumeSize=100,volumeCount=0,volumeType=StandardSSD_LRS\}\],recoveryMode=MANUAL,availabilityZones=\[1,2\]
          nodeCount=0,instanceGroupName=gateway,instanceGroupType=CORE,instanceType=Standard_D8_v3,rootVolumeSize=100,attachedVolumeConfiguration=\[\{volumeSize=100,volumeCount=1,volumeType=StandardSSD_LRS\}\],recoveryMode=MANUAL,availabilityZones=\[2,3\]
          nodeCount=1,instanceGroupName=master,instanceGroupType=GATEWAY,instanceType=Standard_D16_v3,rootVolumeSize=100,attachedVolumeConfiguration=\[\{volumeSize=100,volumeCount=1,volumeType=StandardSSD_LRS\}\],recoveryMode=MANUAL,availabilityZones=\[1,2,3\] 
          nodeCount=3,instanceGroupName=worker,instanceGroupType=CORE,instanceType=Standard_D5_v2,rootVolumeSize=100,attachedVolumeConfiguration=\[\{volumeSize=100,volumeCount=1,volumeType=StandardSSD_LRS\}\],recoveryMode=MANUAL,availabilityZones=\[1,3\] 

Modify environment's AZs for new Data Hubs

You can modify an environment's AZs. In this case, the added AZs will only be used for new Data Hub clusters. The environment clusters and existing Data Hubs running in it will continue to use the original AZs.

Steps

cdp environments update-azure-availability-zones \
--environment-name test-env \
--availability-zones12 3