CDP images hosted in Azure Marketplace

Cloudera publishes virtual hard disks (VHD) images on Azure Marketplace for each minor Runtime release. CDP uses these images by default during environment and Data Hub creation on Azure cloud.

By default, CDP uses VHD images for deploying a CDP environment and Data Hubs. The VHD images go through Microsoft's certification process and are then published on Azure Marketplace for each minor version of Runtime (for example, 7.2.17). Red Hat Enterprise Linux 8 (RHEL 8) images (for Runtime 7.2.17 and newer only) and CentOS 7 images (for Runtime 7.2.17 and earlier only) are available.

In order for CDP to be able to load Cloudera-published virtual machine images in your subscription from the Azure Marketplace, you must first accept Azure Marketplace terms and conditions either via Azure CLI or CDP web UI. If you do not accept the terms and conditions, CDP cannot access the images hosted in Azure Marketplace and so it downloads them from Cloudera’s storage account instead.

Prior to introducing this feature, Cloudera published VHD images for Azure in regional repositories and copied these images to customer’s storage account before environment creation. The images would then remain in the storage account to speed up the provisioning of future environments and clusters. If you use CentOS and you do not fulfill the requirements for Azure Marketplace images, CDP falls back to this storage account method.

The CDP images hosted in Azure Marketplace allow you to:

Deploy your CDP environment, Data Lake and Data Hubs faster than when using an image hosted in Cloudera’s storage account as there is no need to copy the VHD image anymore.
Use RHEL images instead of CentOS images for your CDP environment, Data Lake and Data Hubs. When using CDP on Azure, RHEL 8 images are only available in the Marketplace for Azure.

Cloudera recommends that for best security, performance, and cost-effectiveness you use the Azure Marketplace images.

There are two ways to accept Azure Marketplace terms and conditions:

You can do this by enabling auto-acceptance via the CDP web interface. In this case, you need to do it one time only.
You can do this via Azure CLI. In this case, you need to do it individually for each image set corresponding to a single Runtime version.

Cloudera recommends enabling auto-acceptance via the CDP web interface

Accepting the terms and conditions via CDP UI

You can enable auto-acceptance of Azure Marketplace terms and conditions via CDP web interface. This allows you to accept automatically for all Runtime versions.

Prerequisites

You need to grant the service principal the following Azure permissions on the scope of your Azure subscription:
```
"Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write",
"Microsoft.MarketplaceOrdering/offerTypes/publishers/offers/plans/agreements/read”
```
note
Granting this permissions on the scope of an Azure resource group only is not sufficient. You must grant them on the scope of your Azure subscription.

You need to grant the service principal the following Azure permission on the scope of the CDP Azure resource group:
```
“Microsoft.Resources/deployments/whatIf/action”
```

The Contributor role in Azure includes these permissions.

The Prerequisites for the provisioning credential: Azure permissions have been updated to include the aforementioned permissions.

Required roles

You need to be EnvironmentCreator or PowerUser in CDP.

Steps

Navigate to the Management Console > Global Settings > Azure tab and check the box next to “Enable Terms & Conditions automatic acceptance”. This allows CDP to automatically accept Cloudera image terms and conditions for Azure Marketplace:

Accepting the terms and conditions via Azure CLI

You can accept Azure Marketplace terms and conditions via Azure CLI. In this case, you need to do it individually for each image offering corresponding to a single Runtime version.

Prerequisites

You need to grant the service principal additional Azure permissions on the scope of your Azure subscription:

"Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write",
"Microsoft.MarketplaceOrdering/offerTypes/publishers/offers/plans/agreements/read”

On the scope of the CDP Azure resource group:

“Microsoft.Resources/deployments/whatIf/action”

The Contributor role in Azure includes these permissions.

The Prerequisites for the provisioning credential: Azure permissions have been updated to include the aforementioned permissions.

Steps

Run the following command to obtain image URN for a specific minor Runtime version (this is, a three-digit version such as 7.2.17):

az vm image list -p <NAME-OF-THE-PUBLISHER> --all -f <IMAGE-OFFER>

For example:

az vm image list -p Cloudera --all -f cdp-7_2_17
  {
    "offer": "cdp-7_2_17",
    "publisher": "cloudera",
    "sku": "runtime-7_2_17",
    "urn": "cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077",
    "version": "100.44441663.1694778077"
  },

Run the following command to print the terms and conditions details for the specific image, replacing the <SUBSCRIPTION-NAME-OR-ID> with an actual Azure subscription ID:

az vm image terms show \
  --urn <IMAGE-URN> \
  --subscription "<SUBSCRIPTION-NAME-OR-ID>"

For example:

az vm image terms show \
  --urn cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077 \
  --subscription "azure-eng-cloud-daily"

Among other information, the following details will be printed:

  "accepted": false,
  "id": "/subscriptions/<subscription-id>/providers/Microsoft.MarketplaceOrdering/offerTypes/VirtualMachine/publishers/cloudera/offers/cdp-7_2/plans/runtime-7_2_17/agreements/current",
  "licenseTextLink": "https://mpcprodsa.blob.core.windows.net/legalterms/3E5ED_legalterms_CLOUDERA%253a24CDP%253a2D7%253a5F2%253a24RUNTIME%253a2D7%253a5F2%253a5F6%253a24LLJOPLSWAKNIAHKFQAUSJDXQEUIJ2TS5XNFDHNDYUPRK3LU2T2DJPN2S32U3RVTR4DT2SVCUE5BYHA5UYFXGLSHFKCNQH7YIUS4JNXI.txt",
  "marketplaceTermsLink": "https://mpcprodsa.blob.core.windows.net/marketplaceterms/3EDEF_marketplaceterms_VIRTUALMACHINE%253a24AAK2OAIZEAWW5H4MSP5KSTVB6NDKKRTUBAU23BRFTWN4YC2MQLJUB5ZEYUOUJBVF3YK34CIVPZL2HWYASPGDUY5O2FWEGRBYOXWZE5Y.txt"

Accept the image terms by issuing the following command. The command needs to be issued individually for each image and subscription:

az vm image terms accept \
  --urn <image-urn> \
  --subscription "<subscription-name-or-id>"

For example:

az vm image terms accept \
  --urn cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077 \
  --subscription "azure-eng-cloud-daily"