Cloudera images hosted in Azure Marketplace

Cloudera publishes virtual hard disks (VHD) images on Azure Marketplace for each minor Runtime release. Cloudera uses these images by default during environment and Cloudera Data Hub creation on Azure cloud.

By default, Cloudera uses VHD images for deploying a Cloudera environment and Cloudera Data Hub clusters. The VHD images go through Microsoft's certification process and are then published on Azure Marketplace for each minor version of Cloudera Runtime (for example, 7.2.17). Red Hat Enterprise Linux 8 (RHEL 8) images (for Cloudera Runtime 7.2.17 and newer only) and CentOS 7 images (for Cloudera Runtime 7.2.17 and earlier only) are available.

In order for Cloudera to be able to load Cloudera-published virtual machine images in your subscription from the Azure Marketplace, you must first accept Azure Marketplace terms and conditions either via Azure CLI or Cloudera web UI. If you do not accept the terms and conditions, Cloudera cannot access the images hosted in Azure Marketplace and so it downloads them from Cloudera storage account instead.

Prior to introducing this feature, Cloudera published VHD images for Azure in regional repositories and copied these images to customer’s storage account before environment creation. The images would then remain in the storage account to speed up the provisioning of future environments and clusters. If you use CentOS and you do not fulfill the requirements for Azure Marketplace images, Cloudera falls back to this storage account method.

The Cloudera images hosted in Azure Marketplace allow you to:

Deploy your Cloudera environment, Data Lake and Cloudera Data Hub clusters faster than when using an image hosted in Cloudera’s storage account as there is no need to copy the VHD image anymore.
Use RHEL images instead of CentOS images for your Cloudera environment, Data Lake and Cloudera Data Hub clusters. When using Cloudera on Azure, RHEL 8 images are only available in the Marketplace for Azure.

Cloudera recommends that for best security, performance, and cost-effectiveness you use the Azure Marketplace images.

There are two ways to accept Azure Marketplace terms and conditions:

You can do this by enabling auto-acceptance via the Cloudera web interface. In this case, you need to do it one time only.
You can do this via Azure CLI. In this case, you need to do it individually for each image set corresponding to a single Cloudera Runtime version.

Cloudera recommends enabling auto-acceptance via the Cloudera web interface.

Accepting the terms and conditions via Azure CLI

You can accept Azure Marketplace terms and conditions via Azure CLI. In this case, you need to do it individually for each image offering corresponding to a single Runtime version.

Steps

Run the following command to obtain image URN for a specific minor Cloudera Runtime version (this is, a three-digit version such as 7.2.17):

az vm image list -p <NAME-OF-THE-PUBLISHER> --all -f <IMAGE-OFFER>

For example:

az vm image list -p Cloudera --all -f cdp-7_2_17
  {
    "offer": "cdp-7_2_17",
    "publisher": "cloudera",
    "sku": "runtime-7_2_17",
    "urn": "cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077",
    "version": "100.44441663.1694778077"
  },

Run the following command to print the terms and conditions details for the specific image, replacing the <SUBSCRIPTION-NAME-OR-ID> with an actual Azure subscription ID:

az vm image terms show \
  --urn <IMAGE-URN> \
  --subscription "<SUBSCRIPTION-NAME-OR-ID>"

For example:

az vm image terms show \
  --urn cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077 \
  --subscription "azure-eng-cloud-daily"

Among other information, the following details will be printed:

  "accepted": false,
  "id": "/subscriptions/<subscription-id>/providers/Microsoft.MarketplaceOrdering/offerTypes/VirtualMachine/publishers/cloudera/offers/cdp-7_2/plans/runtime-7_2_17/agreements/current",
  "licenseTextLink": "https://mpcprodsa.blob.core.windows.net/legalterms/3E5ED_legalterms_CLOUDERA%253a24CDP%253a2D7%253a5F2%253a24RUNTIME%253a2D7%253a5F2%253a5F6%253a24LLJOPLSWAKNIAHKFQAUSJDXQEUIJ2TS5XNFDHNDYUPRK3LU2T2DJPN2S32U3RVTR4DT2SVCUE5BYHA5UYFXGLSHFKCNQH7YIUS4JNXI.txt",
  "marketplaceTermsLink": "https://mpcprodsa.blob.core.windows.net/marketplaceterms/3EDEF_marketplaceterms_VIRTUALMACHINE%253a24AAK2OAIZEAWW5H4MSP5KSTVB6NDKKRTUBAU23BRFTWN4YC2MQLJUB5ZEYUOUJBVF3YK34CIVPZL2HWYASPGDUY5O2FWEGRBYOXWZE5Y.txt"

Accept the image terms by issuing the following command. The command needs to be issued individually for each image and subscription:

az vm image terms accept \
  --urn <image-urn> \
  --subscription "<subscription-name-or-id>"

For example:

az vm image terms accept \
  --urn cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077 \
  --subscription "azure-eng-cloud-daily"

Accepting the terms and conditions via Cloudera UI

You can enable auto-acceptance of Azure Marketplace terms and conditions via Cloudera web interface. This allows you to accept automatically for all Runtime versions.

Prerequisites

You need to grant the service principal the following Azure permissions on the scope of your Azure subscription:
```
"Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write",
"Microsoft.MarketplaceOrdering/offerTypes/publishers/offers/plans/agreements/read”
```
note
Granting this permissions on the scope of an Azure resource group only is not sufficient. You must grant them on the scope of your Azure subscription.

You need to grant the service principal the following Azure permission on the scope of the Cloudera Azure resource group:
```
“Microsoft.Resources/deployments/whatIf/action”
```

The Contributor role in Azure includes these permissions.

The Prerequisites for the provisioning credential: Azure permissions have been updated to include the aforementioned permissions.

Required roles

You need to be EnvironmentCreator or PowerUser in Cloudera.

Steps

Navigate to the Cloudera Management Console > Global Settings > Azure tab and check the box next to “Enable Terms & Conditions automatic acceptance”. This allows Cloudera to automatically accept Cloudera image terms and conditions for Azure Marketplace:

Accepting the terms and conditions via Azure CLI

You can accept Azure Marketplace terms and conditions via Azure CLI. In this case, you need to do it individually for each image offering corresponding to a single Runtime version.

Steps

Run the following command to obtain image URN for a specific minor Runtime version (this is, a three-digit version such as 7.2.17):

az vm image list -p <NAME-OF-THE-PUBLISHER> --all -f <IMAGE-OFFER>

For example:

az vm image list -p Cloudera --all -f cdp-7_2_17
  {
    "offer": "cdp-7_2_17",
    "publisher": "cloudera",
    "sku": "runtime-7_2_17",
    "urn": "cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077",
    "version": "100.44441663.1694778077"
  },

Run the following command to print the terms and conditions details for the specific image, replacing the <SUBSCRIPTION-NAME-OR-ID> with an actual Azure subscription ID:

az vm image terms show \
  --urn <IMAGE-URN> \
  --subscription "<SUBSCRIPTION-NAME-OR-ID>"

For example:

az vm image terms show \
  --urn cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077 \
  --subscription "azure-eng-cloud-daily"

Among other information, the following details will be printed:

  "accepted": false,
  "id": "/subscriptions/<subscription-id>/providers/Microsoft.MarketplaceOrdering/offerTypes/VirtualMachine/publishers/cloudera/offers/cdp-7_2/plans/runtime-7_2_17/agreements/current",
  "licenseTextLink": "https://mpcprodsa.blob.core.windows.net/legalterms/3E5ED_legalterms_CLOUDERA%253a24CDP%253a2D7%253a5F2%253a24RUNTIME%253a2D7%253a5F2%253a5F6%253a24LLJOPLSWAKNIAHKFQAUSJDXQEUIJ2TS5XNFDHNDYUPRK3LU2T2DJPN2S32U3RVTR4DT2SVCUE5BYHA5UYFXGLSHFKCNQH7YIUS4JNXI.txt",
  "marketplaceTermsLink": "https://mpcprodsa.blob.core.windows.net/marketplaceterms/3EDEF_marketplaceterms_VIRTUALMACHINE%253a24AAK2OAIZEAWW5H4MSP5KSTVB6NDKKRTUBAU23BRFTWN4YC2MQLJUB5ZEYUOUJBVF3YK34CIVPZL2HWYASPGDUY5O2FWEGRBYOXWZE5Y.txt"

Accept the image terms by issuing the following command. The command needs to be issued individually for each image and subscription:

az vm image terms accept \
  --urn <image-urn> \
  --subscription "<subscription-name-or-id>"

For example:

az vm image terms accept \
  --urn cloudera:cdp-7_2_17:runtime-7_2_17:100.44441663.1694778077 \
  --subscription "azure-eng-cloud-daily"