Important Notes About Cloudera Director and Azure

Azure Limits, Quotas, and Constraints

Azure limits the number of CPU cores that can be allocated in each region. For details, see Azure subscription and service limits, quotas, and constraints in the Azure documentation. If you need to increase the limit, contact Microsoft Azure support before deploying the cluster with Cloudera Director.

Not all Azure VM types are available in all Azure regions. See Products available by region on the Microsoft Azure web site to confirm that a VM type is available in a particular region. See Cloudera Reference Architecture for Microsoft Azure Deployments for the latest supported VM types.

Azure Resources Managed by Cloudera Director

The Azure plug-in for Cloudera Director creates the following resources:
  • A storage account for each VM.
  • A NIC for each VM.
  • A public IP address for each VM, if public IP addresses are enabled.

Deploying Production Clusters

Although the Cloudera Director web UI can be used for proof-of-concept deployments on Azure, you must use the published sample configuration files for production deployments (see Useful Links below). You can modify the sample configuration file to fit your specific deployment environment, remove services you do not need, and customize the sample bootstrap script. Configurations related to logging and data storage for individual services must not be changed. Deploying a cluster using the Cloudera Director command-line interface and configuration file based on the examples ensures a repeatable deployment with the proper settings for Azure.

See the Cloudera Reference Architecture for Microsoft Azure Deployments document for more details.

Updating the Azure Plug-in Timeout Value

Azure backend operations usually complete in a few minutes, but in rare cases they take longer, sometimes up to an hour. This can cause Cloudera Director operations such as allocate to fail prematurely. If this happens, you may want to increase the backend polling timeout value in the azure-plugin.conf file.
  1. Download the latest supported azure-plugin.conf file from the Cloudera Director scripts repository.
  2. Find the parameter azure-backend-operation-polling-timeout-second in the provider section.
  3. Change the value to the required duration in seconds.
This procedure changes only the Azure plug-in timeout. The following Cloudera Director timeout values must also be increased in the server's application.properties file to be at least as large as the Azure plug-in configuration values:
  • lp.cloud.databaseServers.allocate.timeoutInMinutes
  • lp.cloud.instances.terminate.timeoutInMinutes
See Setting Cloudera Director Properties for information on setting configuration properties in the server's application.properties file.

Deletion Behavior

The deletion behavior is as follows:
  • The storage account created by the plug-in is used for the VM OS drive and cluster data drive. If you have manually attached a drive from a different storage account not created by the plug-in, it is not deleted.
  • The NIC created by the plug-in is attached to the VM. Only one NIC is used per VM. Do not manually attach NICs to the VM created by the plug-in.
  • Deleting the NIC also deletes the public IP attached to the NIC. This includes public IPs created by Cloudera Director as well as public IPs attached manually.

Useful Links