Advanced Setup: Installing Cloudera Manager and CDH on Azure

Before You Deploy Cloudera Manager and CDH

This topic describes how to set up Cloudera Manager and a CDH cluster in Microsoft Azure using the Altus Director web UI. The following resources must be created and prerequisites must be met before beginning the deployment:
  • An AD application and a service principal for the AD application. The AD application must have the contributor or similar role so that it has permission to create and delete resources in the subscription.
  • A virtual network and network security group that is readily available for the cluster to use.
  • The virtual network configured to use a customer-provided DNS service that supports reverse lookup. If using the provided DNS service setup guide, the VM that provides the DNS service must be created and running.
  • Resource group to house cluster VMs.
  • An Availability Set created in corresponding resource groups to house cluster VMs.
  • Altus Director server VM.
  • Altus Director server installed and running.
  • Altus Director server access to the Azure virtual network (VNet).
  • Database server that is readily available and reachable from the VNet to be used by cluster nodes.

Details of setting up individual items above is covered in earlier sections.

Deploying Cloudera Manager and CDH on Microsoft Azure

This section describes how to create a cluster using the advanced setup procedure in the Altus Director UI. For the simple setup procedure, see Simple Setup: Creating an Azure Cluster with Default Settings.

To deploy Cloudera Manager and CDH on an Azure VM instance, begin by creating an environment as described in Adding an Altus Director Environment on Azure. The environment defines common settings, like region and key pair, that Altus Director uses with Azure.

To deploy Cloudera Manager and launch CDH clusters on Azure, perform the following steps:
  1. Open a web browser and go to the private IP address of the instance you created running Altus Director server. Include port 7189 in the address, for example: http://192.0.2.0:7189.
  2. In the Altus Director login screen, enter admin in both the Username and the Password fields.
  3. In the Altus Director Welcome screen, click Let's get started. This opens a wizard for adding an environment, Cloudera Manager, and a CDH cluster.
  4. Click Continue to add Cloudera Manager.
  5. In the Add Cloudera Manager screen:
    1. Enter a name for this deployment of Cloudera Manager in the Cloudera Manager name field.
    2. In the Instance Template field, select Create New Instance Template.
    3. The Instance Template model screen displays.
  6. In the Instance Template model screen:
    1. In the Instance Template name field, enter a name for the template.
    2. In the VirtualMachine Size field, select one of the available sizes.
    3. In the Image field, select one of the available images or use this format to define any Azure Marketplace VM image inline (replace the italicized values with the actual names for the publisher, offer, sku, and version): 
      /publisher/publisher/offer/offer/sku/sku/version/version
    4. In the Tags field, add one or more tags to associate with the instance.
    5. In the Compute Resource Group field, enter the name of the resource group you created earlier to house the VM.
    6. In the Virtual Network Resource Group field, enter the name where the virtual network resource resides.
    7. In the Virtual Network field, enter the name of the virtual network.
    8. In the Subnet Name field, enter the name of the subnet you want to use.
    9. In the Host FQDN suffix field, enter the name of the host FQDN suffix you want your cluster host to use. This is the DNS domain of your cluster hosts.
    10. In the Network Security Group Resource Group field, enter the name of the resource group where the network security group resource resides.
    11. In the Network Security Group field, enter the name of the network security group.
    12. Select Yes in the Public IP field if you want to assign a public IP address to the VM. The default value is No.
    13. In the Availability Set field, enter the name of the availability set you created in earlier steps. Note that there are two types of Availability Sets: managed (aligned) and unmanaged (classic). For more information see How to Use Availability Sets in the Microsoft Azure documentation.
    14. In the Instance name prefix field under Advanced Options, enter the desired instance name prefix.
    15. In the Storage Account Type field, select Premium_LRS. For instance templates intended for worker nodes, you can select Standard_LRS. See the Cloudera Reference Architecture for Microsoft Azure Deployments for details on supported storage account types and configurations.
    16. In the Data Disk Count field in Advanced Options, enter the number of data disks to attach for the VM.
    17. In the Data Disk Size in GiB field, leave the value at 1024 or pick from the dropdown of available sizes.
    18. Leave the SSH username field blank to use the username you set when you created the environment at step 9.a on the page Adding an Altus Director Environment on Azure.
    19. In the Bootstrap script field in Advanced Options, paste or upload the desired custom bootstrap script.



    20. Select Use Virtual Machine Scale Set (VMSS) in Advanced Options if you want to use a VMSS for the group.

      Azure VMSS is a set of identical VMs that can be created and managed as a group. For more information about VMSS, see Virtual Machines Scale Sets in the Microsoft Azure documentation.

      For more information about using VMSS in Altus Director, see Using Automatic Instance Groups.

  7. In the Desired License Type field, select one of the following license types:
    • Cloudera Enterprise: Includes the core CDH services (HDFS, Hive, Hue, MapReduce, Oozie, Sqoop 1, YARN, and ZooKeeper) and, depending on the license edition, one or more additional services (Accumulo, HBase, Impala, Navigator, Solr, or Spark). For more information on Cloudera Enterprise licenses, see Managing Licenses in the Cloudera Manager documentation.
    • Cloudera Enterprise Trial: A 60-day trial license that includes all CDH services.
    • Cloudera Express: No license required.


    To enable usage-based billing, you must have a Cloudera Enterprise license and a billing ID provided by Cloudera. In the Add Cloudera Manager screen:
    1. In the Desired License Type field, select Cloudera Enterprise.
    2. In the License Key field, either select a Cloudera Enterprise license file to upload or select Direct Input and input the license file text directly into the text area.
    3. To enable usage-based billing, in the Billing ID field, enter the billing ID provided by Cloudera.
  8. By default, the version of Cloudera Manager installed depends on the version of Altus Director you are using:
    Altus Director version Cloudera Manager version installed
    Altus Director 2.0 Latest released version of Cloudera Manager 5.5
    Altus Director 2.1 Latest released version of Cloudera Manager 5.7
    Altus Director 2.2 Latest released version of Cloudera Manager 5.8
    Altus Director 2.3 Latest released version of Cloudera Manager 5.10
    Altus Director 2.4 Latest released version of Cloudera Manager 5.11
    Altus Director 2.5 Latest released version of Cloudera Manager 5.12
    Altus Director 2.6 Latest released version of Cloudera Manager 5.13
    Altus Director 2.7 Latest released version of Cloudera Manager 5.14
    Altus Director 2.8 Latest released version of Cloudera Manager 5.15
    Altus Director 6.0 Latest released version of Cloudera Manager 6.0
    To install a version of Cloudera Manager higher or lower than the default version, perform the following steps:
    1. In the Configurations section, check Override default Cloudera Manager repository.
    2. In the Repository URL field, enter the repository URL for the version of Cloudera Manager to install. Repository URLs for versions of Cloudera Manager 5 have the form https://archive.cloudera.com/cm5/ followed by the operating system, operating system major version, processor architecture, cm (for Cloudera Manager), and the Cloudera Manager major, minor, and (if applicable) maintenance release number. For example, for Cloudera Manager 5.5.4, the repository URL is https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.5.4/.
    3. In the Repository Key URL field, enter the URL for the repository key. Repository key URLs have the same form as repository URLs except they end with the name of the key file instead of the Cloudera Manager version. For example, the repository key URL for any version of Cloudera Manager 5 on any supported version of Red Hat 7 is https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera.
  9. In the Add Cloudera Manager screen, click Continue.
  10. At the Confirmation prompt, click OK to begin adding a cluster.
  11. On the Add Cluster screen:
    1. Enter a name for the cluster in the Cluster name field.
    2. Enter the version of CDH to deploy in the Version field, or leave the default value. By default, the version of CDH installed depends on the version of Altus Director you are using:
      Altus Director version CDH version installed
      Altus Director 2.0 Latest released version of CDH 5.5
      Altus Director 2.1 Latest released version of CDH 5.7
      Altus Director 2.2 Latest released version of CDH 5.9
      Altus Director 2.3 Latest released version of CDH 5.10
      Altus Director 2.4 Latest released version of CDH 5.11
      Altus Director 2.5 Latest released version of CDH 5.12
      Altus Director 2.6 Latest released version of CDH 5.13
      Altus Director 2.7 Latest released version of CDH 5.14
      Altus Director 2.8 Latest released version of CDH 5.15
      Altus Director 6.0 Latest released version of CDH 6.0
      To install a version of CDH higher or lower than the default version, perform the following steps:
      1. Enter the desired CDH version in the Version field of the Products section. For example, for CDH 5.4.8, enter 5.4.8.
      2. Scroll down to Configurations (optional) and expand the section.
      3. Click Override default parcel repositories.
      4. Enter the repository parcel URL for the version of CDH you want to install. Parcel URLs for versions of CDH 5 have the form https://archive.cloudera.com/cdh5/parcels/, followed by the major, minor, and (if applicable) maintenance release number. For example, the URL for CDH 5.4.8 is https://archive.cloudera.com/cdh5/parcels/5.4.8.
    3. In the Services section, select the services you want to install.
    4. In the Instance groups area, create a new template for the groups or for each group and the number of instances you want.


  12. Click Continue.
  13. At the confirmation prompt, click OK to deploy the cluster. Altus Director displays a status screen.

  14. When the cluster is ready, click Continue.