Creating and Modifying Clusters with the Cloudera Director web UI

Before initially launching a CDH cluster, you can use the Cloudera Director web UI to add, delete, or modify the default roles and instance groups. You can also add, remove, or repair instances in an existing cluster.

Continue reading:

Configuring Instance Groups During Cluster Creation

An instance group is a collection of roles that are installed together on one or more instances. When Cloudera Director creates a Cloudera Manager cluster, it includes three default instance groups: masters, workers, and gateway. Each of these instance groups contains roles of the type represented by that instance group, for the CDH services selected for the cluster. For example, if your cluster includes HDFS and YARN, the masters instance group includes the following roles:
  • For HDFS - NameNode, SecondaryNameNode, Balancer
  • For YARN - ResourceManager, JobHistory Server
The workers instance group will include the following roles:
  • For HDFS - DataNode
  • For YARN - NodeManager
The gateway instance group includes a gateway role for HDFS and another for YARN.

For an introduction to master, worker, and gateway roles, see the Cloudera Manager 5 Overview .

Although the default instance groups are automatically configured with roles of a given type (masters, workers, or gateway), you can add any kind of role to any instance group.

When you create a cluster with Cloudera Director, a default set of instance groups and roles, based on the CDH services you include, is displayed in the Instance Groups section of the Add Cluster page:



By clicking Edit Roles, you can see the roles included in each instance group. These roles will be installed on each instance running that instance group. In this example, by clicking Edit Roles for the workers instance group above, you can see that each of the 10 instances that will be installed for the workers instance group will include two roles, an HDFS DataNode and a YARN NodeManager:



You can modify the default configuration of instance groups during cluster creation by doing the following:
  • Change the number of instances for an instance group by clicking the up or down arrows.
  • Delete an instance group by clicking Delete Group at the right end of the row for that instance group.
  • Add roles to an existing instance group by clicking Edit Roles and then Add Role. Available roles for the services in the cluster are displayed. Click a role to add it to the instance group.
  • Add another instance group to the cluster by clicking Add Group, entering a name for the instance group and assigning roles to it, selecting an instance template, and clicking the up or down arrows to choose the number of instances to install.

Modifying the Number of Instances in an Existing Cluster

Cloudera Director can grow or shrink the size of an existing cluster by adding or removing instances.

Adding Instances to a Cluster

  1. Log in to Cloudera Director at http://director-server-hostname:7189. Cloudera Director opens on the All Environments page, which displays the current environments, deployments, and clusters. Click the cluster you want to modify.
  2. Click Modify Cluster to the right of the cluster name. The Modify Cluster page displays the gateway, masters, and workers instance groups and any additional instance groups that have been added to the cluster, with the current number of instances in each instance group.
  3. You can add instances to an existing instance group or create a new instance group and add roles to it.
    • To add instances to an existing instance group, click Edit to the right of the instance group and click the up or down arrows in the Add Instances section to increase the number of workers and gateways to the desired size. Each new instance will contain the same roles as the existing instances of that group.
    • To create a new instance group, click Add Group, enter a name for the instance group, assign roles to it, select an instance template, and click the up or down arrows to choose the desired number of instances of that group to add.

Ensure Consistency of Instances before Modify Operations

Operations that modify a cluster, such as grow and clone operations, require all the instances in an instance group to be consistent, that is, to have the same role assignments and configurations in Cloudera Manager, and the same properties, such as instance type and image, in the cloud provider.

With Cloudera Director 2.5 and higher, the cluster refresher will detect changes to the instances in Cloudera Manager and in the cloud provider. If the instances in an instance group are inconsistent, the UI will flag the inconsistent instances and prevent grow operations on the cluster. During this time, clone operations on the cluster will fail. Using Cloudera Manager or your cloud provider's tools, you must change role assignments or configurations, or change properties in the cloud provider, such as instance type or image, to make the instances in your instance groups consistent. Once all the instances in the virtual instance group are consistent, the instance template will be updated, and subsequent grow operations on the group, or clone operations on the cluster, will use the new role assignments and configurations, and the new cloud provider properties.

If you are unable to make the instances consistent, you can use the following workarounds. If you created your cluster from a configuration file, you can edit the configuration file before using it to create a new cluster. And if you created the cluster via the UI, you can use the export functionality to get a configuration file, and edit that configuration file to create a new cluster with consistency of instances within instance groups.

Removing Instances from a Cluster

  1. Log in to Cloudera Director at http://director-server-hostname:7189.

    Cloudera Director opens on the All Environments page, which displays the current environments, deployments, and clusters. Click the cluster you want to modify.

  2. Click Modify Cluster to the right of the cluster name. The Modify Cluster page displays the gateway, masters, and workers instance groups and any additional instance groups that have been added to the cluster, with the current number of instances in each instance group.
  3. You can remove an entire instance group, including all of its instances, or remove individual instances from an instance group:
    • To remove an entire instance group, click Delete Group at the right end of the row for that instance group.
    • To remove individual instances from an instance group, click Edit near the right end of the row for the instance group. Click the checkbox for each instance you want to remove, and click the Delete button. The instances you select display an action status of To be deleted.
  4. Click OK to continue, Reset to unselect the selected instances and make a new selection, or Cancel to stop without making any changes.
  5. Click Continue to confirm and delete the selected instances.

Rebalancing the Cluster After Adding or Removing Instances

After you add or remove instances from a cluster, HDFS data is likely to be distributed unevenly across DataNodes. Cloudera Director does not rebalance HDFS when you add instances or remove them from the cluster. If you need to rebalance the cluster, you must do so manually as described in HDFS Balancers in the Cloudera Manager documentation.

The need for rebalancing depends on the amount of data in HDFS and the number of instances added or removed during the cluster. Rebalancing is required only when there is a large movement of data. Cloudera recommends rebalancing the cluster through Cloudera Manager if you increase or reduce the number of DataNodes by 30% or more.

Repairing Worker and Gateway Instances in a Cluster

  1. Log in to Cloudera Director at http://director-server-hostname:7189

    Cloudera Director opens on the All Environments page, which displays the current environments, deployments, and clusters. Click the cluster you want to modify.

  2. Click Modify Cluster to the right of the cluster name. The Modify Cluster page displays the gateway, masters, and workers instance groups and any additional instance groups that have been added to the cluster, with the current number of instances in each instance group.
  3. Click Edit next to the instance count for workers or gateways to repair, and select the instances to repair.
  4. Click the Repair button above the list of instances. The instances you selected display an action status of To be repaired.
  5. Click OK to continue, Reset to unselect the selected instances and make a new selection, or Cancel to stop without making any changes.
  6. Click Continue to confirm and repair the selected instances.