Cloudera Director and Cloudera Manager Usage
Cloudera Director works with Cloudera Manager and the cloud service provider to provide centralized and programmatic administration of clusters in the cloud, including deployment, configuration, and maintenance of CDH clusters. With Cloudera Director, you can monitor and manage multiple Cloudera Manager and CDH deployments, across different cloud environments.
- With Cloudera Director 2.4 and higher and Cloudera Manager and CDH 5.11 and higher, many changes made directly in Cloudera Manager are detected by Cloudera Director, which periodically refreshes its state to reflect the state of the cluster in Cloudera Manager. Cloudera Director also refreshes its stored templates for the cluster so that your updated configuration is used if you create more instances or clone the cluster.
- With Cloudera Director 2.3 and lower or Cloudera Manager and CDH 5.10 or lower, if you perform certain administrative tasks in Cloudera Manager, Cloudera Director and Cloudera Manager will become out of sync. When Cloudera Director and Cloudera Manager are out of sync, Cloudera Director cannot grow or shrink the cluster or perform other updates to the cluster. You can use Cloudera Director 2.3 and lower to deploy new Cloudera Manager instances and clusters, but Cloudera Manager instances that are out of sync with Cloudera Director will function independently of Cloudera Director.
When to Use Cloudera Director
Use Cloudera Director when you want to perform the following types of tasks:
- Deploying Cloudera Manager and CDH clusters for prototyping.
- Deploying Cloudera Manager and CDH clusters when you have finalized the topology and configuration.
- Growing or shrinking a cluster. If you have made changes to the cluster using Cloudera Manager, update Cloudera Director with the changes and redeploy the cluster before you grow or shrink the cluster.
- Setting up clusters with Kerberos authentication or high availability.
When to Use Cloudera Manager
- Adding or removing a service in an existing cluster.
- Changing role assignments for an existing virtual instance group, or migrate roles from one instance to another
- Changing the configuration of a service or role
- Testing and iterating on the topology and configuration of clusters.
Use Cloudera Director to create the cluster when you have finalized the topology and configuration.
CDH Cluster Management Tasks
When you deploy CDH and Cloudera Manager through Cloudera Director, you use Cloudera Director or Cloudera Manager to manage the clusters, depending on the task.
The following table lists cloud administrative tasks and the application where you must perform them to avoid inconsistencies in Cloudera Director and Cloudera Manager:
| Task | Application | Notes |
|---|---|---|
| Cluster setup | Cloudera Director | Cloudera Director cannot manage clusters that are set up directly in Cloudera Manager. |
| Addition of host to a cluster or addition of cluster to Cloudera Manager | Cloudera Director | |
| Host decommission | Cloudera Director | This is done by deleting the instance from the virtual instance group using Cloudera Director. |
| Adding a service | Cloudera Manager | If you add a service to a cluster in Cloudera Manager, Cloudera Director will detect the change and will update its cluster template to match. |
| Removing a service | Cloudera Manager |
If you remove a service from a cluster in Cloudera Manager, Cloudera Director will detect the change and will update its cluster template to match. You can stop a service instead of removing it from a cluster. You can also use the grow and shrink feature of Cloudera Director to create hosts that do not have that service’s roles. |
| Initial role assignment | Cloudera Director | |
| Add new virtual instance groups to a cluster | Cloudera Director | |
| Change role assignments for an existing virtual instance group, or migrate roles from one instance to another | Cloudera Manager | Cloudera Director periodically refreshes its data on the state of cluster roles in existing virtual instance groups to include changes made with Cloudera Manager. The changes must not result in inconsistency with respect to the roles included in different instances of the same virtual instance group. See Ensuring Consistency of Virtual Instance Groups below for more information. |
| Changes to the configuration of a service or role | Cloudera Manager | Cloudera Director will detect service and role configuration changes made in Cloudera Manager and will update the cluster template and instance templates to match. The changes must not result in inconsistency with respect to the roles included in different instances of the same virtual instance group. See Ensuring Consistency of Virtual Instance Groups below for more information. |
| Cloud provider settings for instances, such as the machine image or instance type | Cloud provider management tool | Cloudera Director will detect configuration changes made in the cloud provider, including changes in the instance type and machine image, and will update the cluster template and instance templates to match. The changes must not result in inconsistency with respect to the cloud provider configuration included in different instances of the same virtual instance group. In AWS, for example, you can change the AMI or instance type directly in the AWS management console, but you must change all instances in the instance group to be identical. See Ensuring Consistency of Virtual Instance Groups below for more information. |
| Adding, removing, and modifying parcels | Cloudera Manager | If you activate or deactivate parcels in Cloudera Manager, Cloudera Director will detect this change and update its cluster template to match. Parcel version changes will also be detected. Note that when deactivating a parcel in Cloudera Manager, the associated services for that parcel should also be removed through Cloudera Manager. |
| Cloudera Manager username and password change | Cloudera Manager and Cloudera Director | Change the username and password in Cloudera Manager. After you change the username and password in Cloudera Manager, you must update the information in Cloudera Director. If you do not update the information in Cloudera Director, Cloudera Director will not be able to monitor or modify the cluster. |
| Upgrading a Cloudera Manager license | Cloudera Manager |
Use Cloudera Manager to upgrade from Cloudera Express to Cloudera Enterprise. Cloudera Director will not display the state of the updated license, but will not prevent Cloudera Enterprise functionality. |
| Minor version upgrade to Cloudera Manager or CDH | Cloudera Manager |
You must upgrade Cloudera Manager manually and then use the upgraded Cloudera Manager to upgrade CDH. Cloudera Director will detect the version changes, and new clusters will use the upgraded versions. |
| Enabling high availability during cluster setup | Cloudera Director |
High availability is supported in Cloudera Director version 2.0 or higher. Use the configuration file to enable high availability. Do not use the Cloudera Director web UI. |
| Enabling high availability in an existing cluster | Cloudera Manager | See High Availability in the Cloudera Manager documentation for more information. |
| Modifying a cluster in a highly available deployment | Cloudera Director |
If you enable high availability in Cloudera Manager, you can run modify operations only on instance groups that do not contain highly available master roles. |
| Enabling Kerberos authentication during cluster setup | Cloudera Director |
Kerberos setup is supported in Cloudera Director version 2.0 or higher. Use the configuration file to enable Kerberos during cluster setup. After the cluster is created, the configuration file can no longer be used to enable Kerberos. Do not use the Cloudera Director web UI. If you use Cloudera Director to deploy a cluster but use Cloudera Manager to enable Kerberos authentication, Cloudera Director and Cloudera Manager will become out of sync. |
Ensuring Consistency of Virtual Instance Groups
All instances in a virtual instance group must have identical roles assigned to them, with identical role configurations, in order to enable cluster modifications in Cloudera Director. When using Cloudera Director 2.4 and above with Cloudera Manager and CDH 5.11 and above, you can change role assignments, role configurations, and cloud provider settings in Cloudera Manager, but you must ensure that all instances in a given virtual instance group are configured identically. Cloudera Director will then propagate any changes you make in Cloudera Manager back into the cluster's instance templates.
If you make changes to an instance that create role assignments or configurations different from those of other instances in the virtual instance group, Cloudera Director will detect the inconsistency and will flag the virtual instance group in the Cloudera Director UI, identifying which instance is inconsistent what the inconsistencies are. You will not be able to grow that instance group until the inconsistency is fixed.
- In Cloudera Manager, assign or remove roles in the instances that are flagged as inconsistent so that they are identical to the other instances in the virtual instance group
- In Cloudera Director, shrink the virtual instance group to remove the instances that are flagged as inconsistent
CDH Cluster Management Guidelines for Cloudera Director
When you use Cloudera Director to deploy Cloudera Manager and CDH, the cluster information is saved in the Cloudera Director database. If you make changes to the cluster using the cloud provider management console, the changes are detected by Cloudera Director. But terminating an instance using the cloud provider management console results in poor health of the hosts and services in Cloudera Director. If the health of an instance turns bad or the instance fails, you can migrate to a new instance. Use the Cloudera Director web UI to shrink and grow the worker nodes and migrate the master node to a new instance.
For information about growing or shrinking a cluster, see Modifying the Number of Instances in an Existing Cluster.
For information about migrating HDFS master roles to a new instance, see Migrating HDFS Master Roles.