Altus Director and Cloudera Manager Usage
Altus Director works with Cloudera Manager and the cloud service provider to provide centralized and programmatic administration of clusters in the cloud, including deployment, configuration, and maintenance of CDH clusters. With Altus Director, you can monitor and manage multiple Cloudera Manager and CDH deployments, across different cloud environments.
- With Altus Director 2.4 and higher and Cloudera Manager and CDH 5.11 and higher, many changes made directly in Cloudera Manager are detected by Altus Director, which periodically refreshes its state to reflect the state of the cluster in Cloudera Manager. Altus Director also refreshes its stored templates for the cluster so that your updated configuration is used if you create more instances or clone the cluster.
- With Altus Director 2.3 and lower or Cloudera Manager and CDH 5.10 or lower, if you perform certain administrative tasks in Cloudera Manager, Altus Director and Cloudera Manager will become out of sync. When Altus Director and Cloudera Manager are out of sync, Altus Director cannot grow or shrink the cluster or perform other updates to the cluster. You can use Altus Director 2.3 and lower to deploy new Cloudera Manager instances and clusters, but Cloudera Manager instances that are out of sync with Altus Director will function independently of Altus Director.
When to Use Altus Director
Use Altus Director when you want to perform the following types of tasks:
- Deploying Cloudera Manager and CDH clusters for prototyping.
- Deploying Cloudera Manager and CDH clusters when you have finalized the topology and configuration.
- Growing or shrinking a cluster. If you have made changes to the cluster using Cloudera Manager, update Altus Director with the changes and redeploy the cluster before you grow or shrink the cluster.
- Setting up clusters with Kerberos authentication or high availability.
When to Use Cloudera Manager
- Adding or removing a service in an existing cluster.
- Changing role assignments for an existing virtual instance group, or migrate roles from one instance to another
- Changing the configuration of a service or role
- Testing and iterating on the topology and configuration of clusters.
Use Altus Director to create the cluster when you have finalized the topology and configuration.
CDH Cluster Management Tasks
When you deploy CDH and Cloudera Manager through Altus Director, you use Altus Director or Cloudera Manager to manage the clusters, depending on the task.
The following table lists cloud administrative tasks and the application where you must perform them to avoid inconsistencies in Altus Director and Cloudera Manager:
| Task | Application | Notes |
|---|---|---|
| Cluster setup | Altus Director | Altus Director cannot manage clusters that are set up directly in Cloudera Manager. |
| Addition of host to a cluster or addition of cluster to Cloudera Manager | Altus Director | |
| Host decommission | Altus Director | This is done by deleting the instance from the virtual instance group using Altus Director. |
| Adding a service | Cloudera Manager | If you add a service to a cluster in Cloudera Manager, Altus Director will detect the change and will update its cluster template to match. |
| Removing a service | Cloudera Manager |
If you remove a service from a cluster in Cloudera Manager, Altus Director will detect the change and will update its cluster template to match. You can stop a service instead of removing it from a cluster. You can also use the grow and shrink feature of Altus Director to create hosts that do not have that service’s roles. |
| Initial role assignment | Altus Director | |
| Add new virtual instance groups to a cluster | Altus Director | |
| Change role assignments for an existing virtual instance group, or migrate roles from one instance to another | Cloudera Manager | Altus Director periodically refreshes its data on the state of cluster roles in existing virtual instance groups to include changes made with Cloudera Manager. The changes must not result in inconsistency with respect to the roles included in different instances of the same virtual instance group. See Ensuring Consistency of Virtual Instance Groups below for more information. |
| Changes to the configuration of a service or role | Cloudera Manager | Altus Director will detect service and role configuration changes made in Cloudera Manager and will update the cluster template and instance templates to match. The changes must not result in inconsistency with respect to the roles included in different instances of the same virtual instance group. See Ensuring Consistency of Virtual Instance Groups below for more information. |
| Cloud provider settings for instances, such as the machine image or instance type | Cloud provider management tool | Altus Director will detect configuration changes made in the cloud provider, including changes in the instance type and machine image, and will update the cluster template and instance templates to match. The changes must not result in inconsistency with respect to the cloud provider configuration included in different instances of the same virtual instance group. In AWS, for example, you can change the AMI or instance type directly in the AWS management console, but you must change all instances in the instance group to be identical. See Ensuring Consistency of Virtual Instance Groups below for more information. |
| Adding, removing, and modifying parcels | Cloudera Manager | If you activate or deactivate parcels in Cloudera Manager, Altus Director will detect this change and update its cluster template to match. Parcel version changes will also be detected. Note that when deactivating a parcel in Cloudera Manager, the associated services for that parcel should also be removed through Cloudera Manager. |
| Cloudera Manager username and password change | Cloudera Manager and Altus Director | Change the username and password in Cloudera Manager. After you change the username and password in Cloudera Manager, you must update the information in Altus Director. If you do not update the information in Altus Director, Altus Director will not be able to monitor or modify the cluster. |
| Upgrading a Cloudera Manager license | Cloudera Manager |
Use Cloudera Manager to upgrade from Cloudera Express to Cloudera Enterprise. Altus Director will not display the state of the updated license, but will not prevent Cloudera Enterprise functionality. |
| Minor version upgrade to Cloudera Manager or CDH | Cloudera Manager |
You must upgrade Cloudera Manager manually and then use the upgraded Cloudera Manager to upgrade CDH. Altus Director will detect the version changes, and new clusters will use the upgraded versions. |
| Enabling high availability during cluster setup | Altus Director |
High availability is supported in Altus Director version 2.0 or higher. Use the configuration file to enable high availability. Do not use the Altus Director web UI. |
| Enabling high availability in an existing cluster | Cloudera Manager | See High Availability in the Cloudera Manager documentation for more information. |
| Modifying a cluster in a highly available deployment | Altus Director |
If you enable high availability in Cloudera Manager, you can run modify operations only on instance groups that do not contain highly available master roles. |
| Enabling Kerberos authentication during cluster setup | Altus Director |
Kerberos setup is supported in Altus Director version 2.0 or higher. Use the configuration file to enable Kerberos during cluster setup. After the cluster is created, the configuration file can no longer be used to enable Kerberos. Do not use the Altus Director web UI. If you use Altus Director to deploy a cluster but use Cloudera Manager to enable Kerberos authentication, Altus Director and Cloudera Manager will become out of sync. |
Ensuring Consistency of Virtual Instance Groups
All instances in a virtual instance group must have identical roles assigned to them, with identical role configurations, in order to enable cluster modifications in Altus Director. When using Altus Director 2.4 and above with Cloudera Manager and CDH 5.11 and above, you can change role assignments, role configurations, and cloud provider settings in Cloudera Manager, but you must ensure that all instances in a given virtual instance group are configured identically. Altus Director will then propagate any changes you make in Cloudera Manager back into the cluster's instance templates.
If you make changes to an instance that create role assignments or configurations different from those of other instances in the virtual instance group, Altus Director will detect the inconsistency and will flag the virtual instance group in the Altus Director UI, identifying which instance is inconsistent what the inconsistencies are. You will not be able to grow that instance group until the inconsistency is fixed.
- In Cloudera Manager, assign or remove roles in the instances that are flagged as inconsistent so that they are identical to the other instances in the virtual instance group
- In Altus Director, shrink the virtual instance group to remove the instances that are flagged as inconsistent
CDH Cluster Management Guidelines for Altus Director
When you use Altus Director to deploy Cloudera Manager and CDH, the cluster information is saved in the Altus Director database. If you make changes to the cluster using the cloud provider management console, the changes are detected by Altus Director. But terminating an instance using the cloud provider management console results in poor health of the hosts and services in Altus Director. If the health of an instance turns bad or the instance fails, you can migrate to a new instance. Use the Altus Director web UI to shrink and grow the worker nodes and migrate the master node to a new instance.
For information about growing or shrinking a cluster, see Modifying the Number of Instances in an Existing Cluster.
For information about migrating HDFS master roles to a new instance, see Migrating HDFS Master Roles.