Pausing Cloudera Director Instances

The Cloudera Director server can be stopped at any time to reduce costs, as can clusters running on Microsoft Azure. These actions are described in the sections below.

Pausing the Cloudera Director Server

Although you can stop and start the Cloudera Director server at any time, you should wait for running workflows to complete.

To start or stop the server, enter the following command:

$ sudo service cloudera-director-server [start | stop]

Pausing a Cluster in Azure

There are times when it makes sense to shut down a cluster and stop your Azure virtual machines. The cluster will not be immediately available while stopped and it cannot be used to ingest or process data, but you won't be billed by Microsoft for virtual machines that are stopped and deallocated. Provisioned premium storage disks, and space used by standard storage disks will continue to accrue charges.

Pausing a cluster requires using premium or standard storage disks for all storage, both on management and worker nodes. Data stored on ephemeral disks will be lost after a cluster pause.

Shutdown procedure

To pause the cluster, take the following steps:
  1. Stop the cluster
  2. Stop Cloudera Management Service
  3. Stop all cluster virtual machines, including the Cloudera Manager host

Startup procedure

To restart the cluster after a pause, the steps are reversed:
  1. Start all cluster virtual machines
  2. Start Cloudera Management Service
  3. Start the cluster
Since the cluster was completely stopped before stopping the virtual machines, the cluster should be healthy upon restart and ready for use.

Considerations

After starting the virtual machines, Cloudera Manager and its agents will be running but the cluster will be stopped. There will be gaps in Cloudera Manager’s time-based metrics and charts.

Azure virtual machines that are configured with static private IP addresses will retain their internal IP address and hostname as long as the associated network interface object is not removed, so no reconfiguration of CDH is required after restart. If your nodes were provisioned via Cloudera Director, you should confirm that the IP configuration of your hosts is set to static before shutting down your cluster. The default for the Azure plugin in Cloudera Director 2.4.0 and earlier is dynamic. If you need to change the IP configuration from dynamic to static it can be done via the Azure Portal or PowerShell.

Dynamic public IP addresses will be different after virtual machine shutdown. Static public IP addresses can be configured to remain associated with a stopped virtual machine at additional cost, but it isn’t necessary to maintain proper cluster operation.