This is the documentation for Cloudera Manager 5.1.x. Documentation for other versions is available at Cloudera Documentation.

Rolling Restart

Required Role:

  Important: This feature is available only with a Cloudera Enterprise license.
For other licenses, the following applies:
  • Cloudera Express - The feature is not available.
  • Cloudera Enterprise Data Hub Edition Trial - The feature is available until you end the trial or the trial license expires.
To obtain a license for Cloudera Enterprise, please fill in this form or call 866-843-7207. After you install a Cloudera Enterprise license, the feature will be available.

Rolling restart allows you to conditionally restart the role instances of HDFS, MapReduce, HBase, ZooKeeper, and Flume services to update software or use a new configuration. If the service is not running, rolling restart is not available for that service. You can do a rolling restart of each service individually.

If you have HDFS High Availability enabled, you can also perform a cluster-level rolling restart. At the cluster level, the rolling restart of worker hosts is performed on a host-by-host basis, rather than per service, to avoid all roles for a service potentially being unavailable at the same time. During a cluster restart, in order to avoid having your NameNode (and thus the cluster) being unavailable during the restart, Cloudera Manager will force a failover to the standby NameNode.

JobTracker High Availability is not required for a cluster-level rolling restart. However, if you have JobTracker High Availability enabled, Cloudera Manager will force a failover to the standby JobTracker.

Performing a Service or Role Rolling Restart

You can initiate a rolling restart from either the Status page for one of the eligible services, or from the service's Instances page, where you can select individual roles to be restarted.

  1. Select the service you want to restart.
  2. Do one of the following:
      1. Select Actions > Rolling Restart.
      1. Click the Instances tab.
      2. Select the roles to restart.
      3. Select Actions for Selected > Rolling Restart.
  3. In the pop-up dialog box, select the options you want:
    • Restart only roles whose configurations are stale
    • Restart only roles that are running outdated software versions
    • Which role types to restart
  4. If you select an HDFS, HBase, or MapReduce service, you can have their worker roles restarted in batches. You can configure:
    • How many roles should be included in a batch - Cloudera Manager restarts the slave roles rack-by-rack in alphabetical order, and within each rack, hosts are restarted in alphabetical order. If you are using the default replication factor of 3, Hadoop tries to keep the replicas on at least 2 different racks. So if you have multiple racks, you can use a higher batch size than the default 1. But you should be aware that using too high batch size also means that fewer slave roles are active at any time during the upgrade, so it can cause temporary performance degradation. If you are using a single rack only, you should only restart one slave node at a time to ensure data availability during upgrade.
    • How long should Cloudera Manager wait before starting the next batch.
    • The number of batch failures that will cause the entire rolling restart to fail (this is an advanced feature). For example if you have a very large cluster you can use this option to allow failures because if you know that your cluster will be functional even if some worker roles are down.
      Note:
    • HDFS - If you do not have HDFS High Availability configured, a warning appears reminding you that the service will become unavailable during the restart while the NameNode is restarted. Services that depend on that HDFS service will also be disrupted. It is recommended that you restart the DataNodes one at a time—one host per batch, which is the default.
    • HBase - Administration operations such as any of the following should not be performed during the rolling restart, to avoid leaving the cluster in an inconsistent state:
      • Split
      • Create, disable, enable, or drop table
      • Metadata changes
      • Create, clone, or restore a snapshot. Snapshots rely on the RegionServers being up; otherwise the snapshot will fail.
    • MapReduce - If you restart the JobTracker, all current jobs will fail.
    • ZooKeeper and Flume - For both ZooKeeper and Flume, the option to restart roles in batches is not available. They are always restarted one by one.
  5. Click Confirm to start the rolling restart.

Performing a Cluster-Level Rolling Restart

You can perform a cluster-level rolling restart on demand from the Cloudera Manager Admin Console. A cluster-level rolling restart is also performed as the last step in a rolling upgrade when the cluster is configured with HDFS High Availability enabled.

  1. If you have not already done so, enable High Availability. See HDFS High Availability for instructions. You do not need to enable Automatic Failover for rolling restart to work, though you can enable it if you wish. Automatic Failover does not affect the rolling restart operation.
  2. From the Actions menu for the cluster you want to restart select Rolling Restart....
  3. In the pop-up dialog box, select the services you want to restart. Please review the caveats in the preceding section for the services you elect to have restarted. The services that do not support rolling restart will simply be restarted, and will be unavailable during their restart.
  4. If you select an HDFS, HBase, or MapReduce service you can have their worker roles restarted in batches. You can configure:
    • How many roles should be included in a batch - Cloudera Manager restarts the slave roles rack-by-rack in alphabetical order, and within each rack, hosts are restarted in alphabetical order. If you are using the default replication factor of 3, Hadoop tries to keep the replicas on at least 2 different racks. So if you have multiple racks, you can use a higher batch size than the default 1. But you should be aware that using too high batch size also means that fewer slave roles are active at any time during the upgrade, so it can cause temporary performance degradation. If you are using a single rack only, you should only restart one slave node at a time to ensure data availability during upgrade.
    • How long should Cloudera Manager wait before starting the next batch.
    • The number of batch failures that will cause the entire rolling restart to fail (this is an advanced feature). For example if you have a very large cluster you can use this option to allow failures because if you know that your cluster will be functional even if some worker roles are down.
  5. Click Confirm to start the rolling restart. While the restart is in progress, the Command Details page shows the steps for stopping and restarting the services.
Page generated September 3, 2015.