Cloudera Docs

Method 1: Rolling Upgrade

This topic describes the rolling upgrade process, which allows you to update host operating systems in sequential batches. This method maintains cluster availability and prevents downtime during migration.

Step 1: Planning the Host Upgrade Sequence

Creating a strategic upgrade sequence is vital for maintaining cluster stability and data availability. You must evaluate your cluster's rack awareness and replication factors to ensure that taking specific nodes offline does not compromise data access or compute capacity. This planning phase prevents accidental service outages by coordinating maintenance windows and managing the parallel upgrade of nodes according to Cloudera’s recommended capacity thresholds.

  • Develop a sequence for taking hosts offline and bringing them back up to maintain sufficient compute capacity and data replicas.

  • Utilize rack awareness to place all hosts within a rack into maintenance mode without losing access to data replicas.

  • Upgrade up to 20 nodes or 10% of the cluster’s capacity in parallel (whichever is smaller), depending on your organization's tolerance for reduced capacity.

  • Manually stop roles for services with single-instance roles that you cannot decommission (for example, Kudu Master in a single-master configuration) prior to the OS upgrade.

Step 2: Image Preparation

Proper image preparation ensures that the new operating system environment is compatible with the Cloudera Manager ecosystem. You must systematically remove legacy packages and parcels that are incompatible with the target OS to prevent startup failures. By pre-configuring the correct repository information and installing matching agent versions, you create a reliable foundation for the re-imaging process, reducing the risk of version mismatches during the host's return to the cluster.

Although it is not necessary to use pre-created images, Cloudera recommends using them to decrease the upgrade time for each host.

Remove Old Packages and Parcels

Packages and parcels for the older operating system will not start on the new operating system. Perform the following steps on every host in the cluster:

  1. Remove old package repository information from the snapshot. Run the following command to remove old packages from the host:
    • RHEL / CentOS
      sudo yum remove cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
sudo rm /etc/yum.repos.d/cloudera*manager.repo*
    • SLES
      sudo zypper remove cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
sudo rm /etc/zypp/repos.d/cloudera*manager.repo*
    • Ubuntu
      sudo apt-get remove cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
sudo rm /etc/apt/sources.list.d/cloudera*.list*
  2. Remove old parcels from the image and empty the default parcel storage directories, or modify these paths if you use a custom configuration:
    sudo rm -rf /opt/cloudera/parcels/*
sudo rm -rf /opt/cloudera/parcel-cache/*
  3. Remove the repository file: 
    sudo rm /etc/yum.repos.d/cloudera*manager.repo*
Add New Packages and Parcels
  1. Ensure the new packages and parcels match your existing Cloudera Manager version, your existing Cloudera Runtime version, and the new operating system.
  2. Copy the Repository File: Copy the cloudera_manager.repo file (or cloudera-manager.list for Ubuntu) that you created during the OS upgrade preparation phase to the same location on all cluster hosts.
  3. Run the following command to update the Package Index:
    sudo apt-get update
  4. Install Cloudera Manager Agent packages for the version of Cloudera Manager currently in use:
    sudo yum clean all
sudo yum install cloudera-manager-daemons cloudera-manager-agent
  5. Empty the following directories, as the backup overwrites them after you apply the image.
    • /etc/cloudera-scm-agent
    • /etc/default/cloudera-scm-agent
    • /var/run/cloudera-scm-agent
    • /var/lib/cloudera-scm-agent
  6. You can save time by skipping the parcel distribution step. To do this, add the Cloudera Platform version that matches your current cluster to the variant for your new operating system.

Step 3: Upgrade the Cloudera Manager Server

  1. Stop the Cloudera Manager Server and Agent running on the server:
    sudo systemctl stop cloudera-scm-server
sudo systemctl stop cloudera-scm-agent
  2. Back up the following Cloudera Manager directories:
    • /etc/cloudera-scm-agent
    • /etc/default/cloudera-scm-agent
    • /var/run/cloudera-scm-agent
    • /var/lib/cloudera-scm-agent
  3. Re-image the operating system disk with the image prepared in Step 2: Image Preparation.
  4. Ensure the DNS resolves back to the machine using the original hostname.
  5. Restore the following Cloudera Manager directories from your backup:
    • /etc/cloudera-scm-agent
    • /etc/default/cloudera-scm-agent
    • /var/run/cloudera-scm-agent
    • /var/lib/cloudera-scm-agent
  6. Start the Cloudera Manager Server and Agent on the server:
    sudo systemctl start cloudera-scm-server
sudo systemctl start cloudera-scm-agent
  7. Verify and Merge Configurations

    • Confirm that your backed-up configuration files remain intact. Correct any issues if necessary.

    • If you customized the /etc/cloudera-scm-agent/config.ini file, the system renames your custom file with a .rpmsave or .dpkg-old extension. Merge your customizations back into the new config.ini file that the package manager installs.

Step 4: Per-Host Upgrade Process

The per-host upgrade process involves the actual execution of the OS migration while protecting the Cloudera Manager state and data integrity. This workflow guides you through putting nodes into maintenance mode, backing up critical configurations, and safely re-imaging the OS drive. Following these specific steps ensures that the host returns to the cluster with its identity intact and its services ready to resume normal operations without manual reconfiguration.

  1. Place the host into maintenance mode:

    1. Go to the Hosts > All Hosts page in Cloudera Manager by selecting Begin Maintenance (Suppress Alerts/Decommission) from the Actions for Selected menu.
    2. Option 1: For Data Nodes

      Ensure you select decommission host(s) and Take DataNode offline.

      This action pauses replication for a specified duration (default: 240 minutes). Ensure the chosen value provides enough time to reimage the operating system and bring the host back online with the same hostname.

      Option 2: For Compute Nodes

      Ensure you select decommission host(s) for the host, but do not select decommission the DataNode. Decommissioning the DataNode triggers data relocation, which increases the total time required to shut down the node.

  2. Stop the Cloudera Manager Agent:
    sudo systemctl stop cloudera-scm-agent

  3. Back up the following Cloudera Manager directories:

    • /etc/cloudera-scm-agent

    • /etc/default/cloudera-scm-agent

    • /var/run/cloudera-scm-agent

    • /var/lib/cloudera-scm-agent

    • Example
      sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-agent.tar 
--exclude=*.sock /etc/cloudera-scm-agent
/etc/default/cloudera-scm-agent /var/run/cloudera-scm-agent
/var/lib/cloudera-scm-agent
  4. Re-image the operating system disk with the image prepared in Step 2: Image Preparation.

  5. Ensure DNS resolves correctly to the machine using the original hostname.

  6. Restore the following Cloudera Manager directories from your backup:
    • /etc/cloudera-scm-agent
    • /etc/default/cloudera-scm-agent
    • /var/run/cloudera-scm-agent
    • /var/lib/cloudera-scm-agent
  7. Start the Cloudera Manager Agent:
    sudo systemctl start cloudera-scm-agent

  8. Take the host out of maintenance mode.

  9. Confirm the host health by verifying the following conditions:

    1. Cloudera Manager displays the host health as Good.

    2. All roles start successfully and report a Good status.

      1. HDFS completes replication and shows all 3 replicas are available.

      2. Refresh the NameNode if the host resides on a NameNode to ensure it is fully operational.