Cloudera Docs

Method 2: Offline Upgrade

This topic describes the offline upgrade process, which requires a full cluster shutdown to update all host operating systems simultaneously. This method is faster but causes significant downtime.

Step 1: Safely Shut Down the Cluster

Before modifying the underlying operating system, you must perform a clean shutdown of all active services to prevent data corruption or inconsistent states. This process involves stopping the user-facing clusters, followed by the Cloudera Management Service and the core Cloudera Manager Server. Following a strict "top-down" shutdown sequence ensures that the system safely writes all persistent data to disk and gracefully detaches all service dependencies before the environment goes offline.

Stop Running Clusters
  1. Log in to the Cloudera Manager Admin Console.
  2. On the Home > Status tab, click the options menu to the right of each Cluster and select Stop to stop all the services in the cluster.
  3. Do one of the following to stop the Cloudera Management Service:
      1. Select Clusters > Cloudera Management Service.
      2. Select Actions > Stop.
    • On the Home > Status tab, click the options menu to the right of Cloudera Management Service and select Stop.
Stop Cloudera Manager Server
On the Cloudera Manager Server host, stop the Cloudera Manager Service by running the following command:
sudo systemctl stop cloudera-scm-server
Remove Packages and Parcels

Operating system upgrades change core system library versions, making legacy software components incompatible. You must manually remove the Cloudera Manager Server, Cloudera Manager Agent, and other Cloudera Runtime RPM packages from every host to prepare for a clean environment. This explicit removal ensures that no outdated binaries remain, preventing library conflicts when you reinstall the software optimized for your new operating system version.

Packages and parcels for the older operating system will not start on the new operating system. Perform the following steps on every host in the cluster:

  1. Remove old package repository information from the snapshot. Run the following command to remove old packages from the host:
    • RHEL / CentOS
      sudo yum remove cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
sudo rm /etc/yum.repos.d/cloudera*manager.repo*
    • SLES
      sudo zypper remove cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
sudo rm /etc/zypp/repos.d/cloudera*manager.repo*
    • Ubuntu
      sudo apt-get remove cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
sudo rm /etc/apt/sources.list.d/cloudera*.list*
  2. Remove old parcels from the host and empty the default parcel storage directories, or modify these paths if you use a custom configuration:
    sudo rm -rf /opt/cloudera/parcels/*
sudo rm -rf /opt/cloudera/parcel-cache/*

Step 2: Upgrade the Host Operating System

With all Cloudera roles stopped and legacy packages removed, you can now proceed with the actual operating system migration. It is critical to follow the vendor-specific upgrade paths (such as RHEL leapp or Ubuntu do-release-upgrade) while ensuring that data partitions remain untouched. Maintaining the integrity of these partitions is vital, as they contain the actual HDFS data blocks and service metadata required once the cluster is restarted on the new OS.

  1. Verify that no Hadoop services or Cloudera Manager roles are running.

  2. Upgrade the operating system on all hosts using your vendor’s preferred method (for example, RedHat or Ubuntu).

Step 3: After you upgrade the Operating System

After successfully upgrading the operating system, you must re-establish the Cloudera software environment. This involves verifying Python 3 compatibility, re-registering repository information, and installing the software versions that correspond to your new OS. This phase ensures that the Cloudera Manager Server and Agents are correctly linked to the new system libraries while preserving your previously backed-up configuration files to maintain cluster identity.

  1. Install the correct version of Python 3 before re-installing Cloudera Manager packages. See Python Guidelines for instructions on selecting and installing the required version."
  2. Update Repository
    1. Log in to each cluster host:
      ssh cluster_host
    2. Remove all older files from the existing repository directory:
      RHEL / CentOS
      sudo rm /etc/yum.repos.d/cloudera*manager.repo*
      SLES
      sudo rm /etc/zypp/repos.d/cloudera*manager.repo*
      Ubuntu
      sudo rm /etc/apt/sources.list.d/cloudera*.list*
  3. Copy the cloudera_manager.repo file (or the cloudera-manager.list file for Ubuntu) that you created during the OS upgrade preparation phase to the exact same location.
  4. Reinstall Cloudera Manager Server, Daemon & Agent Packages
    On the Cloudera Manager Server host, reinstall the Cloudera packages that you removed.
    RHEL / CentOS
    sudo yum clean all
sudo yum install cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
    cloudera-manager-server-db-2
    SLES
    sudo zypper clean --all
sudo zypper install cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
    cloudera-manager-server-db-2
    Ubuntu
    sudo apt-get clean
    sudo apt-get update
    sudo apt-get install cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent
    cloudera-manager-server-db-2

  5. Verify that the configuration files (that were backed up) are intact. Correct if necessary.

  6. If you customized the /etc/cloudera-scm-agent/config.ini file, the system renames your custom file with a .rpmsave or .dpkg-old extension. Merge your customizations back into the config.ini file that the package manager installs.

  7. Start Cloudera Manager Server & local Agent

    On the host with Cloudera Manager Server, the appropriate services typically will start automatically on reboot. Otherwise, start the Cloudera Manager Server and the local Agent as necessary.

    1. Start the Cloudera Manager Server by running the following command:
      sudo systemctl start cloudera-scm-server
    2. Start the Cloudera Manager Agent.
      sudo systemctl start cloudera-scm-agent

      If the agent starts without errors, no response displays.

    3. Verify that the Cloudera Manager Agent downloaded a proper parcel for your new operating system. You can use the following command to check-in Cloudera Manager logs for downloaded parcels: 
      grep "Completed download" /var/log/cloudera-scm-agent/cloudera-scm-agent.log
      The download might take some time. Look for the operating system in the names of the downloaded parcels. The output of the command should look similar to the following:
      # grep "Completed download" /var/log/cloudera-scm-agent/cloudera-scm-agent.log
[02/May/2023 07:26:44 +0000] 20004 WorkerThread downloader   INFO     Completed download of http://nightly-7x-rf-1.nightly-7x-rf.root.hwx.site:7180/cmf/parcel/download/CDH-7.2.17-1.cdh7.2.17.p0.40465599-el7.parcel code=200 state=downloaded
[02/May/2023 07:31:00 +0000] 20004 WorkerThread downloader   INFO     Completed download of http://nightly-7x-rf-1.nightly-7x-rf.root.hwx.site:7180/cmf/parcel/download/FLINK-1.12-csa1.4.0.0-cdh7.1.6.0-297-13947709-el7.parcel code=200 state=downloaded

      Verify that the timestamps of the log lines are recent. Re-run the grep command again until the log lines appear. Verify the operating system name and major version in the parcel names and match the new operating system installed on the host.

      In the above output example, el7 refers to Redhat Enterprise Linux 7.

  8. Install and start Cloudera Manager Agent on remaining hosts

    1. Install and start the Cloudera Manager Agent package on all the remaining hosts of the cluster by running the following command:

      RHEL / CentOS
      sudo yum clean all
      sudo yum install cloudera-manager-daemons cloudera-manager-agent
      SLES
      sudo zypper clean –all
      sudo zypper install cloudera-manager-daemons cloudera-manager-agent
      Ubuntu
      sudo apt-get clean
      sudo apt-get update
      sudo apt-get install cloudera-manager-daemons cloudera-manager-agent
    2. Verify that the configuration files (that were backed up) are intact. Correct if necessary.

    3. If you customized the /etc/cloudera-scm-agent/config.ini file, your customized file is renamed with the extension .rpmsave or .dpkg-old. Merge any customizations into the /etc/cloudera-scm-agent/config.ini file that is installed by the package manager.

    4. Start the Cloudera Manager Agent
      sudo systemctl start cloudera-scm-agent
  9. Start Cloudera Management Service and Clusters
    1. Log in to Cloudera Manager as an Administrator.
    2. Go to Hosts > All Hosts page and verify that the status of all the hosts are in green.
    3. Go to Clusters > Parcels and verify that all needed parcels are in the Distributed and Activated state.
    4. Start the Cloudera Management Service as follows:
      1. On the Home > Status tab, click the options menu to the right of Cloudera Management Service and select Start.
    5. For each cluster on the home page, click the options menu to the right of each Cluster and select Start to start all the services in each of the cluster.