Mixed mode RHEL Operating System upgrade on ECS hosts

After installing CDP Private Cloud Data Services on a particular RHEL OS, you can now upgrade RHEL OS on some of your hosts to a new major version. For example, in a 10 node cluster, you can upgrade any number of hosts from RHEL 7.x to RHEL 9.x major version and keep the other hosts, running RHEL 7.x.

You must perform this task on all Embedded Container Service (ECS) hosts when you are ready for an OS upgrade.

Collect the following information:

ECS hosts in the cluster. For example: host-1, host-2.
Navigate to Cloudera Manager UI > ECS Cluster Name > HOSTS, to collect the host info.
Version of the ECS running on cluster. For example: 1.5.2
Navigate to Cloudera Manager UI > DATA SERVICES > Cluster Name, the cluster displays the Version at the bottom of the UI .
Version of the Operating System (OS) running on those hosts. For example: RHEL 7.9
- Login to all of the hosts in the ECS cluster by executing the following command:
```
cat /etc/redhat-release
```
Version of the upgraded OS. For example: RHEL 9.4
Verify the ECS version supported on the upgraded OS version here: https://supportmatrix.cloudera.com/

Shutdown of the ECS Cluster and prepare nodes for OS upgrade

Perform the following steps to shutdown the ECS cluster:

Log in to Cloudera Manager as an Administrator, then navigate to the ECS cluster.
Go to the Home > Status tab.
Click the Actions menu to the right of the Embedded Container Service (ECS) cluster name and select Stop.
Click the Stop option in the confirmation screen.
The Command Details window shows the progress of the services.

Perform the following steps ONLY on the hosts you are upgrading:

SSH into a ECS cluster host as a root user.
Verify the OS version by running the following command:
```
cat /etc/redhat-release
```
Stop Cloudera Manager agent on all the ECS hosts, by executing the following command:
```
systemctl stop cloudera-scm-agent
```
Uninstall Cloudera Manager agent packages on the ECS hosts by executing the following command:
```
yum remove cloudera-manager-agent
```

Major OS Upgrade

Follow the RHEL OS Upgrade documentation for major OS upgrade procedure.

Verify only the hosts you upgraded will show the new OS version on all ECS hosts by executing the following command:

cat /etc/redhat-release

Setup and install new Cloudera Manager agents

Perform the following steps on the hosts you upgraded in your cluster:

If you have not installed before, install python 3.8 and other dependencies on the ECS host.
Example: yum install python3.8 -y
Install the OS compatible Cloudera Manager agent packages.
Example: RHEL 9 package instead of RHEL 7 through commands on the ECS hosts.
- Change the baseurl= link in the cloudera-manager.repo to point to new RedHat version you upgraded to (from RedHat 7 to RedHat 9).
  Example: baseurl=https://archive.cloudera.com/p/cm7/7.11.3.0/redhat7/yum/
```
vim /etc/yum.repos.d/cloudera-manager.repo
```
  Save the cloudera-manager.repo file.
- To update repositories, execute:
```
yum clean all
```
- To verify the version is available, Example: 7.11.3.26-58725444.el9, execute the following command:
```
yum list available | grep -i cloudera-manager
```
- To install the Cloudera Manager Agent, execute the following command:
```
yum install cloudera-manager-agent
```
- To verify the correct version of the OS compatible Cloudera Manager agent installed, execute the following command:
```
 yum list installed | grep -i cloudera-manager-agent
```

Restore the Cloudera Manager agent config file, execute the following command:

cp /etc/cloudera-scm-agent/config.ini.rpmsave /etc/cloudera-scm-agent/config.ini)

Start the Cloudera Manager agent by executing the following command:
```
systemctl start cloudera-scm-agent
```
To verify the status of the Cloudera Manager agents started, execute the following command:
```
systemctl status cloudera-scm-agent
```
Log in to Cloudera Manager UI as an Administrator.
Verify that the Cloudera Manager displays all your hosts before starting ECS Cluster.
Navigate to Cloudera Manager UI > ECS Cluster Name > HOSTS, to collect the host info.
note
Verify all the hosts show green indicating good health.

Start the ECS Cluster

To start the ECS cluster, go to the Home > Status tab.
Click the Actions menu to the right of the Embedded Container Service cluster name and select Start.
Click the Start option.
The Command Details window shows the progress of the services.
Wait for all the pods to start. The wait time depends on the number of nodes in the cluster.
Navigate to ECS > WEB UI and try accessing: STORAGE UI, ECS WEB UI, and CONSOLE.

If you see a Vault sealed issue, after the start of the ECS service and if it does not unseal itself, then follow the step below to manually unseal it:

In the Cloudera Manager UI, Go to the ECS CLUSTER NAME > ECS SERVICE> ACTIONS > UNSEAL VAULT