Upgrading Cloudera Manager 5 Using Tarballs
Minimum Required Role: Full Administrator
This topic describes how to upgrade Cloudera Manager 5.x using tarballs. Tarballs contain both the Cloudera Manager Server and Cloudera Manager Agent in a single file.
In most cases it is possible to upgrade Cloudera Manager without shutting down most CDH services, although you may need to stop some dependent services. CDH daemons can continue running, unaffected, while Cloudera Manager is upgraded. The upgrade process does not affect your CDH installation. After upgrading Cloudera Manager you may also want to upgrade CDH 4 clusters to CDH 5, or upgrade to a more recent minor version of CDH.
Step | Description | Link |
---|---|---|
1 | Collect the information you will need to upgrade Cloudera Manager. This includes user accounts, passwords, database URLs and other items. You must gather this information before beginning the upgrade because some of the information is only available from the Cloudera Manager Admin Console, which will not be available during the upgrade process. | Step 1: Collect Upgrade Information |
2 | Complete the pre-upgrade steps and review some special warnings about upgrades. | Step 2: Complete Pre-Upgrade Steps |
3 | If your Cloudera Manager hosts are using an unsupported version of the JDK, you must upgrade to a supported version of the JDK before upgrading Cloudera Manager. If you also plan to upgrade CDH, you must also upgrade the JDK on all cluster hosts. | Step 4: |
4 |
If the Cloudera Manager host does not have access to the internet, or you wish to install a version other than the latest version of Cloudera Manager, you configure access to the Cloudera Manager software, either from the Cloudera public repository or a local package repository that you create. |
Creating and Using a Package Repository for Cloudera Manager |
5 |
If you are upgrading from Cloudera Navigator 2.6 or lower, you must upgrade the Cloudera Navigator data management component. Cloudera Navigator 2.6 is supported on Cloudera Manager 5.7.x and higher. |
Upgrading the Component |
6 | Upgrade the Cloudera Manager server and agent software. | Upgrading the Cloudera Manager Server and Agents |
7 | Verify and test your upgrade. | Step 8: Verify and Test the Upgrade |
8 | Upgrade any required Cloudera Navigator components:
The Cloudera Navigator Data Management Component is upgraded automatically when you upgrade Cloudera Manager. |
Upgrading Cloudera Navigator Components |
9 | (Optional) Upgrade CDH. You are not required to upgrade CDH after upgrading Cloudera Manager. You can upgrade CDH at a later time. | Step 10: (Optional) Upgrade CDH |
Step 1: Collect Upgrade Information
- Host credentials. You must have SSH access and be able to log in using a root account or an account that has password-less sudo permission.
- The version of Cloudera Manager used in your cluster. Go to .
- The version of CDH. The CDH version number displays next to the cluster name on the Home page.
- The services enabled in your cluster. Go to .
- Whether the cluster was installed using parcels or packages. This information displays next to the CDH version on the Home page of Cloudera Manager.
- The version of the JDK deployed in the cluster. Go to .
- Whether or not you have enabled Cloudera Navigator auditing.
- For all databases used in your cluster:
- Type of database (PostgreSQL, Embedded PostgreSQL, MySQL, MariaDB, or Oracle)
- Hostnames of the databases
- Credentials for the databases
To locate database information:-
Sqoop, Oozie, and Hue – Go to
. - Hive Metastore – Go to the Hive service, select Configuration, and select the Hive Metastore Database category.
- Sentry – Go to the Sentry service, select Configuration, and select the Sentry Server Database category.
- Operating system version.
- Whether the cluster uses AES-256 encryption.
Step 2: Complete Pre-Upgrade Steps
Before beginning a Cloudera Manager upgrade, do the following:
- Review the system requirements for the new versions you are upgrading to.
See CDH 5 and Cloudera Manager 5 Requirements and Supported Versions.
- If you are upgrading from Cloudera Manager 5.4.x to Cloudera Manager 5.5 or higher, perform the following steps required for the associated Cloudera Navigator upgrade:
- Stop the Navigator Metadata Server role.
- Back up the Navigator Metadata Server storage directory.
- Make sure that the Navigator Metadata Server has sufficient memory to complete the upgrade.
- If you are using an Oracle database, in SQL*Plus, ensure that the following additional privileges are set:
GRANT EXECUTE ON sys.dbms_crypto TO nav; GRANT CREATE VIEW TO nav;
where nav is the user of the Navigator Audit Server database.
- Note the following:
- Cloudera Management Service TLS/SSL configuration
If you have enabled TLS security for the Cloudera Manager Admin Console, as of Cloudera Manager 5.1, Cloudera Management Service roles try to communicate with Cloudera Manager using TLS, and fail to start until TLS/SSL properties have been configured.
- Navigator
If you have enabled auditing with Cloudera Navigator, during the upgrade to Cloudera Manager 5, auditing is suspended and is only restarted when you restart the roles of audited services. You will be instructed to stop some services in a later step.
- Cloudera Management Service TLS/SSL configuration
- If you have previously installed Kafka 1.2, and are upgrading from Cloudera Manager 5.4 or lower, remove the Kafka CSD:
- Determine the location of the CSD directory:
- Select .
- Click the Custom Service Descriptors category.
- Retrieve the directory from the Local Descriptor Repository Path property.
- Delete the Kafka CSD from the directory.
- Determine the location of the CSD directory:
- Review package (RPM) dependencies. A Cloudera Manager upgrade may introduce new package dependencies. If your organization has restrictions or requires prior approval for installation of packages, see the list of Package Dependencies before upgrading Cloudera Manager.
Upgrading the Cloudera Manager Server and Agents
- If your cluster is running the embedded PostgreSQL database, stop all services that are using the embedded database. These may include:
- Hive service and all services such as Impala and Hue that use the Hive metastore
- Oozie
- Sentry
- Stop Cloudera Manager Server, Database, and Agent:
- Use the Cloudera Manager Admin Console to stop any running commands. These include commands a user runs and commands Cloudera Manager automatically triggers in response to a state change or a schedule. You can either wait for commands to complete or abort any running commands. For more information on viewing and aborting running commands, see Viewing Running and Recent Commands.
- On the host running the Cloudera Manager Server, stop the Cloudera Manager Server:
sudo service cloudera-scm-server stop
- On the host where the database runs, (usually the Cloudera Manager Server host) stop the database:
sudo service cloudera-scm-server-db stop
- If the Cloudera Manager host is also running the Cloudera Manager Agent, stop the Cloudera Manager Agent:
sudo service cloudera-scm-agent stop
- If you are using JDK 1.6, upgrade the JDK on Cloudera Manager Server JDK 1.7 or 1.8. After you install JDK 7, Cloudera Manager can install JDK 7 on all of the cluster hosts. To upgrade to JDK 1.8, you must manually install the JDK on all cluster hosts. Follow the steps in Java Development Kit Installation, and then return here to continue the upgrade.
- Download tarballs from the locations listed in Cloudera Manager Version and Download Information.
- Copy the tarballs and unpack them on all hosts on which you intend to install Cloudera Manager Server and Cloudera Manager Agents, in a directory of your choosing. If necessary, create
a new directory to accommodate the files you extract from the tarball. For instance, if /opt/cloudera-manager does not exist, create it using a command similar to:
$ sudo mkdir /opt/cloudera-manager
- Extract the contents of the tarball to this directory. For example, to copy a tar file to your home directory and extract the contents of all tar files to the /opt/ directory, use a command similar to the following:
$ sudo tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager
The files are extracted to a subdirectory named according to the Cloudera Manager version being extracted. For example, files could be extracted to /opt/cloudera-manager/cm-5.0/. This full path is needed later and is referred to as tarball_root directory.
- On every Cloudera Manager Agent host, configure the Cloudera Manager Agent to point to the Cloudera Manager Server by setting the following properties in the tarball_root/etc/cloudera-scm-agent/config.ini configuration file:
Property Description server_host Name of the host where Cloudera Manager Server is running. server_port Port on the host where Cloudera Manager Server is running. - By default, a tarball installation has a var subdirectory where state is stored. In a non-tarball installation, state is stored in /var. Cloudera recommends that you configure the tarball installation to use an external directory as the /var equivalent (/var or any other directory outside the tarball) so that when you upgrade Cloudera Manager, the new tarball installation can access this state. Configure the installation to use an external directory for storing state by editing tarball_root/etc/default/cloudera-scm-agent and setting the CMF_VAR variable to the location of the /var equivalent. If you do not reuse the state directory between different tarball installations, duplicate Cloudera Manager Agent entries can occur in the Cloudera Manager database.
- Start Cloudera Manager Server.
The way in which you start the Cloudera Manager Server varies depending on which account you want the Server to run under:
- As root:
sudo tarball_root/etc/init.d/cloudera-scm-server start
- As another user. If you run as another user, ensure the user you created for Cloudera Manager owns the location to which you extracted the tarball including the newly created database
files. If you followed the earlier examples and created the directory /opt/cloudera-manager and the user cloudera-scm, you could use the
following command to change ownership of the directory:
sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager
Once you have established ownership of directory locations, you can start Cloudera Manager Server using the user account you chose. For example, you might run the Cloudera Manager Server as cloudera-service. In this case, you have the following options:
- Run the following command:
$ sudo -u cloudera-service tarball_root/etc/init.d/cloudera-scm-server start
- Edit the configuration files so the script internally changes the user. Then run the script as root:
- Remove the following line from tarball_root/etc/default/cloudera-scm-server:
export CMF_SUDO_CMD=" "
- Change the user and group in tarball_root/etc/init.d/cloudera-scm-server to the
user you want the server to run as. For example, to run as cloudera-service, change the user and group as follows:
USER=cloudera-service GROUP=cloudera-service
- Run the server script as root:
$ sudo tarball_root/etc/init.d/cloudera-scm-server start
- Remove the following line from tarball_root/etc/default/cloudera-scm-server:
- Run the following command:
- To start the Cloudera Manager Server automatically after a reboot:
- Run the following commands on the Cloudera Manager Server host:
- RHEL-compatible and SLES
$ cp tarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server $ chkconfig cloudera-scm-server on
- Debian/Ubuntu
$ cp tarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server $ update-rc.d cloudera-scm-server defaults
- RHEL-compatible and SLES
- On the Cloudera Manager Server host, open the /etc/init.d/cloudera-scm-server file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} to tarball_root/etc/default.
- Run the following commands on the Cloudera Manager Server host:
- As root:
- To stop the Cloudera Manager Agent, run this command on each Agent host:
$ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed
If you are running single user mode, stop Cloudera Manager Agent using the user account you chose. For example, if you are running the Cloudera Manager Agent as cloudera-scm, you have the following options:- Run the following command:
$ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed
- Edit the configuration files so the script internally changes the user, and then run the script as root:
- Remove the following line from tarball_root/etc/default/cloudera-scm-agent:
export CMF_SUDO_CMD=" "
- Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the
user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:
USER=cloudera-scm GROUP=cloudera-scm
- Run the Agent script as root:
$ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed
- Remove the following line from tarball_root/etc/default/cloudera-scm-agent:
- Run the following command:
- Start the Cloudera Manager Agent according to the account you want the Agent to run under:
- To start the Cloudera Manager Agent, run this command on each Agent host:
$ sudo tarball_root/etc/init.d/cloudera-scm-agent start
When the Agent starts, it contacts the Cloudera Manager Server. - If you are running single user mode,
start Cloudera Manager Agent using the user account you chose. For example, to run the Cloudera Manager Agent as cloudera-scm, you have the following options:
- Run the following command:
$ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent start
- Edit the configuration files so the script internally changes the user, and then run the script as root:
- Remove the following line from tarball_root/etc/default/cloudera-scm-agent:
export CMF_SUDO_CMD=" "
- Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to
the user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:
USER=cloudera-scm GROUP=cloudera-scm
- Run the Agent script as root:
$ sudo tarball_root/etc/init.d/cloudera-scm-agent start
- Remove the following line from tarball_root/etc/default/cloudera-scm-agent:
- Run the following command:
- To start the Cloudera Manager Agents automatically after a reboot:
- Run the following commands on each Agent host:
- RHEL-compatible and SLES
$ cp tarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent $ chkconfig cloudera-scm-agent on
- Debian/Ubuntu
$ cp tarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent $ update-rc.d cloudera-scm-agent defaults
- RHEL-compatible and SLES
- On each Agent, open the tarball_root/etc/init.d/cloudera-scm-agent file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} to tarball_root/etc/default.
- Run the following commands on each Agent host:
- To start the Cloudera Manager Agent, run this command on each Agent host:
- Log in to the Cloudera Manager Admin Console.
- Restart all services:
- From the tab click next to the cluster name and select Restart.
- In the confirmation dialog box that displays, click Restart.
- Upgrade any required Cloudera Navigator components:
- (Optional) Upgrade CDH.
Step 8: Verify and Test the Upgrade
- In the Cloudera Manager Admin Console, click the Hosts tab.
- Click Host Inspector. On large clusters, the host inspector may take some time to finish running. You must wait for the process to complete before proceeding to the next step.
- Click Show Inspector Results. All results from the host inspector process are displayed, including the currently installed versions. If this includes listings of current component versions, the installation completed as expected.
- Verify that the monitoring features are working as expected; follow the instructions in Testing the Installation.
Step 10: (Optional) Upgrade CDH
For more information on upgrading CDH, see Upgrading CDH and Managed Services Using Cloudera Manager.