This is the documentation for Cloudera Manager 5.0.x. Documentation for other versions is available at Cloudera Documentation.

Upgrading Cloudera Manager 5 to the Latest Cloudera Manager

This process applies to upgrading all versions of Cloudera Manager 5.

In most cases it is possible to complete the following upgrade without shutting down most CDH services, although you may need to stop some dependent services. CDH daemons can continue running, unaffected, while Cloudera Manager is upgraded. The upgrade process does not affect your CDH installation. After upgrading Cloudera Manager you may also want to upgrade CDH 4 clusters to CDH 5.

Upgrading from a version of Cloudera Manager 5 to the latest version of Cloudera Manager involves the following broad steps.

  1. Review Warning
  2. Perform Prerequisite Steps
  3. Stop Selected Services
  4. Stop Cloudera Manager Server, Database, and Agent
  5. (Optional) Upgrade JDK on Cloudera Manager Server Host and Agent Hosts
  6. Upgrade Cloudera Manager Server Packages
  7. Start the Cloudera Manager Server
  8. Upgrade Cloudera Manager Agent Packages
  9. Verify the Upgrade Succeeded
  10. Start Selected Services
  11. Deploy Updated Client Configurations
  12. Test the Installation
  13. (Optional) Upgrade CDH

Review Warning

  Warning: If you have enabled auditing with Cloudera Navigator, during the process of upgrading Cloudera Manager 5 auditing is suspended and is only restarted when you restart the roles of audited services.

Perform Prerequisite Steps

Ensure that you have performed the following steps:

Stop Selected Services

If your cluster meets any of the conditions listed in the following table, you must stop the indicated services or roles.
Condition Procedure
Running a version of Cloudera Manager that has the Cloudera Management Service Stop the Cloudera Management Service.
Running Cloudera Navigator Stop any of the following roles whose service's Queue Policy configuration (navigator.batch.queue_policy) is set to SHUTDOWN:
  • HDFS - NameNode
  • HBase - Master and RegionServers
  • Hive - HiveServer2
  • Hue - Beeswax Server
Stopping these roles renders any service depending on these roles unavailable. For the HDFS - NameNode case this implies most of the services in the cluster will be unavailable until the upgrade is finished.

Stop Cloudera Manager Server, Database, and Agent

  1. On the host running the Cloudera Manager Server, stop the Cloudera Manager Server:
    $ sudo service cloudera-scm-server stop
  2. If you are using the embedded PostgreSQL database for Cloudera Manager, stop the database:
    $ sudo service cloudera-scm-server-db stop
      Important: If you are not running the embedded database service and you attempt to stop it, you will get a message to the effect that the service cannot be found. If instead you get a message that the shutdown failed, this means the embedded database is still running, probably due to services connected to the Hive Metastore. Do not proceed with the installation until you have stopped all your Metastore-dependent services and the database successfully shuts down (restart the Cloudera Manager server to shut down services as necessary). If you continue without solving this, your upgrade will fail and you will be left with a non-functional Cloudera Manager installation.
  3. If the Cloudera Manager host is also running the Cloudera Manager Agent, stop the Cloudera Manager Agent:
    $ sudo service cloudera-scm-agent stop

(Optional) Upgrade JDK on Cloudera Manager Server Host and Agent Hosts

If you are manually upgrading the Cloudera Manager Agent packages in Upgrade Cloudera Manager Agent Packages, and you plan to upgrade to CDH 5, ensure that JDK1.7u45 is installed on the Agent hosts following the instructions in Java Development Kit Installation.

If you are not running Cloudera Manager Server on the same host as a Cloudera Manager Agent, and you want all hosts to run the same JDK version, optionally install JDK1.7u45 on that host.

Upgrade Cloudera Manager Server Packages

  1. To upgrade the Cloudera Manager Server Packages, you can upgrade from Cloudera's repository at http://archive.cloudera.com/cm5/ or you can create your own repository, as described in Understanding Custom Installation Solutions. Creating your own repository is necessary if you are upgrading a cluster that does not have access to the Internet.
    1. Find the Cloudera repo file for your distribution by starting at http://archive.cloudera.com/cm5/ and navigating to the directory that matches your operating system.

      For example, for Red Hat or CentOS 6, you would navigate to http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/. Within that directory, find the repo file that contains information including the repository's base URL and GPG key. The contents of the cloudera-manager.repo file might appear as follows:

      [cloudera-manager]
      # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
      name=Cloudera Manager
      baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
      gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
      gpgcheck = 1
      For Ubuntu or Debian systems, the repo file can be found by navigating to the appropriate release directory, for example, http://archive.cloudera.com/cm4/debian/wheezy/amd64/cm. The repo file, in this case, cloudera.list, may appear as follows:
      # Packages for Cloudera Manager, Version 5, on Debian 7.0 x86_64
      deb http://archive.cloudera.com/cm5/debian/wheezy/amd64/cm wheezy-cm5 contrib
      deb-src http://archive.cloudera.com/cm5/debian/wheezy/amd64/cm wheezy-cm5 contrib
    2. Replace the repo file in the configuration location for the package management software for your system.
      Operating System Commands
      RHEL Copy cloudera-manager.repo to /etc/yum.repos.d/.
      SLES Copy cloudera-manager.repo to /etc/zypp/repos.d/.
      Ubuntu or Debian Copy cloudera.list to /etc/apt/sources.list.d/.
    3. Run the following commands:
      Operating System Commands
      RHEL
      $ sudo yum clean all
      $ sudo yum upgrade 'cloudera-*' 
        Note:
      • yum clean all cleans up yum's cache directories, ensuring that you download and install the latest versions of the packages
      • If your system is not up to date, and any underlying system components need to be upgraded before this yum update can succeed. yum will tell you what those are.
      SLES
      $ sudo zypper clean --all
      $ sudo zypper up -r http://archive.cloudera.com/cm5/sles/11/x86_64/cm/5/
      To download from your own repository:
      $ sudo zypper clean --all
      $ sudo zypper rr cm
      $ sudo zypper ar -t rpm-md http://myhost.example.com/<path_to_cm_repo>/cm
      $ sudo zypper up -r http://myhost.example.com/<path_to_cm_repo>
      Ubuntu or Debian Use the following commands to clean cached repository information and update Cloudera Manager components:
      $ sudo apt-get clean
      $ sudo apt-get update
      $ sudo apt-get dist-upgrade
      $ sudo apt-get install cloudera-manager-server cloudera-manager-agent cloudera-manager-daemons
      As this process proceeds, you may be prompted concerning your configuration file version:
      Configuration file `/etc/cloudera-scm-agent/config.ini'
      ==> Modified (by you or by a script) since installation.
      ==> Package distributor has shipped an updated version.
      What would you like to do about it ? Your options are:
      Y or I : install the package maintainer's version
      N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
      The default action is to keep your current version.

      You will receive a similar prompt for /etc/cloudera-scm-server/db.properties. Answer N to both these prompts. The config.ini file should be carefully inspected and the files merged together to ensure the new entries are incorporated.

At the end of this process you should have the following packages, corresponding to the version of Cloudera Manager you installed, on the host that will become the Cloudera Manager Server host.
OS Packages
RPM-based distributions
$ rpm -qa 'cloudera-manager-*'
cloudera-manager-agent-5.0.7-0.cm5.p0.932.el6.x86_64
cloudera-manager-server-5.0.7-0.cm5.p0.932.el6.x86_64
cloudera-manager-daemons-5.0.7-0.cm5.p0.932.el6.x86_64
Ubuntu or Debian
~# dpkg-query -l 'cloudera-manager-*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                   Version                Description
+++-======================-======================-============================================================
ii  cloudera-manager-agent 5.0.7-0.cm5.p0.932~sq The Cloudera Manager Agent
ii  cloudera-manager-daemo 5.0.7-0.cm5.p0.932~sq Provides daemons for monitoring Hadoop and related tools.
ii  cloudera-manager-serve 5.0.7-0.cm5.p0.932~sq The Cloudera Manager Server  
You may also see an entry for the cloudera-manager-server-db-2 if you are using the embedded database, and additional packages for plug-ins, depending on what was previously installed on the server host. If the cloudera-manager-server-db-2 package is installed, and you don't plan to use the embedded database, you can remove this package.

Start the Cloudera Manager Server

On the Cloudera Manager Server host (the system on which you installed the cloudera-manager-server package) do the following:
  1. If you are using the embedded PostgreSQL database for Cloudera Manager, start the database:
    $ sudo service cloudera-scm-server-db start
  2. Start the Cloudera Manager Server:
    $ sudo service cloudera-scm-server start
    You should see the following:
    Starting cloudera-scm-server:                              [  OK  ]
  Note: If you have problems starting the server, such as database permissions problems, you can use the server's log /var/log/cloudera-scm-server/cloudera-scm-server.log to troubleshoot the problem.

Upgrade Cloudera Manager Agent Packages

  Important: All hosts in the cluster must have access to the Internet if you plan to use archive.cloudera.com as the source for installation files. If you do not have Internet access, create a custom repository.
  1. Log in to the Cloudera Manager Admin Console.
  2. Upgrade hosts using one of the following methods:
    • Cloudera Manager installs Agent software
      1. Select Yes, I would like to upgrade the Cloudera Manager Agent packages now and click Continue.
      2. Select the release of the Cloudera Manager Agent to install. Normally, this will be the Matched Release for this Cloudera Manager Server. However, if you used a custom repository (that is, a repository other than archive.cloudera.com) for the Cloudera Manager server, select Custom Repository and provide the required information. The custom repository allows you to use an alternative location, but that location must contain the matched Agent version. Click Continue to proceed to the Configure Java Encryption screen.
      3. ​If local laws permit you to deploy unlimited strength encryption, and you want to run a secure cluster, check the Install Java Unlimited Strength Encryption Policy Files checkbox to install unlimited strength policy files for Java. This is required because during upgrade Cloudera Manager installs a copy of the Java 7 JDK, which does not include the unlimited strength policy files. Click Continue to proceed to the Upgrade Cloudera Manager Agent Packages screen.
      4. Specify credentials and initiate Agent installation:
        1. Select root or enter the user name for an account that has password-less sudo permission.
        2. Select an authentication method:
          • If you choose to use password authentication, enter and confirm the password.
          • If you choose to use public-key authentication provide a passphrase and path to the required key files.
        3. You can choose to specify an alternate SSH port. The default value is 22.
        4. You can specify the maximum number of host installations to run at once. The default value is 10.
      5. Click Continue. The Cloudera Manager Agent packages are installed.
    • Manually install Agent software. On all cluster hosts except the Cloudera Manager server host:
      1. Select No, I would like to skip the agent upgrade now and click Continue.
      2. Copy the appropriate repo file as described in step 3 of Upgrade Cloudera Manager Server Packages.
      3. Run the following commands:
        Operating System Commands
        RHEL
        $ sudo yum clean all
        $ sudo yum upgrade 'cloudera-*' 
          Note:
        • yum clean all cleans up yum's cache directories, ensuring that you download and install the latest versions of the packages
        • If your system is not up to date, and any underlying system components need to be upgraded before this yum update can succeed. yum will tell you what those are.
        SLES
        $ sudo zypper clean --all
        $ sudo zypper up -r http://archive.cloudera.com/cm5/sles/11/x86_64/cm/5/
        To download from your own repository:
        $ sudo zypper clean --all
        $ sudo zypper rr cm
        $ sudo zypper ar -t rpm-md http://myhost.example.com/<path_to_cm_repo>/cm
        $ sudo zypper up -r http://myhost.example.com/<path_to_cm_repo>
        Ubuntu or Debian Use the following commands to clean cached repository information and update Cloudera Manager components:
        $ sudo apt-get clean
        $ sudo apt-get update
        $ sudo apt-get dist-upgrade
        $ sudo apt-get install cloudera-manager-agent cloudera-manager-daemons
        As this process proceeds, you may be prompted concerning your configuration file version:
        Configuration file `/etc/cloudera-scm-agent/config.ini'
        ==> Modified (by you or by a script) since installation.
        ==> Package distributor has shipped an updated version.
        What would you like to do about it ? Your options are:
        Y or I : install the package maintainer's version
        N or O : keep your currently-installed version
        D : show the differences between the versions
        Z : start a shell to examine the situation
        The default action is to keep your current version.

        You will receive a similar prompt for /etc/cloudera-scm-server/db.properties. Answer N to both these prompts. The config.ini file should be carefully inspected and the files merged together to ensure the new entries are incorporated.

      4. If local laws permit you to deploy unlimited strength encryption, and you want to run a secure cluster, install the unlimited strength JCE policy files. This is required because during upgrade Cloudera Manager installs a copy of the Java 7 JDK, which does not include the unlimited strength policy files.
  3. Click Continue. The Host Inspector runs to inspect your managed hosts for correct versions and configurations. If there are problems, you can make changes and then re-run the inspector. When you are satisfied with the inspection results, click Continue and then click Finish.
  4. Review the configuration changes to be applied. Confirm the settings entered for file system paths. The file paths required vary based on the services to be installed. For example, you might confirm the NameNode Data Directory and the DataNode Data Directory for HDFS.
      Warning: DataNode data directories should not be placed on NAS devices.
    Click Continue. The wizard starts the services.
  5. Click Finish.

All services (except for the services you stopped in Stop Selected Services) should be running.

Verify the Upgrade Succeeded

If the commands to update and start the Cloudera Manager Server complete without errors, you can assume the upgrade has completed as desired. For additional assurance, you can check that the server versions have been updated.
  1. In the Cloudera Manager Admin Console, click the Hosts tab.
  2. Click Host Inspector. On large clusters, the host inspector may take some time to finish running. You must wait for the process to complete before proceeding to the next step.
  3. Click Show Inspector Results. All results from the host inspector process are displayed including the currently installed versions. If this includes listings of current component versions, the installation completed as expected.

Start Selected Services

Start services you shut down in Stop Selected Services:
  1. Do one of the following, depending on which services you shut down:
    • From the Home page click next to the cluster name and select Start.
    • From the Home page click next to the name of each service you shut down and select Start.
  2. In the confirmation dialog that displays, click Start.

Deploy Updated Client Configurations

The services whose client configurations require redeployment are indicated with icon on the Home page Status tab. To ensure clients have current information about resources, update client configuration:
  1. From the Home page click next to the cluster name and select Deploy Client Configuration.
  2. In the confirmation dialog that displays, click Deploy Client Configuration.

Test the Installation

When you have finished the upgrade to Cloudera Manager, you can test the installation to verify that the monitoring features are working as expected; follow instructions under Testing the Installation.

(Optional) Upgrade CDH

Cloudera Manager 5 can manage both CDH 4 and CDH 5, so upgrading existing CDH 4 installations is not required, but you may want to upgrade to the latest version. For more information on upgrading CDH, see Upgrading CDH and Managed Services.

Page generated September 3, 2015.