Troubleshooting Installation and Upgrade Problems

There were active commands running before upgrade. This includes commands a user might have run and also for commands Cloudera Manager automatically triggers, either in response to a state change, or something that's on a schedule.

For information on known issues, see Known Issues and Workarounds in Cloudera Manager 5.

The KMS user is not part of the nfast group on the host(s) running the Navigator HSM KMS backed by Thales HSM role.

You might have SELinux enabled.

Disable SELinux by running sudo setenforce 0 on the Cloudera Manager Server host. To disable it permanently, edit /etc/selinux/config. For more information, see Disabling SELinux.

You need to do some manual cleanup.

See Uninstalling Cloudera Manager and Managed Software.

Make sure that the InnoDB engine is configured, not the MyISAM engine. To check what engine your tables are using, run the following command from the MySQL shell: mysql> show table status;

For more information, see Install and Configure MySQL for Cloudera Software.

You might have SELinux or iptables enabled.

Check /var/log/cloudera-scm-server/cloudera-scm-server.log on the Server host and /var/log/cloudera-scm-agent/cloudera-scm-agent.log on the Agent hosts. Disable SELinux and iptables. For more information, see Disabling SELinux and Disabling the Firewall.

You may have network connectivity problems.

Hostname mapping or permissions are not set up correctly.

• For hostname configuration, see Configuring Network Names.
• For permissions, make sure the values you enter into the wizard match those you used when you configured the databases. The value you enter into the wizard as the database hostname must match the value you entered for the hostname (if any) when you configured the database.

  For example, if you had entered the following when you created the database

  grant all on activity_monitor.* TO 'amon_user'@'myhost1.myco.com' IDENTIFIED BY 'amon_password';

  the value you enter here for the database hostname must be myhost1.myco.com. If you did not specify a host, or used a wildcard to allow access from any host, you can enter either the fully qualified domain name (FQDN), or localhost. For example, if you entered

  grant all on activity_monitor.* TO 'amon_user'@'%' IDENTIFIED BY 'amon_password';

  the value you enter for the database hostname can be either the FQDN or localhost.

MySQL binlog format problem.

Set binlog_format=mixed in /etc/my.cnf. For more information, see this MySQL bug report. See also Cloudera Manager and Managed Service Datastores.

You may have mismatched versions of the Cloudera Manager Server and Agents.

Make sure you have upgraded the Cloudera Manager Agents on all hosts. (The previous version of the Agents will heartbeat with the new version of the Server, but you cannot start HDFS and MapReduce with this combination.)

Java might not be installed or might be installed at a custom location.

See Configuring a Custom Java Home Location for more information on resolving this issue.

The MySQL thread stack is too small.

1. Update the thread_stack value in my.cnf to 256KB. The my.cnf file is normally located in /etc or /etc/mysql.
2. Restart the mysql service: $ sudo service mysql restart
3. Restart Activity Monitor.

The binlog_format is not set to mixed.

Modify the mysql.cnf file to include the entry for binlog format as specified in Install and Configure MySQL for Cloudera Software.

It is possible to install, uninstall, and reinstall CDH and Cloudera Manager. In certain cases, this does not complete as expected. If you install Cloudera Manager 5 and CDH 5, then uninstall Cloudera Manager and CDH, and then attempt to install CDH 4 and Cloudera Manager 4, incorrect cached information may result in the installation of an incompatible version of the Oracle JDK.

Clear information in the yum cache:

1. Connect to the CDH host.
2. Execute either of the following commands:

   $ yum --enablerepo='*'clean
             all

   or

   $ rm -rf
               /var/cache/yum/cloudera*

3. After clearing the cache, proceed with installation.

PostgreSQL versions 9 and higher require special configuration for Hive because of a backward-incompatible change in the default value of the standard_conforming_strings property. Versions up to PostgreSQL 9.0 defaulted to off, but starting with version 9.0 the default is on.

HDFS caching, which is enabled by default in CDH 5, requires new memlock functionality from Cloudera Manager Agents.

Do the following:

1. Stop all CDH and managed services.
2. On all hosts with Cloudera Manager Agents, hard-restart the Agents. Before performing this step, ensure you understand the semantics of the hard_restart command by reading Hard Stopping and Restarting Agents.
   • Packages

     RHEL 7, SLES 12, Debian 8, Ubuntu 16.04

     sudo /etc/init.d/cloudera-scm-agent next_stop_hard
     sudo systemctl restart cloudera-scm-agent

     RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04, 14.04

     sudo service cloudera-scm-agent hard_restart

   • Tarballs
     • To stop the Cloudera Manager Agent, run this command on each Agent host:
       • RHEL-compatible 7 and higher:

         $ sudo tarball_root/etc/init.d/cloudera-scm-agent next_stop_hard
         $ sudo tarball_root/etc/init.d/cloudera-scm-agent restart

       • All other Linux distributions:

         $ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_restart

     • If you are running single user mode, start Cloudera Manager Agent using the user account you chose. For example, to run the Cloudera Manager Agent as cloudera-scm, you have the following options:
       • Run the following command:
         • RHEL-compatible 7 and higher:

           $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent next_stop_hard
           $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent restart

         • All other Linux distributions:

           $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent hard_restart 

       • Edit the configuration files so the script internally changes the user, and then run the script as root:
         1. Remove the following line from tarball_root/etc/default/cloudera-scm-agent:

            export CMF_SUDO_CMD=" "

         2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:

            USER=cloudera-scm
            GROUP=cloudera-scm

         3. Run the Agent script as root:
            • RHEL-compatible 7 and higher:

              $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent next_stop_hard
              $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent restart

            • All other Linux distributions:

              $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent hard_restart 

3. Start all services.

You upgraded CDH to 5.2 using Cloudera Manager and did not run the HDFS Metadata Upgrade command.

Stop the HDFS service in Cloudera Manager and follow the steps for upgrade (depending on whether you are using packages or parcels) described in Upgrading CDH and Managed Services Using Cloudera Manager.

You have not granted execute permission to sys.dbms_crypto.

Run GRANT EXECUTE ON sys.dbms_crypto TO nav;, where nav is the user of the Navigator Audit Server database.