Database Considerations for Cloudera Manager Upgrades
Cloudera Manager uses databases to store information about system configurations and tasks. Before upgrading, complete the pre-upgrade database tasks that apply in your environment.
Cloudera Manager 4.5 added support for Hive, which includes the Hive Metastore Server role type. This role manages the Metastore process when Hive is configured with a remote Metastore.
When upgrading from Cloudera Manager prior to 4.5, Cloudera Manager automatically creates new Hive service(s) to capture the previous implicit Hive dependency from Hue and Impala. Your previous services will continue to function without impact. If Hue was using a Hive Metastore backed by a Derby database, then the newly created Hive Metastore Server will also use Derby. Since Derby does not allow concurrent connections, Hue will continue to work, but the new Hive Metastore Server will fail to run. The failure is harmless (because nothing uses this new Hive Metastore Server at this point) and intentional, to preserve the set of cluster functionality as it was before upgrade. Cloudera discourages the use of a Derby backed Hive Metastore due to its limitations. You should consider switching to a different supported database.
After you have completed these steps, the upgrade processes automatically complete any additional updates to database schema and service data stored. You do not need to complete any data migration.
Back up Databases
Before beginning the upgrade process, shut down the services that are using databases. This includes the Cloudera Manager Management Service roles, the Hive Metastore server, and Cloudera Navigator, if it is in use. Cloudera strongly recommends that you then back up all databases, however backing up the Activity Monitor database is optional. This is especially important if you are upgrading from Cloudera Manager 4 to Cloudera Manager 5. For information on backing up databases see Backing up Databases.
If any additional database will be required as a result of the upgrade, complete any required preparatory work to install and configure those databases. For example, if you are upgrading from Cloudera Manager Free Edition, Cloudera Manager 5 with Cloudera Express requires a database for the Activity Monitor. The upgrade instructions assume all required databases have been prepared. For more information on using databases, see Cloudera Manager and Managed Service Databases.
Modify Databases to Support UTF-8
Cloudera Manager 4.0 adds support for UTF-8 character sets. Update any existing databases in your environment that are not configured to support UTF-8.
Modifying MySQL to Support UTF-8
To modify a MySQL database to support UTF-8, the default character set must be changed and then you must restart the mysql service. Use the following commands to complete these tasks:
mysql> alter database default character set utf8; mysql> quit $ sudo service mysql restart
Modifying PostgreSQL to Support UTF-8
There is no single command available to modify an existing PostgreSQL database to support UTF-8. As a result, you must complete the following process:
- Use pg_dump to export the database to a file. This creates a backup of the database that you will import into a new, empty database that supports UTF-8.
- Drop the existing database. This deletes the existing database.
- Create a new database that supports Unicode encoding and that has the same name as the old database. Use a command of the following form, replacing the database name and user name with values that match your environment:
CREATE DATABASE scm_database WITH OWNER scm_user ENCODING 'UTF8'
- Review the contents of the exported database for non-standard characters. If you find unexpected characters, modify these so the database backup file contains the expected data.
- Import the database backup to the newly created database.
Modifying Oracle to Support UTF-8
Work with your Oracle database administrator to ensure any Oracle databases support UTF-8.
Modify Databases to Support Appropriate Maximum Connections
Check existing databases configurations to ensure the proper maximum number of connections is supported. Update the maximum configuration values, as required.
Modify the Maximum Number of MySQL Connections
Allow 100 maximum connections for each database and then add 50 extra connections. For example, for two databases set the maximum connections to 250. If you store five databases on one host (the databases for Cloudera Manager Server, Activity Monitor, Reports Manager, Cloudera Navigator, and Hive Metastore), set the maximum connections to 550.
Modify the Maximum Number of PostgreSQL Connections
Update the max_connection parameter in the /etc/postgresql.conf file.
You may have to increase the system resources available to PostgreSQL, as described at http://www.postgresql.org/docs/9.1/static/kernel-resources.html.
Modify the Maximum Number of Oracle Connections
Work with your Oracle database administrator to ensure appropriate values are applied for your Oracle database settings. You must determine the number of connections, transactions, and sessions to be allowed.
Allow 100 maximum connections for each database and then add 50 extra connections. For example, for two databases set the maximum connections to 250. If you store five databases on one host (the databases for Cloudera Manager Server, Activity Monitor, Reports Manager, Cloudera Navigator, and Hive Metastore), set the maximum connections to 550.
From the maximum number of connections, you can determine the number of anticipated sessions using the following formula:
sessions = (1.1 * maximum_connections) + 5
For example, if a host has two databases, you anticipate 250 maximum connections. If you anticipate a maximum of 250 connections, plan for 280 sessions.
Once you know the number of sessions, you can determine the number of anticipated transactions using the following formula:
transactions = 1.1 * sessions
Continuing with the previous example, if you anticipate 280 sessions, you can plan for 308 transactions.
Work with your Oracle database administrator to apply these derived values to your system.
Using the sample values above, Oracle attributes would be set as follows:
alter system set processes=250; alter system set transactions=308; alter system set sessions=280;
Next Steps
After you have completed any required database preparatory tasks, continue to Upgrading Cloudera Manager 4 to Cloudera Manager 5.
<< Upgrading Cloudera Manager | Upgrading Cloudera Manager 5 to the Latest Cloudera Manager >> | |