Backing Up Cloudera Manager

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

The following steps create a complete backup of Cloudera Manager:

  1. Collect Information
  2. Back Up Cloudera Manager Agent
  3. Back Up the Cloudera Management Service
  4. Back Up Cloudera Navigator Data
  5. Stop Cloudera Manager Server & Cloudera Management Service
  6. Back Up the Databases
  7. Back Up Cloudera Manager Server
  8. Start Cloudera Manager Server & Cloudera Management Service

Collect Information for Backing Up Cloudera Manager

  1. Log in to the Cloudera Manager Server host. 
    ssh my_cloudera_manager_server_host
  2. Collect database information by running the following command: 
    cat /etc/cloudera-scm-server/db.properties
    For example: 
    ...
com.cloudera.cmf.db.type=...
com.cloudera.cmf.db.host=database_hostname:database_port
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=SOME_PASSWORD
  3. Collect information (host name, port number, database name, user name and password) for the following databases.
    • Reports Manager
    • Navigator Audit Server
    • Navigator Metadata Server
    • Activity Monitor

    You can find the database information by using the Cloudera Manager Admin Console. Go to Clusters > Cloudera Management Service > Configuration and select the Database category. You may need to contact your database administrator to obtain the passwords.

  4. Find the host where the Service Monitor, Host Monitor and Event Server roles are running. Go to Clusters > Cloudera Manager Management Service > Instances and note which hosts are running these roles.
  5. Identify the location of the Cloudera Navigator Metadata Server storage directory:
    1. Go to Clusters > Cloudera Management Service > Instances.
    2. Click the Configuration tab.
    3. Select Scope > Navigator Metadata Server.
    4. The Navigator Metadata Server Storage Dir property stores the location of the directory.
  6. Ensure that Navigator Metadata Server Java heap is large enough to complete the upgrade. You can estimate the amount of heap needed from the number of elements and relations stored in the Solr storage directory.
    1. Go to Clusters > Cloudera Management Service > Instances.
    2. In the list of instances, click Navigator Metadata Server.
    3. Select Log Files > Role Log File.
    4. Search the log file for solr core nav_elements and note the number of element documents.
    5. Search the log file for solr core nav_relations and note the number of relation documents.
    6. Multiply the total number of documents by 200 bytes per document and add to it a baseline of 2 GB: 
      ((num_nav_elements + num_nav_relations) * 200 bytes) + 2 GB
      For example, if you had 68813088 elements and 78813930 relations, the recommended Java heap size is ~30 GB: 
      ((68813088 + 78813930) * 200) + 2 GB = 29525403600 bytes = ~29.5 GB + 2 GB = ~ 31.5 GB
    7. Set the heap value in the Java Heap Size of Navigator Metadata Server in Bytes property in Clusters > Cloudera Management Service > Configuration.

Back Up Cloudera Manager Agent

Backup up the following Cloudera Manager agent files on all hosts:

  • Create a top level backup directory. 
    export CM_BACKUP_DIR="`date +%F`-CM"
echo $CM_BACKUP_DIR
mkdir -p $CM_BACKUP_DIR
  • Back up the Agent directory and the runtime state. 
    sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-agent.tar --exclude=*.sock /etc/cloudera-scm-agent /etc/default/cloudera-scm-agent /var/run/cloudera-scm-agent /var/lib/cloudera-scm-agent
  • Back up the existing repository directory.
    RHEL / CentOS
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.d
    SLES
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/zypp/repos.d
    Debian / Ubuntu
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/apt/sources.list.d

Back Up the Cloudera Management Service

  1. On the host where the Service Monitor role is configured to run, backup the following directory: 
    sudo cp -rp /var/lib/cloudera-service-monitor /var/lib/cloudera-service-monitor-`date +%F`-CM
  2. On the host where the Host Monitor role is configured to run, backup the following directory: 
    sudo cp -rp /var/lib/cloudera-host-monitor /var/lib/cloudera-host-monitor-`date +%F`-CM
  3. On the host where the Event Server role is configured to run, back up the following directory: 
    sudo cp -rp /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-eventserver-`date +%F`-CM

Back Up Cloudera Navigator Data

  2. Make sure a purge task has run recently to clear stale and deleted entities.
    • You can see when the last purge tasks were run in the Cloudera Navigator console (From the Cloudera Manager Admin console, go to Clusters > Cloudera Navigator. Select Administration > Purge Settings.)
    • If a purge hasn't run recently, run it by editing the Purge schedule on the same page.
    • Set the purge process options to clear out as much of the backlog of data as you can tolerate for your upgraded system. See Managing Metadata Storage with Purge.
  3. Stop the Navigator Metadata Server.
    1. Go to Clusters > Cloudera Management Service > Instances.
    2. Select Navigator Metadata Server.
    3. Click Actions for Selected > Stop.
  4. Back up the Cloudera Navigator Solr storage directory. 
    sudo cp -rp /var/lib/cloudera-scm-navigator /var/lib/cloudera-scm-navigator-`date +%F`-CM
  5. If you are using an Oracle database for audit, in SQL*Plus, ensure that the following additional privileges are set: 
      GRANT EXECUTE ON sys.dbms_crypto TO nav;
  GRANT CREATE VIEW TO nav;
    where nav is the user of the Navigator Audit Server database.

Preparing to upgrade Cloudera Navigator

Cloudera Navigator is upgraded as part of the Cloudera Manager upgrade process. There are no additional steps required. However, to ensure that Navigator metadata is efficiently upgraded and the upgraded version performs well, make sure to check that Navigator Metadata Server is configured with the correct amount of Java heap. For information on how to determine the correct heap size and to change the heap size if necessary, see Estimating Optimal Java Heap Size for Cloudera Navigator

Stop Cloudera Manager Server & Cloudera Management Service

  1. Stop the Cloudera Management Service.
    1. Log in to the Cloudera Manager Admin Console.
    2. Select Clusters > Cloudera Management Service.
    3. Select Actions > Stop.
  2. Log in to the Cloudera Manager Server host. 
    ssh my_cloudera_manager_server_host
  3. Stop the Cloudera Manager Server.
    RHEL 7, SLES 12, Debian 8, Ubuntu 16.04
    sudo systemctl stop cloudera-scm-server
    RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04, 14.04
    sudo service cloudera-scm-server stop

Back Up the Cloudera Manager Databases

  1. Back up the Cloudera Manager server database – Run the following command. (The command displayed below depends on the database you selected in the form at the top of this page. Replace placeholders with the actual values returned from the db.properties file):
    MySQL
    mysqldump --databases database_name --host=database_hostname --port=database_port -u user_name -p > $HOME/database_name-backup-`date +%F`-CM.sql
    PostgreSQL/Embedded
    pg_dump -h database_hostname -U user_name -W -p database_port database_name > $HOME/database_name-backup-`date +%F`-CM.sql
    Oracle
    Work with your database administrator to ensure databases are properly backed up.

    For more information about backing up databases, see Backing Up Databases.

  2. Back up All other Cloudera Manager databases - Use the database information that you collected in a previous step. You may need to contact your database administrator to obtain the passwords.
    These databases can include the following:
    • Reports Manager
    • Navigator Audit Server
    • Navigator Metadata Server
    • Activity Monitor (Only used for MapReduce 1 monitoring).

    Run the following commands to back up the databases. (The command displayed below depends on the database you selected in the form at the top of this page. Replace placeholders with the actual values.):

    MySQL
    mysqldump --databases database_name --host=database_hostname --port=database_port -u database_username -p > $HOME/database_name-backup-`date +%F`-CM.sql
    PostgreSQL/Embedded
    pg_dump -h database_hostname -U database_username -W -p database_port database_name > $HOME/database_name-backup-`date +%F`-CM.sql
    Oracle
    Work with your database administrator to ensure databases are properly backed up.

Back Up Cloudera Manager Server

  1. Log in to the Cloudera Manager Server host. 
    ssh my_cloudera_manager_server_host
  2. Create a top-level backup directory. 
    export CM_BACKUP_DIR="`date +%F`-CM"
echo $CM_BACKUP_DIR
mkdir -p $CM_BACKUP_DIR
  3. Back up the Cloudera Manager Server directories: 
    sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-server.tar /etc/cloudera-scm-server /etc/default/cloudera-scm-server
  4. Back up the existing repository directory.
    RHEL / CentOS
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.d
    SLES
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/zypp/repos.d
    Debian / Ubuntu
    sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/apt/sources.list.d

Start Cloudera Manager Server & Cloudera Management Service

  1. Log in to the Cloudera Manager Server host. 
    ssh my_cloudera_manager_server_host
  2. Start the Cloudera Manager Server.
    RHEL 7, SLES 12, Debian 8, Ubuntu 16.04
    sudo systemctl start cloudera-scm-server
    If the Cloudera Manager Server starts without errors, no response displays.
    RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04, 14.04
    sudo service cloudera-scm-server start
    You should see the following: 
    Starting cloudera-scm-server: [ OK ]
  3. Start the Cloudera Management Service.
    1. Log in to the Cloudera Manager Admin Console.
    2. Select Clusters > Cloudera Management Service.
    3. Select Actions > Start.