Backing up before an upgrade

The Data Hub upgrade is an in-place upgrade and the data on attached volumes will be retained. The following section outlines data stored in the Data Hub clusters by different services. To prepare for a worst-case scenario where the cluster can't be restored, retain the data in these clusters before you upgrade.

Service configurations in Cloudera Manager

  • Any service configurations that you have updated since the cluster creation. Typically these can be found by selecting the “Non-default values” filter on the Configuration menu in CM.

Data Engineering clusters

The underlying data is in cloud storage. However, the following data is specific to each Data Hub cluster:

  • Oozie - workflow and coordinator definitions
  • Hue - saved queries
  • Hive UDFs and Spark applications
  • Oozie custom sharelibs

Streams Messaging clusters

Data stored on volumes:

  • Zookeeper: /hadoopfs/fs1/zookeeper (only on the first attached volume)
  • Kafka: /hadoopfs/fsN/kafka (for on all attached volumes, N stands for volume index)

Data stored in database:

  • Schema Registry schemas

Data stored in S3/ADLS:

  • Schema Registry serde jars

Flow Management clusters

Verify that you have committed all of your flows, then backup the following files/directories into your S3 backup location:

  • nifi.properties - on any NiFi node, run: ps -ef | grep nifi.properties

    This will indicate the path to the file. For example:

    -Dnifi.properties.file.path=/var/run/cloudera-scm-agent/process/1546335400-nifi-NIFI_NODE/nifi.properties

  • bootstrap.conf - on any NiFi node, run: ps -ef | grep bootstrap.conf

    This will indicate the path to the file. For example:

    -Dorg.apache.nifi.bootstrap.config.file=/var/run/cloudera-scm-agent/process/1546335400-nifi-NIFI_NODE/bootstrap.conf

  • Zookeeper data for /nifi znode. From a NiFi node:

    /opt/cloudera/parcels/CFM-<CFMversion>/TOOLKIT/bin/zk-migrator.sh -r -z <zk node>:2181/nifi
              -f /tmp/zk-data.json

    If a Zookeeper node is co-located with NiFi (light duty template, it’s possible to use localhost:2181), then back up the created zk-data.json file.

  • NiFi Registry data. On the Management node, locate the nifi-registry.properties file:

    ps -ef | grep nifi-registry.properties

    Then locate the providers.xml file from the nifi-registry.properties file:

    grep providers.xml
              /var/run/cloudera-scm-agent/process/1546335488-nifiregistry-NIFI_REGISTRY_SERVER/nifi-registry.properties

    In the providers.xml file, find the directory where flow versions are stored and backup this directory. For example:

    <flowPersistenceProvider>
    <class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class>
    <property name="Flow Storage Directory">./flow_storage</property>
    </flowPersistenceProvider>
  • /var/lib/nifi/working-dir (for the NiFi nodes) - without the ‘work’ directory inside it. This directory contains, for example, the local state directory as well as the flow.xml.gz which represents the flow definitions.
  • /var/lib/nifiregistry/working-dir (for the Management node)
  • Any other “custom” directory, for example directories where client configs and JDBC driver are located.

Data Discovery & Exploration clusters

Complete the following steps to backup DDE clusters:

  • Assign the datalake-admin-bucket policy to RANGER_AUDIT_ROLE
  • Create a subfolder for this Data Hub inside your S3 backup location.
  • Log in to Data Hub gateway node.
  • kinit with the proper user keytab if needed.
  • Follow this backup guide, with these exceptions:
    • Use solrctl --get-solrxmlto get the solr.xml file.

      Edit it by changing your backup repository solr.hdfs.home parameter (see image) to your backup location (for example: s3a://datalake-bucket/backup/solr).

      User solrctl --put-solrxml to publish your edited solr.xml.

      Restart Solr.

    • You do not need to launch the prepare-backupcommand; instead launch create-snapshot and export-snapshot for each collection.
    • For export-snapshot, specify your destination as s3a://[your backup location]/[your data hub subfolder] (for example: s3a://datalake-bucket/backup/solr).
    • Verify on the YARN history server that all of the executions succeeded. If not, troubleshoot and relaunch any export-snapshot commands that failed.

Operational Database clusters

Complete the following steps to backup Operational Database clusters:

  • Create a subfolder for this Data Hub inside of your S3 backup location.
  • Ensure snapshots are enabled on Cloudera Manager. They are enabled by default.
  • Log in to Data Hub gateway node.
  • kinit with the proper user keytab if needed.
  • Launch hbase shell
  • Issue the snapshotcommand for each of the tables that you want to back up (for example: snapshot ‘mytable’, ‘mytable-snapshot’)
  • Exit hbase shell
  • Launch ExportSnapshot command for each of the snapshots you created (for example:
    hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot mytable-snapshot -copy-to
            s3a://datalake-bucket/backup/hbase -mappers 10)
  • Verify on the YARN history server that all of the executions succeeded. If not, troubleshoot and relaunch any ExportSnapshot commands that failed.