Cloudera Docs

ZooKeeper Migration Steps

  1. Collect the following information:

    Required Information

    Description

    Source ZooKeeper hostname (sourceHostname)

    The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the server.N properties can be used.

    Destination ZooKeeper hostname (destinationHostname)

    The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the server.N properties can be used.

    Source ZooKeeper port (sourceClientPort)

    This can be found in <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the clientPort property or at the end of the server string.

    Destination ZooKeeper port (destinationClientPort)

    This can be found in <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the clientPort property or at the end of the server string.

    Export data path

    Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.

    Source ZooKeeper Authentication Information

    This information is in <NiFi installation dir>/conf/state-management.xml. For NiFi 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the -a,--auth argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the -k,--krb-conf argument.

    If the state-management.xml specifies Open, no authentication is required.

    Destination ZooKeeper Authentication Information

    This information is in <NiFi installation dir>/conf/state-management.xml. For NiFi 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the -a,--auth argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the -k,--krb-conf argument.

    If the state-management.xml specifies Open, no authentication is required.

    Root path to which NiFi writes data in Source ZooKeeper (sourceRootPath)

    This information can be found in <NiFi installation dir>/conf/state-management.xml under the Root Node property in the cluster-provider element. (default: /nifi)

    Root path to which NiFi writes data in Destination ZooKeeper (destinationRootPath)

    This information can be found in <NiFi installation dir>/conf/state-management.xml under the Root Node property in the cluster-provider element.

  2. Stop all processors in the NiFi flow. If you are migrating between two NiFi installations, the flows on both must be stopped.

  3. Export the NiFi component data from the source ZooKeeper. The following command reads from the specified ZooKeeper running on the given hostname:port, using the provided path to the data, and authenticates with ZooKeeper using the given username and password. The data read from ZooKeeper is written to the file provided.

    • For NiFi 0.x

      • For an open ZooKeeper:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -f /path/to/export/zk-source-data.json

      • For a ZooKeeper using username:password for authentication:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -a <username:password> -f /path/to/export/zk-source-data.json

    • For NiFi 1.x

      • For an open ZooKeeper:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -f /path/to/export/zk-source-data.json

      • For a ZooKeeper using Kerberos for authentication:

        • zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-source-data.json

  4. (Optional) If you have used the new NiFi installation to do any processing, you can also export its ZooKeeper data as a backup prior to performing the migration.

    • For an open ZooKeeper:

      • zk-migrator.sh -r -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-destination-backup-data.json

    • For a ZooKeeper using Kerberos for authentication:

      • zk-migrator.sh -r -z destinationHostname:destinationClientPort/destinationRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-destination-backup-data.json

  5. Migrate the ZooKeeper data to the destination ZooKeeper. If the source and destination ZooKeepers are the same, the --ignore-source option can be added to the following examples.

    • For an open ZooKeeper:

      • zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-source-data.json

    • For a ZooKeeper using Kerberos for authentication:

      • zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-source-data.json

  6. Once the migration has completed successfully, start the processors in the NiFi flow. Processing should continue from the point at which it was stopped when the NiFi flow was stopped.