ZooKeeper Migration Steps
Collect the following information:
Required Information
Description
Source ZooKeeper hostname (sourceHostname)
The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the server.N properties can be used.
Destination ZooKeeper hostname (destinationHostname)
The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the server.N properties can be used.
Source ZooKeeper port (sourceClientPort)
This can be found in zookeeper.properties of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the clientPort property.
Destination ZooKeeper port (destinationClientPort)
This can be found in zookeeper.properties of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the clientPort property.
Export data path
Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.
Source ZooKeeper Authentication Information
This information is in <NiFi installation dir>/conf/state-management.xml. For NiFi 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the
-a,--authargument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the-k,--krb-confargument. + If the state-management.xml specifies Open, no authentication is required.Destination ZooKeeper Authentication Information
This information is in <NiFi installation dir>/conf/state-management.xml. For NiFi 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the
-a,--authargument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the-k,--krb-confargument. + If the state-management.xml specifies Open, no authentication is required.Root path to which NiFi writes data in Source ZooKeeper (sourceRootPath)
This information can be found in <NiFi installation dir>/conf/state-management.xml under the Root Node property in the cluster-provider element. (default: /nifi)
Root path to which NiFi writes data in Destination ZooKeeper (destinationRootPath)
This information can be found in <NiFi installation dir>/conf/state-management.xml under the Root Node property in the cluster-provider element.
Stop all processors in the NiFi flow. If you are migrating between two NiFi installations, the flows on both must be stopped.
Export the NiFi component data from the source ZooKeeper. The following command reads from the specified ZooKeeper running on the given hostname:port, using the provided path to the data, and authenticates with ZooKeeper using the given username and password. The data read from ZooKeeper is written to the file provided.
For NiFi 0.x
For an open ZooKeeper:
zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -f /path/to/export/zk-source-data.json
For a ZooKeeper using username:password for authentication:
zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -a <username:password> -f /path/to/export/zk-source-data.json
For NiFi 1.x
For an open ZooKeeper:
zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -f /path/to/export/zk-source-data.json
For a ZooKeeper using Kerberos for authentication:
zk-migrator.sh -r -z sourceHostname:sourceClientPort/sourceRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-source-data.json
(Optional) If you have used the new NiFi installation to do any processing, you can also export its ZooKeeper data as a backup prior to performing the migration.
For an open ZooKeeper:
zk-migrator.sh -r -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-destination-backup-data.json
For a ZooKeeper using Kerberos for authentication:
zk-migrator.sh -r -z destinationHostname:destinationClientPort/destinationRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-destination-backup-data.json
Migrate the ZooKeeper data to the destination ZooKeeper. If the source and destination ZooKeepers are the same, the
--ignore-sourceoption can be added to the following examples.For an open ZooKeeper:
zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-source-data.json
For a ZooKeeper using Kerberos for authentication:
zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -k /path/to/jaasconfig/jaas-config.conf -f /path/to/export/zk-source-data.json
Once the migration has completed successfully, start the processors in the NiFi flow. Processing should continue from the point at which it was stopped when the NiFi flow was stopped.