Configuring flow.snapshot for Stateless NiFi connectors
Learn about the options you have in Streams Messaging Manager (SMM) when configuring the flow.snapshot Stateless NiFi connector property as well as the Save and Enhance feature that can be used to automatically import NiFi parameters into the connector configuration.
NiFi dataflows within Kafka Connect are deployed using the Stateless NiFi Source and Sink
connectors. When you deploy your custom developed dataflows with the Stateless NiFi
connectors, you must configure the
flow.snapshot property. This property
specifies the dataflow that is run within the connector.
Because the value of this property is the dataflow itself, the value you add to the property is not a simple string or integer. Instead, the value is the full contents of a flow definition JSON file, which depending on the dataflow, can be considerable in size.
As a result of this, the property is handled in a unique way by SMM. You can use multiple configuration methods to specify value. Additionally, you can choose to automatically enhance (amend) the configuration with the NiFi parameters that are available within the flow definition.
You can use the following configuration methods when specifying a value for
flow.snapshot. The following instructions assume that a flow definition
JSON file is available to you and that you are logged in to the Connect tab of the SMM
- Copy and paste the contents of the flow definition
- This method is similar to configuring any other connector property. The only
difference is that the value you specify is considerably longer than other property
- Open the flow definition in an editor.
- Copy the full contents of the file.
- Paste the contents of the file in the value field of the
- Upload the flow definition
- This method involves uploading the flow definition from your own machine. This method
is convenient if the flow definition JSON is available on the machine you are using to
access the SMM UI.
- Click next to
- Click Browse... in the modal window that appears.
- Select the flow definition file that you want to use.
- Click Save or Save and Enhance.
- Click next to
- Reference a file path
flow.snapshotproperty accepts file paths. This means that instead of copy-pasting or uploading the full contents of the flow definition JSON, you can also specify a location on disk where the file is available. Cloudera recommends that you use this method if the flow definition is larger than 1 MB.
- Deploy the flow definition file on all Kafka Connect hosts.
- Set the file path as the value of the
Using Save and Enhance
NiFi dataflows that you run within Kafka Connect are in the majority of cases parameterized. You can assign values to the parameters available within the dataflow on the SMM UI during connector configuration. This is done by adding each parameter, with specific prefixes, in the Properties section. However, this can take a considerable amount of time if done manually. This is especially true if the dataflow has many parameters and multiple Parameter Contexts.
In cases like this, you can choose to enhance the configuration by clicking the
Save and Enhance option after a value for the
flow.snapshot property is added. If this option is used, SMM parses the
dataflow specified in
flow.snapshot, extracts all parameters that are
available in the dataflow, and adds them to properties of the connector. The Save
and Enhance option is available on the modal window that appears after you
click next to the property.