Dataflow management with schema-based routing

Learn how to use Schema Registry to support NiFi dataflow management.

If you are using NiFi to move different types of syslog events to downstream systems, you have data movement requirements where you need to parse the syslog event to extract the event type and route the event to a certain downstream system (for example, different Kafka topics) based on the event type.

Without Schema Registry, NiFi uses regular expressions or other utilities to parse the event type value from the payload and stores it into a flowfile attribute. Then NiFi uses routing processors (for example, RouteOnAttribute) to use the parsed value for routing decisions. If the structure of the data changes considerably, this type of extract and routing pattern is brittle and requires frequent changes.

With the introduction of Schema Registry, NiFi queries the registry for schema and then retrieves the value for a certain element in the schema. In this case, even if the structure changes, as long as compatibility policies are adhered to, NiFi's extract and routing rules do not change. This is another common use case for Schema Registry.