Kafka Connect worker assignment and Stateless NiFi

Learn how NiFi Stateless connectors and tasks get assigned to Connect workers, in what cases you need to be aware of worker assignments, and how you can look up worker assignments.

When running Kafka Connect in a cluster environment, each cluster node in the Kafka Connect cluster has a Kafka Connect worker running on it. In CDP, when the cluster is managed by Cloudera Manager, Kafka Connect workers are represented by Kafka Connect role instances, which are deployed under your Kafka service. This means that the number of workers available in the cluster is identical to the number of Kafka Connect role instances.

When you deploy a connector, the connector is assigned to a worker. Additionally the task or tasks of the connector are also assigned to workers. The workers that the connector and tasks are assigned to might be the same, but might also be different. If the connector has multiple tasks (tasks.max>1), each task might be assigned to the same worker, or they might be assigned to different workers. In addition, if a Kafka Connect worker is stopped, the connectors and tasks that were assigned to it are moved to different workers in the Connect cluster. In general, you have no agency over worker assignment, it is decided by the Connect Framework, and it is not configurable. Additionally, worker assignments might change dynamically, however, this is a normal and expected behavior of Kafka Connect.

With Stateless NiFi connectors, knowing exact worker assignments is required when you are running a listen-type connector (for example, HTTP Source, Syslog TCP Source, or Syslog UDP Source). With these types of connectors you must ensure that you send messages to the host of the task's worker and not the connector's worker.

You can look up worker assignments in Streams Messaging Manager (SMM) on the Connector Profile page of the connector. The worker assigned to the connector is displayed in the Connector Profile section under ASSIGNED WORKER (1). The worker assigned to the task is displayed in the Tasks section under Worker ID (2).