Kafka Connect log files and Stateless NiFi

Learn how and where Stateless NiFi connectors produce their logs.

Each cluster host that runs a Kafka Connect worker (Kafka Connect role) has a log file specific to Kafka Connect. The logs are stored by default in /var/log/kafka. The Connect related logs are named kafka-connect-[***HOSTNAME***].log. The log directory path is specified by the Kafka Connect Log Directory Kafka Connect role property. This property is configured in Cloudera Manager.

Every log entry produced by a NiFi dataflow that is running in a Kafka Connect connector is directed to the Kafka Connect log file. Specifically, the dataflow log entries are written to the log file located on the cluster host that is running the Kafka Connect worker that is executing the connector’s task.

For example, if your connector has a single task, which is executed by worker-1 running on host-1, then the log for that connector will be located on host-1. If a connector has multiple tasks, each running on a different worker, then each task’s dataflow writes logs to the respective worker’s log file.

If a Kafka Connect worker is stopped, its tasks are moved to other workers. As a result, it is possible that a specific connector's log entries first appear in the log file located on one host, but later on appear in another log file located on a different host.