Introduction to Kafka Connect in SMM
Get started with Kafka Connect in SMM.
Kafka Connect is a tool for streaming data between Apache Kafka and other systems in a reliable and scalable fashion. Kafka Connect makes it simple to quickly define connectors that move large collections of data into and out of Kafka. Source connectors can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. Sink connectors can deliver data from Kafka topics into secondary storage and query systems or into batch systems for offline analysis.
Kafka Connect in CDP is shipped with many different Cloudera developed as well as publicly available sink and source connectors. Each of which cover a specific use case for streaming data. In addition to the connectors available by default, installing custom developed or third-party connectors is also possible. All connectors can be deployed, managed, and monitored using the Streams Messaging Manager UI (recommended), Streams Messaging Manager REST API, or Kafka Connect REST API.
For a comprehensive list of connectors shipped in CDP, see Connectors.
Default view of Kafka Connect in the SMM UI
Learn where the Kafka Connect option lies in the SMM UI and what are the different options that you can use while connecting Kafka with external systems.
Connect Cluster
page appears.
The Connect Cluster page shows the connector and cluster
details.
At the top-right corner of the Connect Cluster
page, you can see the name of the cluster.
The New Connector option below the cluster name enables you to create connectors in your cluster.
In the Connectors section, you can view the number of total connectors, running connectors, failed connectors, degraded connectors, and paused connectors.
In the Connectors tab, you can view details of the source connectors, topics, and sink connectors in the cluster. The Source Connectors and Sink Connectors sections show all, running, paused, and failed connectors with connector name and associated task details. Both Source Connectors and Sink Connectors sections contain a Search option that enables you to search for particular connector details. The Topics section shows the Kafka topic names where data is read from or written to. For more details on connectors, see the Monitoring Connectors section.
In the Cluster Profile tab, you can view details of the cluster and workers. For more details on cluster profile, see the Monitoring Cluster Profile section.
