Cloudera Docs

Create a Kafka Topic to Store your Events

You must create a Kafka topic to store the events that you plan to stream. You can create a topic from the command line or the from the SMM UI.

Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages) across many machines.

Example events are payment transactions, geolocation updates from mobile phones, shipping orders, sensor measurements from IoT devices or medical equipment, and much more. These events are organized and stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder.

So before you can write your first events, you must create a topic.

  • You have installed the trial version of CDP Private Cloud Base and setup the streaming cluster.
  1. To create a Kafka topic from the SMM UI:
    1. Go to the Cloudera Manager UI, click the Streams Messaging Manager service, and select the Streams Messaging Manager Web UI.
    2. Click the Topics icon on the left bar.
      The Topics page appears.
    3. Click Add New.
    4. Provide the following information:
      • Topic name
      • Number of partitions
      • Level of availability
      • Cleanup policy. Cloudera recommends the delete option.
    5. SMM has automatically set Kafka topic configuration parameters. To manually adjust them, click Advanced.
    6. Click Save when done.
  2. To create a Kafka topic from the command line:
    1. Open a terminal session and run: 
      $ kafka-topics --create --topic quickstart-events --bootstrap-server <hostname>:9092
 --partitions 10 --replication-factor 3
    2. Run the kafka-topics.sh command without any arguments to display usage information.
    3. You can view details such as the partition count of the new topic: 
      $ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server <hostname>:9092

      Topic:quickstart-events  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: quickstart-events Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    For more Kafka command-line tools, see Kafka Command Line Tools.
Write a few events into the topic.