Installation
Also available as:
PDF
loading table of contents...

Creating Kafka Topics

You need to create a Kafka topic for each supported data source.

  1. Assign the ZooKeeper host to the zookeeper variable

    zookeeper={{zookeeper_quorum}}
    
  2. From the kafka-bin directory (/usr/hdp/current/kafka-broker/bin), enter the following commands:

    Because you assigned the zookeeper variable in Step 1, you can copy and paste the following commands.

    #bro
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic bro 
    
    #yaf
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic yaf
    
    #snort
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic snort
    
    #Parser Invalid
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic parser_invalid
    
    #Parser Error
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic parser_error
    
    #enrichments
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic enrichments
    
    #indexing
    ./kafka-topics.sh \
    --zookeeper ${zookeeper} \
    --create --partitions 1 --replication-factor 1 \
       	--config retention.bytes=10737418240 \
    --topic indexing
    
    ./kafka-topics.sh --zookeeper ${zookeeper} --list
    

    The final command lists the Kafka topics you just created. You should see the following topics listed:

    bro
    enrichments
    indexing
    parser_error
    parser_invalid
    snort
    yaf