Creating Kafka Topics
You need to create a Kafka topic for each supported data source.
Assign the ZooKeeper host to the zookeeper variable
zookeeper={{zookeeper_quorum}}
From the
kafka-bin
directory (/usr/hdp/current/kafka-broker/bin
), enter the following commands:Because you assigned the zookeeper variable in Step 1, you can copy and paste the following commands.
#bro ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic bro #yaf ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic yaf #snort ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic snort #Parser Invalid ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic parser_invalid #Parser Error ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic parser_error #enrichments ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic enrichments #indexing ./kafka-topics.sh \ --zookeeper ${zookeeper} \ --create --partitions 1 --replication-factor 1 \ --config retention.bytes=10737418240 \ --topic indexing ./kafka-topics.sh --zookeeper ${zookeeper} --list
The final command lists the Kafka topics you just created. You should see the following topics listed:
bro enrichments indexing parser_error parser_invalid snort yaf