Execute the Flow
Install the elasticsearch head plugin by executing:
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
.Stop all other parser topologies via monit.
Create the
mad
Kafka topic by executing:/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic mad --partitions 1 --replication-factor 1
.Push the modified configs by executing:
$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i $METRON_HOME/config/zookeeper/
.Start the profiler by executing:
$METRON_HOME/bin/start_profiler_topology.sh
.Start the parser topology by executing:
$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s mad
.Ensure that the enrichment and indexing topologies are started. If not, then start those via monit or by hand.
Generate data into kafka by executing the following for at least 10 minutes:
~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad
. Note: If you chose to use the t-distribution script above, you would adjust the parameters of therand_gen.py
script accordingly.Stop the above with ctrl-c and send an obvious outlier into kafka:
echo "1000" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad
.
You should be able to find the outlier via the elasticsearch head plugin by searching
for the messages where is_alert
is true
.