Execute the Flow
When you have finished configuring the MAD functionality, you will need to execute the flow.
-
Install the elasticsearch head plugin by executing:
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
. - Stop all other parser topologies via monit.
-
Create the
mad
Kafka topic by executing:/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic mad --partitions 1 --replication-factor 1
. -
Push the modified configs by executing:
$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i $METRON_HOME/config/zookeeper/
. -
Start the profiler by executing:
$METRON_HOME/bin/start_profiler_topology.sh
. -
Start the parser topology by executing:
$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s mad
. - Ensure that the enrichment and indexing topologies are started. If not, then start those via monit or by hand.
-
Generate data into kafka by executing the following for at least 10 minutes:
~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad
. Note: If you chose to use the t-distribution script above, you would adjust the parameters of therand_gen.py
script accordingly. -
Stop the above with ctrl-c and send an obvious outlier into kafka:
echo "1000" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad
.You should be able to find the outlier via the elasticsearch head plugin by searching for the messages whereis_alert
istrue
.