Execute the Flow

Install the elasticsearch head plugin by executing:

/usr/share/elasticsearch/bin/plugin install
                        mobz/elasticsearch-head

.

Stop all other parser topologies via monit.

Create the mad Kafka topic by executing:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
                        node1:2181 --create --topic mad --partitions 1 --replication-factor
                        1

.

Push the modified configs by executing:

$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i
                        $METRON_HOME/config/zookeeper/

.

Start the profiler by executing: $METRON_HOME/bin/start_profiler_topology.sh.

Start the parser topology by executing:

$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z
                        node1:2181 -s mad

.

Ensure that the enrichment and indexing topologies are started. If not, then start those via monit or by hand.

Generate data into kafka by executing the following for at least 10 minutes:

~/rand_gen.py 0 1 1 |
                        /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
                        node1:6667 --topic mad

. Note: If you chose to use the t-distribution script above, you would adjust the parameters of the rand_gen.py script accordingly.

Stop the above with ctrl-c and send an obvious outlier into kafka:

echo "1000" |
                        /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
                        node1:6667 --topic mad

.

You should be able to find the outlier via the elasticsearch head plugin by searching for the messages where is_alert is true.