Run the Threat Intelligence Loader
Now that you have the threat intel source, threat intel exractor, and threat intel mapping config defined, you can run the loader to move the data from the threat intel source to the Metron threat intel Store and store the enrichment config in ZooKeeper.
Log into the $HOST_WITH_ENRICHMENT_TAG as root.
Run the loader:
/usr/metron/$METRON_RELEASE/bin/flatfile_loader.sh -n enrichment_config.json -i domainblocklist.csv -t threatintel -c t -e extractor_config_temp.json
The previous command adds the threat intel data into HBase and establishes a ZooKeeper mapping. The data is populated into an HBase table called threatintel.
To verify that the logs were properly ingested into HBase, run the following command:
hbase shell scan 'threatintel'
Now check if the ZooKeeper enrichment tag was properly populated:
/usr/metron/$METRON_RELEASE/bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER_HOST:2181
You should see a configuration for the Squid sensor something like the following
{ "index" : "squid", "batchSize" : 1, "enrichment" : { "fieldMap" : { "hbaseEnrichment" : [ "ip_src_addr" ] }, "fieldToTypeMap" : { "ip_src_addr" : [ "user" ] }, "config" : { } }, "threatIntel" : { "fieldMap" : { }, "fieldToTypeMap" : { }, "config" : { }, "triageConfig" : { "riskLevelRules" : { }, "aggregator" : "MAX", "aggregationConfig" : { } } }, "configuration" : { } }
Generate some data by using the Squid client to execute http requests.
squidclient http://www.actdhaka.com