Run the CSV Threat Intel Loader

After you define the threat intelligence source, threat intelligence extractor, and threat intelligence mapping configuration, you must run the loader to move the data from the threat intelligence source to the Metron threat intelligence store and to store the enrichment configuration in ZooKeeper.

  1. Log into $HOST_WITH_ENRICHMENT_TAG as root and navigate to $METRON_HOME/config.
  2. Use the loader to move the enrichment source to the enrichment store in ZooKeeper:
    $METRON_HOME/bin/ -n threatintel_config.json -i domainblocklist_ref.csv -t threatintel -c t -e threatintel_extractor_config.json
    This command modifies the Squid enrichment config in ZooKeeper to include the threat intel enrichment.
    The parameters for the utility are as follows:
    -b,--batchSize <SIZE>
    The batch size to use for HBase puts
    -c,--hbase)cf <CF>
    HBase column family to ingest the data into. This must be column family t.
    -e,--extractor_config <JSON_FILE>
    JSON Document describing the extractor for this input data source
    Generate Help screen
    -i,--input <FILE>
    The CSV File to load
    -l,--log4j <FILE>
    The log4j properties file to load
    -m,--import_mode <MODE>
    The Import mode to use: LOCAL,MR.Default: LOCAL
    -n,--enrichment_config <JSON_FILE>
    JSON Document describing the enrichment configuration details. This is used to associate an enrichment type with a field type in ZooKeeper.
    -p,--threads <NUM_THREADS>
    The number of threads to use when extracting data. The default is the number of cores of your machine.
    Do not update progress
    -t,--hbase_table <TABLE>
    HBase table to ingest the data into.
    The data is populated into an HBase table called enrichment.
  3. Verify that the logs are properly ingested to HBase:
    hbase shell
    scan 'threatintel'
    You should see a configuration for the sensor that looks something like the following:

  4. Generate some data to populate the Metron dashboard.