Using Stellar Data to Transform Threat Intelligence Data

You can transform and filter threat intelligence data as it is loaded into HBase by using Stellar extractor properties in the extractor configuration file. This task is optional.

1. Transform and filter the threat intelligence data as it is loaded into HBase by using the following Stellar extractor properties in the extractor_config.json file at $METRON_HOME/config:

   Extractor Property	Description	Example
   value_transform	Transforms fields defined in the columns mapping with Stellar transformations. New keys introduced in the transform are added to the key metadata.	"value_transform" : { "domain" : "DOMAIN_REMOVE_TLD(domain)"
   value_filter	Allows additional filtering with Stellar predicates based on results from the value transformations. In the following example, records whose domain property is empty after removing the TLD are omitted.	"value_filter" : "LENGTH(domain) > 0", "indicator_column" : "domain",
   indicator_transform	Transforms the indicator column independent of the value transformations. You can refer to the original indicator value by using indicator as the variable name, as shown in the following example. In addition, if you prefer to piggyback your transformations, you can refer to the variable domain, which allows your indicator transforms to inherit transformations done to this value during the value transformations.	"indicator_transform" : { "indicator" : "DOMAIN_REMOVE_TLD(indicator)"
   indicator_filter	Allows additional filtering with Stellar predicates based on results from the value transformations. In the following example, records whose indicator value is empty after removing the TLD are omitted.	"indicator_filter" : "LENGTH(indicator) > 0", "type" : "top_domains",

   The following example uses a CSV list of top domains as an enrichment and filters the value metadata, as well as the indicator column, with each of the supported Stellar expressions:

   {
    "config" : {
    "zk_quorum" : "$ZOOKEEPER_HOST:2181",
    "columns" : {
    "rank" : 0,
    "domain" : 1
    },
    "value_transform" : {
    "domain" : "DOMAIN_REMOVE_TLD(domain)"
    },
    "value_filter" : "LENGTH(domain) > 0",
    "indicator_column" : "domain",
    "indicator_transform" : {
    "indicator" : "DOMAIN_REMOVE_TLD(indicator)"
    },
    "indicator_filter" : "LENGTH(indicator) > 0",
    "type" : "top_domains",
    "separator" : ","
    },
    "extractor" : "CSV"
    }

   Running a file import with the above data and extractor configuration will result in the following two extracted data records:

   Indicator	Type	Value
   google	top_domains	{ "rank" : "1", "domain" : "google" }
   yahoo	top_domains	{ "rank" : "2", "domain" : "yahoo" }

   	Note
   	The extractor_config.json file is not stored anywhere by the loader. This file is used once by the bulk loader to parse the enrichment dataset. If you would like to keep a copy of this file, be sure to save a copy to another location.

2. To access properties that reside in the global configuration file, provide a ZooKeeper quorum via the zk_quorum property. If the global configuration looks like "global_property" : "metron-ftw", enter the following to expand the value_transform:

   "value_transform" : {
       "domain" : "DOMAIN_REMOVE_TLD(domain)",
        "a-new-prop" : "global_property"
    },

   The resulting value data will look like the following:

   Indicator	Type	Value
   google	top_domains	{ "rank" : "1", "domain" : "google", "a-new-prop" : "metron-ftw" }
   yahoo	top_domains	{ "rank" : "2", "domain" : "yahoo", "a-new-prop" : "metron-ftw" }

3. Remove any non-ASCII characters:

   iconv -c -f utf-8 -t ascii extractor_config_temp.json -o extractor_config.json