Bulk Loading Threat Intelligence Sources Using STIX/TAXII
Hortonworks Cybersecurity Platform (HCP) is designed to work with STIX/TAXII threat feeds. OpenTAXII is a Python implementation of TAXII services that delivers a rich feature set and friendly pythonic API.
TAXII loader is a
stand-alone Java application that never stops streaming cyber threat intelligence
information. You can use the shell script
$METRON_HOME/bin/threatintel_taxii_load.sh
to poll a Taxii server
for STIX documents and ingest them into HBase.
It is common for the Taxii server to be an aggregation server such as Soltra Edge.
To set up an opentaxii service on HCP, you must deploy an opentaxii role, create an extractor file and a connection file, and push the feeds to HBase. In addition to the Enrichment and Extractor configs described in the following sections, this loader requires a configuration file describing the connection information to the Taxii server. The following is an example of a configuration file:
{
"endpoint" : "http://localhost:8282/taxii-discovery-service"
,"username" : "guest"
,"password" : "guest"
,"type" : "DISCOVER"
,"collection" : "guest.Abuse_ch"
,"table" : "threat_intel"
,"columnFamily" : "cf"
,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}
where:
- endpoint
-
The URL of the endpoint
- type
-
POLL
orDISCOVER
depending on the endpoint. - collection
-
The Taxii collection to ingest
- table
-
The HBase table to import into
- columnFamily
-
The column family to import into
- allowedIndicatorTypes
-
an array of acceptable threat intelligence types (see the "Enrichment Type Name" column of the Stix table above for the possibilities).
The parameters for the utility are as follows:
Short Code | Long Code | Is Required? | Description |
---|---|---|---|
-h | No | Generate the help screen/set of options | |
-e | --extractor_config | Yes | JSON Document describing the extractor for this input data source |
-c | --taxii_connection_config | Yes | The JSON config file to configure the connection |
-p | --time_between_polls | No | The time between polling the Taxii server in milliseconds. (default: 1 hour) |
-b | --begin_time | No | Start time to poll the Taxii server (all data from that point will be gathered in the first pull). The format for the date is yyyy-MM-dd HH:mm:ss |
-l | --log4j | No | The Log4j Properties to load |
-n | --enrichment_config | No | The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. |