Bulk Loading Threat Intelligence Sources Using STIX/TAXII

Cloudera Cybersecurity Platform (CCP) is designed to work with STIX/TAXII threat feeds. OpenTAXII is a Python implementation of TAXII services that delivers a rich feature set and friendly pythonic API.

TAXII loader is a stand-alone Java application that never stops streaming cyber threat intelligence information. You can use the shell script $METRON_HOME/bin/threatintel_taxii_load.sh to poll a Taxii server for STIX documents and ingest them into HBase.

It is common for the Taxii server to be an aggregation server such as Soltra Edge.

To set up an opentaxii service on CCP, you must deploy an opentaxii role, create an extractor file and a connection file, and push the feeds to HBase. In addition to the Enrichment and Extractor configs described in the following sections, this loader requires a configuration file describing the connection information to the Taxii server. The following is an example of a configuration file:

                        {
   "endpoint" : "http://localhost:8282/taxii-discovery-service"
  ,"username" : "guest"
  ,"password" : "guest"
  ,"type" : "DISCOVER"
  ,"collection" : "guest.Abuse_ch"
  ,"table" : "threat_intel"
  ,"columnFamily" : "cf"
  ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}

                     

where:

endpoint

The URL of the endpoint

type

POLL or DISCOVER depending on the endpoint.

collection

The Taxii collection to ingest

table

The HBase table to import into

columnFamily

The column family to import into

allowedIndicatorTypes

an array of acceptable threat intelligence types (see the "Enrichment Type Name" column of the Stix table above for the possibilities).

The parameters for the utility are as follows:

Short Code Long Code Is Required? Description
-h No Generate the help screen/set of options
-e --extractor_config Yes JSON Document describing the extractor for this input data source
-c --taxii_connection_config Yes The JSON config file to configure the connection
-p --time_between_polls No The time between polling the Taxii server in milliseconds. (default: 1 hour)
-b --begin_time No Start time to poll the Taxii server (all data from that point will be gathered in the first pull). The format for the date is yyyy-MM-dd HH:mm:ss
-l --log4j No The Log4j Properties to load
-n --enrichment_config No The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified.