Create Global Configurations
The global configuration file is accessible to all configurable components in the system. The global configuration file can be used to assign a property to multiple parser topologies. For example, every message from every sensor is validated against global configuration rules. The global configuration file can also be used to assign properties to enrichments and the profiler which each use a single topology. For example, you can use the global configuration to configure the enrichment toplogy's writer batching settings.
- To configure a global configuration file, create a file called global.json at $METRON_HOME/config/zookeeper.
- Using the following format, populate the file with enrichment values that you want to apply to
all sensors:
Solr Indexwil
{ "es.clustername": "metron", "es.ip": "node1", "es.port": "9300", "es.date.format": "yyyy.MM.dd.HH", "fieldValidations" : [ { "input" : [ "ip_src_addr", "ip_dst_addr" ], "validation" : "IP", "config" : { "type" : "IPV4" } } ] }
- source.type.field
- The name used in the real-time store. Defaults to
source:type
. - threat.triage.score.field
- The threat triage score field name used in the real-time store. Defaults to
threat:triage:score
. - solr.zookeeper
- The ZooKeeper quorum associated with the SolrCloud instance. This is a required field with no default.
- solr.commitPerBatch
-
This is a boolean which defines whether the writer commits every batch. The default is
true
.NoteIf you set this tofalse
, then commits will happen based on the SolrClient's internal mechanism and worker failure might result in data being acknowledged in Storm but not written in Solr. - solr.commit.soft
-
This is a boolean that defines whether the writer makes a soft commit or a durable commit. See Solr Ref Guide 6.6 - AutoCommits for more information. The default is
false
.NoteIf you set the value totrue
, then commits will happen based on the SolrClient's internal mechanism and worker failure might result in data being acknowledged in Storm but not written in Solr. - solr.commit.waitSearcher
-
This is a boolean which defines whether the writer blocks the commit until the data is available to search. See Solr Ref Guide 6.6 - AutoCommits for more information. The default is
true
.NoteIf you set the value tofalse
, then commits will happen based on the SolrClient's internal mechanism and worker failure might result data being acknowledged in Storm but not written in Solr. - solr.commit.waitFlush
-
This is a boolean which defines whether the writer blocks the commit until the data is flushed. See See Solr Ref Guide 6.6 - AutoCommits for more information. The default is
true
.NoteIf you set the value tofalse
, then commits will happen based on the SolrClient's internal mechanism and worker failure might result data being acknowledged in Storm but not written in Solr. - solr.collection
-
The default Solr collection (if unspecified, the name is
metron
). By default, sensors will write to a collection associated with the index name in the indexing configuration for that sensor. If that index name is the empty string, then the default collection is used. - solr.http.config
-
This is a map which allows users to configure the Solr client's HTTP client.
Possible fields here are:
- socketTimeout
-
Socket timeout measured in ms; closes a socket if read takes longer than x ms to complete. Displays the following error message:
java.net.SocketTimeoutException: Read timed out exception
. - connTimeout
-
Connection timeout measures in ms; closes a socket if connection cannot be established within x ms and displays the following error message:
java.net.SocketTimeoutException: Connection timed out
. - maxConectionsPerHost
- Maximum connections allowed per host.
- maxConnections
- Maximum total connections allowed.
- retry
- Retry http requests on error.
- allowCompression
- Allow compression (deflate,gzip) if server supports it.
- followRedirects
- Follow redirects.
- httpBasicAuthUser
- Basic authorized username.
- httpBasicAuthPassword
- Basic authorized password.
- solr.ssl.checkPeerName
- Check peer name.
Elasticsearch Index{ "es.clustername": "metron", "es.ip": "node1", "es.port": "9300", "es.date.format": "yyyy.MM.dd.HH", "fieldValidations" : [ { "input" : [ "ip_src_addr", "ip_dst_addr" ], "validation" : "IP", "config" : { "type" : "IPV4" } } ] }
- es.ip
-
A single or collection of elastic search master nodes.
They might be specified using the
hostname:port
syntax. If a port is not specified, then a separate global propertyes.port
is required:- Example:
es.ip
: [ “10.0.0.1:1234”, “10.0.0.2:1234”] - Example:
es.ip
: “10.0.0.1” (thus requiringes.port
to be specified as well) - Example:
es.ip
: “10.0.0.1:1234” (thus not requiringes.port
to be specified)
- Example:
- es.port
-
The port of the elastic search master node.
This is not strictly required if the port is specified in the
es.ip global
property as described above. It is expected that this be an integer or a string representation of an integer.- Example:
es.port
: “1234" - Example:
es.port
: 1234
- Example:
- es.clustername
-
The elastic search cluster name to which you want to write.
- Example:
es.clustername
: “metron” (providing your ES cluster is configured to have metron be a valid cluster name)
- Example:
- es.date.format
-
The format of the date that specifies how the information is parsed time-wise.
or example:
es.date.format
: “yyyy.MM.dd.HH” (this would shard by hour creating, for example, a Bro shard of bro_2016.01.01.01, bro_2016.01.01.02, etc.)es.date.format
: “yyyy.MM.dd” (this would shard by day, creating, for example, a Bro shard of bro_2016.01.01, bro_2016.01.02, etc.)
- fieldValidations
-
A validation framework that enables you to construct validation rules that cross all sensors.
The fieldValidations enrichment value use validation plugins or assertions about fields or whole messages
- input
-
An array of input fields or a single field. If this is omitted, then the whole messages is passed to the validator.
- config
-
A String to Object map for validation configuration. This is optional if the validation function requires no configuration.
- validation
-
The validation function to be used. This is one of the following:
- STELLAR
-
Execute a Stellar Language statement. Expects the query string in the
condition
field of the config. - IP
-
Validates that the input fields are an IP address. By default, if no configuration is set, it assumes IPV4, but you can specify the type by passing in type with either
IPV6
orIPV4
or by passing in a list [IPV4,IPV6
] in which case the input is validated against both. - DOMAIN
-
Validates that the fields are all domains.
-
Validates that the fields are all email addresses.
- URL
-
Validates that the fields are all URLs.
- DATE
-
Validates that the fields are a date. Expects
format
in the configuration. - INTEGER
-
Validates that the fields are an integer. String representation of an integer is allowed.
- REGEX_MATCH
-
Validates that the fields match a regex. Expects
pattern
in the configuration. - NOT_EMPTY
-
Validates that the fields exist and are not empty (after trimming.)