Stellar Enrichment Configuration
For the geo
, host
, and hbaseEnrichment
, this is
sufficient. However, more complex enrichments might contain their own configuration.
Currently, the stellar
enrichment is more adaptable and thus requires a more
nuanced configuration.
At its most basic, we want to take a message and apply a couple of enrichments, such
as converting the hostname
field to lowercase. We do this by specifying the
transformation inside of the config
for the stellar
fieldMap.
There are two syntaxes that are supported, specifying the transformations as a map with
the key as the field and the value the stellar expression:
"fieldMap": { ... "stellar" : { "config" : { "hostname" : "To_LOWER(hostname)" } } }
Another approach is to make the transformations as a list with the same var :=
expr
syntax as is used in the Stellar REPL, such as:
"fieldMap": { ... "stellar" : { "config" : [ "hostname := TO_LOWER(hostname)" ] } }
Sometimes arbitrary stellar enrichments may take enough time that you would prefer to split some of them into groups and execute the groups of stellar enrichments in parallel. Take, for instance, if you wanted to do an HBase enrichment and a profiler call which were independent of one another. This use case is supported by splitting the enrichments up as groups.
For example:
"fieldMap": { ... "stellar" : { "config" : { "malicious_domain_enrichment" : { "is_bad_domain" : "ENRICHMENT_EXISTS('malicious_domains', ip_dst_addr, 'enrichments', 'cf')" }, "login_profile" : [ "profile_window := PROFILE_WINDOW('from 6 months ago')", "global_login_profile := PROFILE_GET('distinct_login_attempts', 'global', profile_window)", "stats := STATS_MERGE(global_login_profile)", "auth_attempts_median := STATS_PERCENTILE(stats, 0.5)", "auth_attempts_sd := STATS_SD(stats)", "profile_window := null", "global_login_profile := null", "stats := null" ] } } }
Here we want to perform two enrichments that hit HBase and we would rather not run in
sequence. These enrichments are entirely independent of one another (i.e. neither relies
on the output of the other). In this case, we've created a group called
malicious_domain_enrichment
to inquire about whether the destination
address exists in the HBase enrichment table in the malicious_domains
enrichment type. This is a simple enrichment, so we can express the enrichment group as a
map with the new field is_bad_domain
being a key and the stellar expression
associated with that operation being the associated value.
In contrast, the stellar enrichment group login_profile
is interacting
with the profiler, has multiple temporary expressions (for example,
profile_window
, global_login_profile
, and stats
)
that are useful only within the context of this group of stellar expressions. In this
case, we would need to ensure that we use the list construct when specifying the group and
remember to set the temporary variables to null
so they are not passed
along.
In general, things to note from this section are as follows:
The stellar enrichments for the
stellar
enrichment adapter are specified in theconfig
for thestellar
enrichment adapter in thefieldMap
Groups of independent (for example, no expression in any group depend on the output of an expression from an other group) may be executed in parallel
If you have the need to use temporary variables, you may use the list construct. Ensure that you assign the variables to
null
before the end of the group.Ensure that you do not assign a field to a stellar expression which returns an object which JSON cannot represent.
Fields assigned to Maps as part of stellar enrichments have the maps unfolded, similar to the HBase Enrichment
For example the stellar enrichment for field
foo
which assigns a map such asfoo := { 'grok' : 1, 'bar' : 'baz'}
would yield the following fields:foo.grok
==1
foo.bar
=='baz'