Stellar Examples
Stellar examples help to illustrate how you can use to Stellar statements to transform and enrich steaming data to identify suspicious behavior.
Consider the basic example of taking a message and applying a couple of enrichments,
such as converting the hostname
field to lowercase. For this
conversion, you must specify the transformation inside of the config
file for the stellar
fieldMap option. Two syntaxes are supported,
specifying the transformations as a map with the key as the field and the value as the
Stellar expression:
"fieldMap": { ... "stellar" : { "config" : { "hostname" : "To_LOWER(hostname)" } } }
Let's consider a situation
where you have a message containing field ip_src_addr
and you want to
determine if the src address is one of a few subnet ranges. You also want to store the
information in a variable called is_local
:
"fieldMap": {
...
"stellar" : {
"config" : {
"is_local := IN_SUBNET( ip_src_addr, '192.168.0.0/16', '10.10.0.0/8')"
}
}
}
Now, let's consider a situation where you want to determine if the top level domain of a
domain name, stored in a field called domain
, is from a specific set of
whitelisted TLDs:
is_government := DOMAIN_TO_TLD(domain) in [ 'mil', 'gov' ]
Let’s assume further that the data coming in is known to be spotty with possible spaces and a dot at the end periodically due to a known upstream data ingest mistake. You can do that with three Stellar statements, the first two sanitizing the domain field and the final statement performing the whitelist check:
sanitized_domain := TRIM(domain)
sanitized_domain := if ENDS_WITH(sanitized_domain, '.') then CHOP(sanitized_domain) else sanitized_domain
is_government := DOMAIN_TO_TLD( sanitized_domain ) in [ 'mil', 'gov' ]
Now, let’s consider a situation where you have a blacklist of known malicious domains.
You can use the HCP data importer to store this data in HBase under the enrichment type
malicious_domains
. As data streams by, you will want to indicate
whether a domain is malicious or not. Further, as before, you still have some ingestion
cruft to adjust:
sanitized_domain := TRIM(domain)
sanitized_domain := if ENDS_WITH(sanitized_domain, '.') then CHOP(sanitized_domain) else sanitized_domain
in_blacklist := ENRICHMENT_EXISTS('malicious_domains', sanitized_domains, 'enrichments', 't')