Develop Profiles

Troubleshooting issues when programming against a live stream of data can be difficult. The Stellar REPL (an interactive top level or language shell) is a powerful tool to help work out the kinds of enrichments and transformations that are needed. The Stellar REPL can also be used to help when developing profiles for the Profiler.

1. Take a first pass at defining your profile.
   For example, in the editor copy/paste the basic Hello, World profile below.

   [Stellar]>>> conf := SHELL_EDIT()
   [Stellar]>>> conf
   {
     "profiles": [
       {
         "profile": "hello-world",
         "onlyif":  "exists(ip_src_addr)",
         "foreach": "ip_src_addr",
         "init":    { "count": "0" },
         "update":  { "count": "count + 1" },
         "result":  "count"
       }
     ]
   }
   

2. Initialize the Profiler.

   [Stellar]>>> profiler := PROFILER_INIT(conf) 
   [Stellar]>>> profiler 
   org.apache.metron.profiler.StandAloneProfiler@4f8ef473 

3. Create a message to simulate the type of telemetry that you expect to be profiled.
   For example, in the editor copy/paste the JSON below.

   [Stellar]>>> message := SHELL_EDIT() 
   [Stellar]>>> message 
   { 
      "ip_src_addr": "10.0.0.1", 
      "protocol": "HTTPS", 
      "length": "10", 
      "bytes_in": "234" 
   } 

4. Apply some telemetry messages to your profiles. The following applies the same message 3 times.

   [Stellar]>>> PROFILER_APPLY(message, profiler) 
   org.apache.metron.profiler.StandAloneProfiler@4f8ef473 
   
   [Stellar]>>> PROFILER_APPLY(message, profiler) 
   org.apache.metron.profiler.StandAloneProfiler@4f8ef473 
   
   [Stellar]>>> PROFILER_APPLY(message, profiler) 
   org.apache.metron.profiler.StandAloneProfiler@4f8ef473 

5. Flush the Profiler to see what has been calculated.

   [Stellar]>>> values := PROFILER_FLUSH(profiler) 
   [Stellar]>>> values 
   [{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000}, 
      profile=hello-world, groups=[], value=3, entity=10.0.0.1}]

   A flush is what occurs at the end of each 15 minute period in the Profiler. The result is a list of profile measurements. Each measurement is a map containing detailed information about the profile data that has been generated.
   This profile counts the number of messages by IP source address. Notice that the value is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr' of '10.0.0.1'. There will always be one measurement for each [profile, entity] pair.
6. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat' this process. After you are satisfied with the data being generated by the profile, then follow the Getting Started guide to use the profile against your live, streaming data in a Metron cluster.