Troubleshoot Streaming Profiles By Using Stellar

Troubleshooting issues when programming against a live stream of data can be difficult. The Stellar REPL (an interactive top level or language shell) is a powerful tool to help work out the kinds of enrichments and transformations that are needed. The Stellar REPL can also be used to help when developing profiles for the profiler.

Follow these steps in the Stellar REPL to see how it can be used to help create profiles.
  1. Launch the Stellar REPL.
    Stellar, Go!
  2. Ensure the following functions are accessible.
    [Stellar]>>> %functions PROFILER
  3. Use the SHELL_EDIT function to create a simple hello-world profile that will count the number of messages for each ip_src_addr. The SHELL_EDIT function will open an editor into which you can add the following profiler configuration.
    [Stellar]>>> conf := SHELL_EDIT()
    [Stellar]>>> conf
      "profiles": [
          "profile": "hello-world",
          "foreach": "ip_src_addr",
          "init":    { "count": "0" },
          "update":  { "count": "count + 1" },
          "result":  "count"
    You can also include the timestampField to:
    • List the system time, which is the time at which you are processing the data.
    • List the event time, which is the time contained in the data itself.
  4. Create the profile execution environment.
    The profiler will output the number of profiles that have been defined, the number of messages that have been applied, and the number of routes that have been followed. A route is defined when a message is applied to a specific profile.
    • If a message is not needed by any profile, then there are no routes.
    • If a message is needed by one profile, then one route has been followed.
    • If a message is needed by two profiles, then two routes have been followed.
    [Stellar]>>> profiler := PROFILER_INIT(conf)
    [Stellar]>>> profiler
    Profiler{1 profile(s), 0 messages(s), 0 route(s)}
  5. Create a message to simulate the type of telemetry that you expect to be profiled.
    This message can be as simple or complex as you like. For the hello-world profile, all you need is a message containing an ip_src_addr field.
    [Stellar]>>> msg := SHELL_EDIT()
    [Stellar]>>> msg
    	"ip_src_addr": ""
  6. Apply some telemetry messages to your profiles.
    [Stellar]>>> PROFILER_APPLY(msg, profiler)
    Profiler{1 profile(s), 1 messages(s), 1 route(s)}
    [Stellar]>>> PROFILER_APPLY(msg, profiler)
    Profiler{1 profile(s), 2 messages(s), 2 route(s)}
    [Stellar]>>> PROFILER_APPLY(msg, profiler)
    Profiler{1 profile(s), 3 messages(s), 3 route(s)}
  7. Flush the profiler:
    [Stellar]>>> values := PROFILER_FLUSH(profiler)
    [Stellar]>>> values
    [{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000},
    profile=hello-world, groups=[], value=3, entity=}]
    A flush occurs in the profiler every 15 minutes. The result is a list of profile measurements. Each measurement is a map containing detailed information about the profile data that has been generated. The value field is written to HBase when running the profiler in either Storm or Spark.
    There will always be one measurement for each profile, entity pair. This profile counts the number of messages by IP source address. Notice that the value is 3 for the entity because we applied 3 messages with an ip_src_addr of
  8. In addition to testing with mock data, you can also apply real, live telemetry to your profile.
    The following example extracts 10 messages of live, enriched telemetry to test your profile(s):
    [Stellar]>>> msgs := KAFKA_GET("indexing", 10)
    [Stellar]>>> LENGTH(msgs)
    This can be useful to test your profile against the complexities that exist in real data.
  9. Apply the 10 messages to your profile:
    [Stellar]>>> PROFILER_APPLY(msgs, profiler) Profiler{1 profile(s), 10 messages(s), 10 route(s)}
After you are satisfied with the data being generated by the profile, then use the profile against your live stream of telemetry being captured by CCP.