Specify Profiles

You can specify the profile to use for batch processing as either a JSON file on disk or by utilizing a profile already loaded into ZooKeeper for use by the streaming profiler.

  1. To load a profile from disk:
    1. If a profile file does not already exist, create a profile definition by editing $METRON_HOME/config/zookeeper/profiler.json as follows:
      cat $METRON_HOME/config/zookeeper/profiler.json
      {
        "profiles": [
          {
            "profile": "hello-world",
            "foreach": "'global'",
            "init":    { "count": "0" },
            "update":  { "count": "count + 1" },
            "result":  "count"
          }
        ],
        "timestampField": "timestamp"
      }
    2. When launching the batch profiler directly, use the --profiles <path to profiler.json> option.
      If using the wrapper script to launch the batch profiler, it will automatically add the command argument --profiles $METRON_HOME/config/zookeeper/profiler.json to the batch launching process if $SPARK_PROFILER_USE_ZOOKEEPER is not defined.
  2. To load a profile from ZooKeeper:
    1. Choose to use profiles already loaded into zookeeper (for example, for use by the streaming profiler) by setting the environment variable $SPARK_PROFILER_USE_ZOOKEEPER.
      This will cause the wrapper script to add --zookeeper $ZOOKEEPER to the batch launching process, which will cause the spark profiler to extract profiles from the zookeeper quorum located at $ZOOKEEPER.