You can specify the profile to use for batch processing as either a JSON file on disk
or by utilizing a profile already loaded into ZooKeeper for use by the streaming profiler.
-
To load a profile from disk:
-
If a profile file does not already exist, create a profile definition by
editing
$METRON_HOME/config/zookeeper/profiler.json
as
follows:
cat $METRON_HOME/config/zookeeper/profiler.json
{
"profiles": [
{
"profile": "hello-world",
"foreach": "'global'",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
],
"timestampField": "timestamp"
}
-
When launching the batch profiler directly, use the
--profiles
<path to profiler.json>
option.
If using the wrapper script to launch the batch profiler, it will
automatically add the command argument --profiles
$METRON_HOME/config/zookeeper/profiler.json
to the batch launching
process if $SPARK_PROFILER_USE_ZOOKEEPER
is not
defined.
-
To load a profile from ZooKeeper:
-
Choose to use profiles already loaded into zookeeper (for example, for use by
the streaming profiler) by setting the environment variable
$SPARK_PROFILER_USE_ZOOKEEPER
.
This will cause the wrapper script to add --zookeeper
$ZOOKEEPER
to the batch launching process, which will cause the
spark profiler to extract profiles from the zookeeper quorum located at
$ZOOKEEPER
.