Run the Batch Profiler
CCP provides a script called start_batch_profiler.sh
to simplify
running the batch profiler.
The
start_batch_profiler.sh
script assumes
the following:- The script does one of the following:
- Builds the profiles defined in
$METRON_HOME/config/zookeeper/profiler.json
- Utilizes the profiles already loaded into ZooKeeper quorum at
$ZOOKEEPER
if the environment variable$SPARK_PROFILER_USE_ZOOKEEPER
is set.
- Builds the profiles defined in
- The properties defined in
$METRON_HOME/config/batch-profiler.properties
are passed to both the profiler and Spark. You can define both Spark and profiler properties in this same file. - The script will also configure the event time field to use if the field value is
stored in the
${SPARK_PROFILER_EVENT_TIMESTAMP_FIELD}
environment variable. - Spark is installed at
/usr/hdp/current/spark-client
. This can be overridden if you define an environment variable calledSPARK_HOME
prior to executing the script.
Start the batch profile by entering the following:
source /etc/default/metron
cd $METRON_HOME
bin/start_batch_profiler.sh