Creating Profiles
Also available as:
PDF

Configure the Batch Profiler

You can customize the batch profiler to specify where the profiler runs, the format of the telemetry input, and various other properties available in the batch profiler properties file.

  1. To run the batch profiler using Spark on Yarn, specify the location in $METRON_HOME/config/batch-profiler.properties:
    spark.master=yarn
    By default, the batch profiler instructs Spark to run in local mode: spark.master=local. This mode is only useful for testing with a limited set of data.
  2. You might also want to set the YARN deploy mode to cluster:
    spark.submit.deployMode=cluster
    In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. See the Spark documentation for more information.
  3. Specify the appropriate input format for the batch profiler to consume by modifying $METRON_HOME/config/batch-profiler.properties:
    profiler.batch.input.format=text
    profiler.batch.input.path=hdfs://localhost:8020/apps/metron/indexing/indexed/*/*
    The Profiler can consume archived telemetry stored in a variety of input formats. By default, it is configured to consume the text/json that HCP archives in HDFS. This is often not the best format for archiving telemetry.
  4. Review the batch profiler's properties located at $METRON_HOME/config/batch-profiler.properties to customize them for your profiler needs.
    See Batch Profiler Properties for more information on these properties.