Edit Profiler Configuration
In addition to turning on and off the profiler configurations, the individual profilers can be run with their own execution parameters. These parameters are for submission of the profiler job onto Spark. You can edit the configuration of profilers and update these parameters to run profiler jobs.
- Click Profilers in the main navigation menu on the left.
- Click Configs to view all of configured profilers.
- Select the cluster for which you need to edit profiler configuration.
The list of profilers for the selected clusters is displayed.
- Click the name of the profiler whose configuration you wish to edit.
The Profiler Configuration tab is displayed in the right panel.
Select a schedule to run the profiler. This is implemented as a quartz cron
For more information, see Understanding Cron Expression generator.
- Select Last Run Check.
Last Run Check configuration enables profilers like Hive Column Statistics and Cluster Sensitivity Profiler to avoid profiling the same asset on each scheduled run. If you have scheduled a cron job, say for about an hour, and have enabled the Last Run Check configuration for two days, this set-up ensures that the job scheduler filters out any asset which was already profiled in the last two days.
If the Last Run Check configuration is disabled, assets will be picked up for profiling as per the job cron schedule, honouring the asset filter rules, if any.
- Update the advanced options.
For more information, see Configuring SPARK on YARN Applications and Tuning Resource Allocation.
- Number of Executors - Enter the number of executors to launch for running this profiler.
- Executor Cores - Enter the number of cores to be used for each executor.
- Executor Memory - Enter the amount of memory in GB to be used per executor process.
- Driver Cores - Enter the number of cores to be used for the driver process.
- Driver Memory - Enter the memory to be used for the driver processes.
- Toggle the state of the profiler from Active to Inactive as needed.
- Click Save to apply the configuration changes to the selected profiler. The changes should appear in the profiler description.