Tuning Parameters for the Profiler Service
You can update the configuration settings to use minimal resources for the Profiler service if you have limited resources in your queue.
The instructions specified here are not the minimal requirements. When you have limited resources in your queue, you can tune your settings to ensure optimum performance.
If you have resource limitations with your queue, make sure the minimum resource requirements are as follows:
- RAM should be greater than or equal to 16 GB.
- CPU Cores should be greater than equal to 8.
Set the number of executors for Livy interactive sessions to 1.
- Log in to Ambari.
- From Ambari, select Dataplane Profiler and click Configs.
In the Advanced dpprofiler-livy-config section, set the following properties:
- Number Of Executors For Write Session =1
- Number Of Executors For Read Session=1
Set the number of executors for profiler jobs to 1.
- Log in to the Data Steward Studio app.
- Go to profiler configuration and click edit.
- Select a profiler. The Profile Configuration tab opens on the right.
- Update the details of the number of executors in the Advanced Options section. Set the number of executors to 1.
Update the YARN parameters as follows:
yarn.scheduler.capacity.maximum-am-resource-percentparameter on YARN > Scheduler (let this be x) such that
(x * total_memory_in_yarn) >= 8G
For example, for 16 GB it is advised to set x to 0.5.
All these resources must be allocated exclusively for profiler agent and profilers. It is advisable to have a separate queue.