You can update the configuration settings to use minimal resources for the Profiler
service.
Make sure the minimum resource requirements are as follows:
- RAM should be greater than or equal to 16 GB.
- CPU Cores should be greater than equal to 8.
-
Set the number of executors for Livy interactive sessions to 1.
-
Log in to Ambari.
-
From Ambari, select Dataplane Profiler and click Configs.
-
In the Advanced dpprofiler-livy-config section, set the following properties:
- Number Of Executors For Write Session =1
- Number Of Executors For Read Session=1
-
Set the number of executors for profiler jobs to 1.
-
Log in to the Data Steward Studio app.
-
Go to profiler configuration and click edit.
-
Select a profiler. The Profile Configuration tab opens on the right.
-
Update the details of the number of executors in the Advanced Options section. Set the number of executors to 1.
-
Update the YARN parameters as follows:
Set the yarn.scheduler.capacity.maximum-am-resource-percent
parameter on YARN > Scheduler (let this be x) such that
(x * total_memory_in_yarn) >= 8G
For example, for 16 GB it is advised to set x to 0.5.
All these resources must be allocated exclusively for profiler agent and profilers. It is advisable to have a separate queue.