Configuring the Data Compliance Profiler

In addition to the generic configuration, you can configure scheduling and available resources for the Cluster Sensitivity Profiler.

Go to Profilers and select your data lake.
Go to Profilers > Data Compliance > Profiler Details > Configuration > All Configurations.
Select a schedule to run profiler using either UNIX Cron Expression or the Basic scheduler.

note
Both the Basic and Cron Expression scheduler use the UTC timezone instead of the local timezone of the user.

Figure 1. Profiler schedule with cron expression

Figure 2. Profiler schedule with natural language
Select Incremental Profiling when needed.

Using Incremental Profiling can decrease the compute resources and the time needed for the profiling job by processing only the information (only Iceberg tables) updated or added since the previous job.

With Incremental Profiling, you can refine the results from the Last Run Check. Incremental Profiling checks the data (rows) in assets, while Last Run Check filters complete assets.
Select Last Run Check and set a period in Day Range if needed.

note

The Last Run Check configuration enables profilers to avoid profiling the same asset on each scheduled run.

If you have scheduled a cron job, for example set to start in about an hour, and have enabled the Last Run Check configuration for two days, this setup ensures that the job scheduler filters out any asset which was already profiled in the last two days.

If the Last Run Check configuration is disabled, assets will be picked up for profiling as per the job cron schedule, honoring the asset filter rules.

The Last Run Check precedes Incremental Profiling.
Configure the resources.

note
Is your infrastructure is deployed completely in AWS, profilers might hang if the underlying AWS cloud provider environment cannot provide the necessary memory for the executor instances. In this case, reconfigure your executors with 4-5 GB memory in Profiler Details > Configuration.
1. Set the Maximum number of executors.
  
  Specifies the number of processes that are used by the distributed computing framework. The recommended value is at least 10 executors.
2. Set the Maximum cores per executor.
  
  Specifies the maximum number of cores that can be allocated to an executor.
3. Set the Executor memory limit in GBs.
4. Set the Number of driver cores.
  
  Specifies the maximum number of driver cores. Increase the number of cores to improve the speed of profiler job scheduling.
5. Set the Maximum driver memory in GBs.
  
  Specifies the maximum amount of memory that can be allocated to a driver core. Increasing the available memory accelerates the profiling of larger and more complex tables and prevents out-of-memory errors.
Click Save to apply the configuration changes to the selected profiler.

Add Asset Filtering Rules as needed to customize the selection and deselection of assets which the profiler profiles.

note

Profiler configurations apply to both scheduled and on-demand profiler jobs.
Asset filtering rules apply to assets, such as tables, and not to complete databases. This means that filtering rules work at table level and not at a database level. You are able to filter out all tables of a specific database, but each table will be checked whether its parent is the selected database.
Multiple asset filtering rules are evaluated together as if connected by the OR operator.
You cannot enable conflicting Allow and Deny list rules at the same time. Enabling conflicting rules results in the following error message:

Set your Deny List and Allow-list.

The profiler will skip profiling assets that meet any criteria in the Deny List and will include assets that meet any criteria in the Allow List.

Click Add New Rule to define new rules.
Use the radio buttons to define your new rule for the Allow List or Deny List.

Select one of the following keys from the drop-down list and select the relevant operator:


Key	Operator
Database name	equals starts with ends with
Name (of asset)	equals contains starts with ends with
Owner (of asset)	equals contains starts with ends with
Creation date¹	greater than less than

¹ By Creation Date, Greater than 7 days means an asset older than seven days. Less than 7 days means an asset younger than seven days.

Enter the value corresponding to the key. For example, you can enter a string as mentioned in the previous example.
Click Add Rule. Once a rule is added (enabled by default), you can enable or disable it as needed by clicking the state toggle.

Figure 3. Affected Assets in Asset Filtering Rules configuration

Job Summary shows the asset filtering rules applied for the particular profiling job.