How Cloudera Lakehouse Optimizer performs table maintenance
Cloudera Lakehouse Optimizer and its components perform several operations in
the background after you create or define a policy to initiate, evaluate, and complete the
Iceberg table maintenance.
After you create a policy, Cloudera Lakehouse Optimizer performs the following steps:
Initiates the schedule for each table based on the policies attached to the
resource, such as catalog, namespace, and table.
Consolidates the policies depending on the CRON schedule in all the active
policies, and creates a schedule entry for all the policies that share the same
CRON trigger.
Initiates the evaluation phase when a CRON trigger is activated.
The evaluation phase consists of the following steps:
Initiating the evaluation of the policy script per table.
Determining the sequence of maintenance actions per table where the
policy script:
compares the current Iceberg metadata and table statistics to
the expected statistics to evaluate and determine the table
maintenance actions,
generates a list of (potentially empty) maintenance actions, and
then
queues up the generated maintenance actions.
This list of
actions is called a task, and the task has a task
ID.
Sends the maintenance actions to the Spark Engine through Livy in the correct
order of sequence. The Spark engine performs each maintenance action as a Spark
job.
Supervises the task execution, and collects all the relevant task execution
information in the events logs.
Cloudera Lakehouse Optimizer captures every maintenance action, Spark
job, and the job results as task metadata. The task metadata includes the
following elements:
The task ID
The maintenance actions and its results
The Spark computation metrics for the Spark jobs
The current task status
The task metadata is stored in the sys.task_events
table.
The following diagram shows the high-level steps for Iceberg table maintenance
by Cloudera Lakehouse Optimizer:
