Cloudera Lakehouse Optimizer features

Cloudera Lakehouse Optimizer supports several features.

The following table lists the supported features and the supported methods to use these features:

Table 1. Supported feature list
Features	Available methods to use the feature	Description
Event-based policy	REST API – at table, namespace, and catalog level	Schedules the policies to be evaluated when an HMS event is triggered, such as an insert, update or delete operation on the table. You can create only one version of the policy definition at the catalog level in the UI. However, you can create multiple versions of the policy definition at catalog, namespace, or table level using REST APIs. For example, when you create policy P1 in the UI, the definition is defined at the catalog level. However, using REST APIs you can create another definition for P1 at namespace level or table level.
Schedule-based policy	REST API – at table, namespace, and catalog level	Schedules policies to be evaluated at regular intervals. You can create only one version of the policy definition at the catalog level in UI. However, you can create multiple versions of the policy definition at catalog, namespace, or table level using REST APIs. For more information, see Defining Cloudera Lakehouse Optimizer resources using REST APIs.
Manual (ad hoc) evaluation	REST API	Manually run the policies to optimize the Iceberg tables when required. For more information, see Performing manual Iceberg table maintenance using Cloudera Lakehouse Optimizer REST APIs. note Ensure that you perform a dry run on the policy before you run it manually.
Dry-run policies	REST API	Dry run existing policies to ensure they run effectively without failure. Generates the table maintenance actions but does not initiate any maintenance actions. For more information, see Preparing and defining Cloudera Lakehouse Optimizer policies using REST APIs.
Small file compaction options include: Target file size Minimum number of input files Delete file threshold Maximum concurrent file group rewrites Enable partial progress Maximum number of commits during partial progress Use starting sequence number of snapshot Rewrite all	REST API	Automates the Iceberg data file compaction maintenance actions. In Apache Iceberg documentation, this procedure is called rewrite_data_files, and it supports Table, Strategy (binpack or sort), sort_order (zorder, sortDirection, NullOrder), options, and where arguments which are also supported by Cloudera Lakehouse Optimizer.
Orphan file removal includes: Delete older than	REST API	Automates the Iceberg orphan file removal maintenance action.
Snapshot expiration options include: Maximum snapshot age Retain last Expire snapshot ID Clean expired files	REST API	Automates the Iceberg snapshot management maintenance actions.
Rewrite manifest options include: Target file size Use caching	REST API	Automates the Iceberg manifest rewrite maintenance actions.
Positional delete rewrite options include: Rewrite job order Enable partial progress Maximum number of commits during partial progress Minimum number of input files Maximum concurrent group rewrites Target file size	REST API	Automates the Iceberg positional delete rewrite maintenance actions.
Data file compression options include: Compression method Compression level Filter	REST API	Re-encodes Iceberg data files that match a SQL filter with a more efficient compression codec to reduce storage costs without changing query results. A filter is required when compression is enabled. Create a separate compression policy from compaction, associate it at the namespace level, and run it on an ad hoc basis or on a monthly cadence because compression targets cold data. Individual tables opt in with `ALTER TABLE … SET TBLPROPERTIES`. For more information, see Data file compression for Cloudera Lakehouse Optimizer.
Pause and resume table maintenance manually	REST API	Pauses table maintenance. The table maintenance is paused in the following scenarios: You manually paused the table maintenance. The recurring failures, during the execution phase of the policy, exceeded the retry value. note When you pause maintenance for a table or namespace, all the associated policies for that table or namespace are not evaluated. Therefore, you must be cautious before you pause maintenance for a table or a namespace. For more information, see Pausing and resuming table maintenance.
High availability	Cloudera Manager configuration	Runs multiple Cloudera Lakehouse Optimizer service instances with automatic leader election and failover. At any time, one instance is the leader that accepts API requests and performs state-mutating operations. Standby instances take over automatically when the leader fails. For more information, see High availability for Cloudera Lakehouse Optimizer and Configuring high availability for Cloudera Lakehouse Optimizer.
Maintenance task prioritization	UI - at catalog level REST API - at catalog, namespace, and table level	Allows you to define a priority level for maintenance tasks that are configured for a policy. Higher-priority tasks are scheduled before lower-priority tasks when multiple policies compete for cluster resources. You can set task priority when you create or edit a policy in the Cloudera Lakehouse Optimizer UI or define the `priority` value in the policy JSON constants when you use REST APIs.
State persistence	Cloudera Manager configuration (shared PostgreSQL database)	Persists scheduler and task state—including dispatched tasks, event-based schedules, and in-flight Livy task submission data—in a shared database so maintenance work survives restarts and high availability failover. For more information, see State persistence for Cloudera Lakehouse Optimizer and Cloudera Lakehouse Optimizer persistence configuration properties.
Puffin statistics	REST API	Allows you to create a custom policy that computes NDV statistics for Iceberg table columns and writes them to a Puffin statistics file registered in the table metadata. Query engines that read Iceberg Puffin files can use these estimates for cost-based optimization. For more information, see Puffin statistics for Cloudera Lakehouse Optimizer.
Auditing	Cloudera Manager configuration	Provides an audit trail of user and system activities on Cloudera Lakehouse Optimizer, including policy creation, editing, deletion, association, namespace operations, authentication events, and table maintenance actions. Audit events are written as NDJSON files to a configurable filesystem location. For more information, see Auditing for Cloudera Lakehouse Optimizer.
Cloudera Lakehouse Optimizer event logging	REST API	Ingests the maintenance task metadata, also called an event, into the sys.clo_events Iceberg table. You can use the table to analyze the event logs, use it for troubleshooting purposes and for root cause analysis, and to generate reports. For more information, see Viewing logs for Cloudera Lakehouse Optimizer.
Monitoring policy jobs	REST API	Monitor the policy jobs using one of the following methods: View the latest status for the recent tasks that ran for the table or policy on the UI. Use the GET /tasks or GET /tasks/id/{id} APIs. Monitor the Spark jobs on the Cloudera Consumption dashboard. For more information, see Viewing table maintenance status and Monitoring table maintenance tasks on Cloudera Observability dashboard.
Backup policies and association	REST API	Backs up all the existing policies and associations to a TAR file. You can use the backup file to restore these configurations to any other Cloudera Lakehouse Optimizer service instance, when required. This feature is useful when deleting a current Cloudera Lakehouse Optimizer service instance. For more information, see Cloudera Lakehouse Optimizer REST APIs.
Fine-grained access to namespaces	Ranger UI	Creates Ranger policies and provides the required access to groups or users at namespace level.