State persistence for Cloudera Lakehouse Optimizer
State persistence ensures that Cloudera Lakehouse Optimizer operational state such as scheduled tasks, triggers, and maintenance jobs handles service restarts, failures, and network issues. When you restart a Cloudera Lakehouse Optimizer instance, it resumes from persisted state in the shared database.
Cloudera Lakehouse Optimizer stores scheduler and task state in a shared PostgreSQL database. On startup, the service installs or migrates the database schema if needed, loads persisted triggers and tasks, and resumes scheduling. Cloudera Lakehouse Optimizer does not start without a reachable database.
Configuration
Configure the database connection for Cloudera Lakehouse Optimizer in . Cloudera Lakehouse Optimizer uses this database connection to persist dispatched tasks, event-based triggers, trigger policies, and in-flight Livy task submission data.
Currently, only PostgreSQL is supported. Set dlm_db_type to
postgres. Configure the connection using individual host, port, database,
and credential properties, or provide a JDBC URL with user and password credentials. For the
complete list of properties, see Cloudera Lakehouse Optimizer persistence configuration properties.
What gets persisted
Cloudera Lakehouse Optimizer persists the following state in the database:
- Dispatched tasks: Maintenance jobs that have been scheduled, including tasks that have not yet started executing. Dispatched tasks that are not yet executing survive a service restart.
- Event-based triggers: Custom event-based schedules, next fire times, and associated policies.
- Trigger policies: Maintenance policies attached to each trigger.
- In-flight Livy task submissions: Livy task submission data and session information for executing tasks, therefore Cloudera Lakehouse Optimizer can restore state after a restart and retrieve results from Livy.
Schedule-based (cron) triggers are rebuilt from Cloudera Lakehouse Optimizer configuration on restart and are not persisted long term.
For the database tables that store persisted state, see Cloudera Lakehouse Optimizer persistence database schema.
How persistence works
Rather than bulk-loading entire state on every change, Cloudera Lakehouse Optimizer uses incremental database operations. Each save, update, or delete runs in a database transaction.
On startup, Cloudera Lakehouse Optimizer checks the database schema version and applies any pending schema migrations automatically. Schema migrations run forward only. Back up the database before you upgrade Cloudera Lakehouse Optimizer.
State lifecycle
On startup: The database schema is installed or migrated if needed, existing triggers and tasks are loaded, and the scheduler begins processing.
During runtime: Tasks and triggers are saved, updated, and deleted in the database because they are scheduled, executed, and completed.
On restart: Pending tasks and triggers are reloaded from the database and scheduling resumes from persisted state.
Integration with high availability
In a high availability (HA) deployment, all Cloudera Lakehouse Optimizer instances share the same database. Only the elected leader writes scheduler state. Standby instances read state but do not modify it. On failover, the new leader reads current state from the database without reloading from individual instances.
For more information, see High availability for Cloudera Lakehouse Optimizer.
Limitations
- Cloudera Lakehouse Optimizer does not provide export or import tools for scheduler
state. Use native database backup tools such as
pg_dump. - Database must be reachable at startup.
- Database corruption or extended unavailability requires restore from backup or manual intervention.
- Only PostgreSQL is supported for the persistence database.
