Data file compression for Cloudera Lakehouse Optimizer

Cloudera Lakehouse Optimizer can automatically re-encode Iceberg data files that match a SQL filter with a more efficient compression codec. This reduces storage costs without changing query results.

Data file compression is distinct from data file compaction (rewriteDataFiles). Compaction is statistics-driven and merges small files. Compression rewrites every file that matches the configured filter using a bin-pack strategy and changes the encoding of existing files regardless of file size or count.

Create a separate compression policy. Do not combine data file compression with compaction or other maintenance actions in the same policy. Because compression targets cold data, run the compression policy on an ad hoc basis when possible, or on a monthly cadence.

Configure data file compression by adding a compressDataFiles block to the policy constants JSON file and associating the policy at the namespace level. Individual tables opt in by setting the filter and other compression properties with ALTER TABLE … SET TBLPROPERTIES.

For compressDataFiles policy constants, supported codecs, per-table overrides, and configuration examples, see JSON file.

Configure data file compression at the namespace level

Associate a compression policy at the namespace level and use a default filter that matches no data until individual tables opt in. Set filter to 1 = 0 in the policy constants JSON file:

"compressDataFiles": {
  "enabled": true,
  "method": "zstd",
  "level": 12,
  "filter": "1 = 0"
}

This value passes validation and produces a no-op for any table that does not have a table-level filter override. Iceberg evaluates the filter against file metadata, finds no matching files, and exits without rewriting anything.

Opt in tables individually by setting the filter with ALTER TABLE … SET TBLPROPERTIES:

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'dlm.compressDataFiles.filter' = 'event_date < ''2024-01-01'''
);

Supported codecs for data file compression

Table 1. Compression codecs
Codec	Level range	Notes
`zstd`	1–22	Best compression ratio; recommended for cold data.
`gzip`	1–9	Wide compatibility.
`brotli`	0–11	—
`snappy`	—	No level; optimizes for speed.
`lz4`	—	No level.
`lzo`	—	No level.
`zlib`	—	No level.
`uncompressed`	—	Removes compression.
`none`	—	Alias for `uncompressed`.

Codec availability depends on the table file format (Parquet, ORC, or Avro). Cloudera Lakehouse Optimizer accepts any value from this list; format-specific filtering happens at execution time in Spark.

Per-table overrides for data file compression

Set the following Iceberg table properties with ALTER TABLE … SET TBLPROPERTIES. Table properties override namespace-level policy constants.

Table 2. compressDataFiles table properties
Table property	Overrides	Description
`dlm.compressDataFiles.method`	`method`	Overrides the compression codec from policy constants.
`dlm.compressDataFiles.level`	`level`	Overrides the compression level from policy constants.
`dlm.compressDataFiles.filter`	`filter`	Overrides the SQL filter from policy constants.
`dlm.compressDataFiles.validateFilterColumns`	`validateFilterColumns`	Set to `false` to skip pre-flight column validation for this table.

Column validation for data file compression

By default, Cloudera Lakehouse Optimizer validates that all column names referenced in the filter exist in the table schema before dispatching the task. If a column is missing, the action is skipped for that table and an error is logged.

Disable this pre-flight check when the filter is known-correct but references constructs the column extractor cannot parse, such as quoted identifiers or complex sub-expressions. Spark still evaluates the filter at runtime. Set validateFilterColumns to false in policy constants or set dlm.compressDataFiles.validateFilterColumns to false on the table.