Puffin statistics for Cloudera Lakehouse Optimizer

You can create a custom Cloudera Lakehouse Optimizer policy that computes NDV (number of distinct values) statistics for Iceberg table columns and writes them to a Puffin statistics file registered in the table metadata.

The COMPUTE_TABLE_STATS action reads a target Iceberg table, computes an NDV (theta-sketch) blob for each requested column, and writes those blobs to a Puffin statistics file. Cloudera Lakehouse Optimizer registers the file in the table metadata statisticsFiles list in metadata.json.

Query engines that read Iceberg Puffin files such as Spark cost-based optimizer, Hive CBO, Impala, and Trino can use the NDV estimates to plan joins and aggregations.

Puffin statistics require a custom policy with a JEXL script and an optional JSON constants file that enable the computeTableStats action.

When the policy job completes, Cloudera Lakehouse Optimizer registers a Puffin statistics file in the Iceberg table metadata. Query engines configured to read Puffin files can use the NDV estimates for cost-based optimization.