How compaction works
When data changes are made on Cloudera Data Warehouse (CDW) with inserts, updates, and deletes, delta files are created. The more changes that are made, the more delta files are created. When a large number of delta files are created, query performance degrades. Compaction removes these delta files to enhance query performance.
There are two types of compaction:
- Minor compaction
- Compacts multiple delta files into a single delta file.
- Major compaction
- Compacts one or more delta files and the base file for the bucket and creates a single new base file per bucket.
The goal of compaction is to "self heal" tables in order to restore the baseline query performance. All compactions are done in the background and do not prevent concurrent reads and writes of the data. After compacting, the system waits for all readers of the old files to finish and then removes the old files.
