Compaction tasks
Compaction in Hive goes hand-in-hand with Hive ACID. Compaction is not, however, necessarily required for Hive ACID. You need to understand when you want, or do not want, compaction to occur.
If you confine your ACID operations tables to full reloads and some delta merges, the performance is steady. As tables are always rewritten (dropped and recreated), there is no need for compaction. In this case, consider disabling automatic compaction.
- You explicitly trigger compaction on a table or partition.
- Automatic compaction finds something to compact.
You run an ALTER TABLE statement to start explicit compaction. Automatic compaction happens without your intervention when Hive periodically crawls through the transaction information, finds tables/partitions affected and those fit for the pre-defined conditions, and marks them for compaction. Conditions are based on the number of deltas, amount of change, and so on.
Compaction of sorted tables is not supported.
In a data pipeline, creating staging or temporary tables can significantly increase the compaction throughput. Avoid compaction of these tables.
