Compactor properties

You check and change a number of Apache Hive properties to configure the compaction of delta files that accumulate during data ingestion. You need to know the defaults and valid values.

Basic compactor properties

hive.compactor.initiator.on
Default=false
Whether to run the initiator and cleaner threads on this metastore instance or not. Set in only one instance.
hive.compactor.worker.threads
Default=0
Set this to a positive number to enable Hive transactions, which are required to trigger transactions. Worker threads spawn jobs to perform compactions, but do not perform the compactions themselves. Increasing the number of worker threads decreases the time that it takes tables or partitions to be compacted. However, increasing the number of worker threads also increases the background load on the CDP cluster because they cause more jobs to run in the background.
hive.metastore.runworker.in
Default=HS2
Specifies where to run the Worker threads that spawn jobs to perform compactions. Valid values are HiveServer (HS2) or Hive metastore (HMS).
hive.compactor.abortedtxn.threshold
Default=1000 aborts
The number of aborted transactions that triggers compaction on a table/partition.
hive.compactor.aborted.txn.time.threshold
Default=12 hours
The hours of aborted transactions that trigger compaction on a table/partition.

Advanced compactor properties

hive.compactor.worker.timeout
Default=86400s

Expects a time value using d/day, h/hour, m/min, s/sec, ms/msec, us/usec, or ns/nsec (default is sec). Time in seconds after which a compaction job will be declared failed and the compaction re-queued.

hive.compactor.check.interval
Default=300s
A valid value is a time with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.
Time in seconds between checks to see if any tables or partitions need to be compacted. This value should be kept high because each check for compaction requires many calls against the NameNode. Decreasing this value reduces the time it takes to start compaction for a table or partition that requires it. However, checking if compaction is needed requires several calls to the NameNode for each table or partition involved in a transaction done since the last major compaction. Consequently, decreasing this value increases the load on the NameNode.
hive.compactor.delta.num.threshold
Default=10
Number of delta directories in a table or partition that triggers a minor compaction.
hive.compactor.delta.pct.threshold
Default=0.1
Percentage (fractional) size of the delta files relative to the base that triggers a major compaction. (1.0 = 100%, so the default 0.1 = 10%.)
hive.compactor.max.num.delta
Default=500
Maximum number of delta files that the compactor attempts to handle in a single job.
hive.compactor.wait.timeout
Default=300000
The value must be greater than 2000 milliseconds.
Time out in milliseconds for blocking compaction.
hive.compactor.initiator.failed.compacts.threshold
Default=2
A valid value is between 1 and 20, and must be less than hive.compactor.history.retention.failed.
The number of consecutive compaction failures (per table/partition) after which automatic compactions are not scheduled any longer.
hive.compactor.cleaner.run.interval
Default=5000ms
A valid value is a time with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.
The time between runs of the cleaner thread.
hive.compactor.job.queue
Specifies the Hadoop queue name to which compaction jobs are submitted. If the value is an empty string, Hadoop chooses the queue.
hive.compactor.compact.insert.only
Default=true
The compactor compacts insert-only tables, or not (false). A safety switch.
hive.compactor.crud.query.based
Default=false
Performs major compaction on full CRUD tables as a query, and disables minor compaction.
hive.split.grouping.mode
Default=query
A valid value is either query or compactor.
This property is set to compactor from within the query-based compactor. This setting enables the Tez SplitGrouper to group splits based on their bucket number, so that all rows from different bucket files for the same bucket number can end up in the same bucket file after the compaction.
hive.compactor.history.retention.succeeded
Default=3
A valid value is between 0 and 100.
Determines how many successful compaction records are retained in compaction history for a given table/partition.
hive.compactor.history.retention.failed
Default=3
A valid value is between 0 and 100.
Determines how many failed compaction records are retained in compaction history for a given table/partition.
hive.compactor.history.retention.attempted
Default=2
A valid value is between 0 and 100.
Determines how many attempted compaction records are retained in compaction history for a given table/partition.
hive.compactor.history.reaper.interval
Default=2m
A valid value is a time with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.
Determines how often the compaction history reaper runs.