Compactor properties

You check and change a number of Apache Hive properties to configure the compaction of delta files that accumulate during data ingestion. You need to know the defaults and valid values.

Basic compactor properties

hive.compactor.initiator.on: Default=false; Whether to run the initiator threads on this metastore instance or not. Set in only one instance.

hive.compactor.cleaner.on: Default=false; Whether to run the cleaner threads on this metastore instance or not. Set in only one instance.
note
Before HIVE-26908, the cleaner thread could only be enabled or disabled through the hive.compactor.initiator.on property. With the introduction of the new cleaner property, you can enable or disable the cleaner and initiator threads independently.

hive.compactor.worker.threads: Default=0; Set this to a positive number to enable Hive transactions, which are required to trigger transactions. Worker threads spawn jobs to perform compactions, but do not perform the compactions themselves. Increasing the number of worker threads decreases the time that it takes tables or partitions to be compacted. However, increasing the number of worker threads also increases the background load on the CDP cluster because they cause more jobs to run in the background.

hive.metastore.runworker.in: Default=HS2; Specifies where to run the Worker threads that spawn jobs to perform compactions. Valid values are HiveServer (HS2) or Hive metastore (HMS).

hive.compactor.abortedtxn.threshold: Default=1000 aborts; The number of aborted transactions that triggers compaction on a table/partition.

hive.compactor.aborted.txn.time.threshold: Default=12 hours; The hours of aborted transactions that trigger compaction on a table/partition.

Advanced compactor properties

hive.compactor.worker.timeout: Default=86400s; Expects a time value using d/day, h/hour, m/min, s/sec, ms/msec, us/usec, or ns/nsec (default is sec). Time in seconds after which a compaction job will be declared failed and the compaction re-queued.

hive.compactor.check.interval: Default=300s; A valid value is a time with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is sec if not specified.; Time in seconds between checks to see if any tables or partitions need to be compacted. This value should be kept high because each check for compaction requires many calls against the NameNode. Decreasing this value reduces the time it takes to start compaction for a table or partition that requires it. However, checking if compaction is needed requires several calls to the NameNode for each table or partition involved in a transaction done since the last major compaction. Consequently, decreasing this value increases the load on the NameNode.

hive.compactor.delta.num.threshold: Default=10; Number of delta directories in a table or partition that triggers a minor compaction.

hive.compactor.delta.pct.threshold: Default=0.1; Percentage (fractional) size of the delta files relative to the base that triggers a major compaction. (1.0 = 100%, so the default 0.1 = 10%.)

hive.compactor.max.num.delta: Default=500; Maximum number of delta files that the compactor attempts to handle in a single job.

hive.compactor.wait.timeout: Default=300000; The value must be greater than 2000 milliseconds.; Time out in milliseconds for blocking compaction.

hive.compactor.initiator.failed.compacts.threshold: Default=2; A valid value is between 1 and 20, and must be less than hive.compactor.history.retention.failed.; The number of consecutive compaction failures (per table/partition) after which automatic compactions are not scheduled any longer.

hive.compactor.cleaner.run.interval: Default=5000ms; A valid value is a time with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.; The time between runs of the cleaner thread.

hive.compactor.job.queue: Specifies the Hadoop queue name to which compaction jobs are submitted. If the value is an empty string, Hadoop chooses the default queue to submit compaction jobs.; Providing an invalid queue name results in compaction job failures.

hive.compactor.compact.insert.only: Default=true; The compactor compacts insert-only tables, or not (false). A safety switch.

hive.compactor.crud.query.based: Default=false; Performs major compaction on full CRUD tables as a query, and disables minor compaction.

hive.split.grouping.mode: Default=query; A valid value is either query or compactor.; This property is set to compactor from within the query-based compactor. This setting enables the Tez SplitGrouper to group splits based on their bucket number, so that all rows from different bucket files for the same bucket number can end up in the same bucket file after the compaction.

hive.compactor.history.retention.succeeded: Default=3; A valid value is between 0 and 100.; Determines how many successful compaction records are retained in compaction history for a given table/partition.

hive.compactor.history.retention.failed: Default=3; A valid value is between 0 and 100.; Determines how many failed compaction records are retained in compaction history for a given table/partition.

hive.compactor.history.retention.attempted: Default=2; A valid value is between 0 and 100.; Determines how many attempted compaction records are retained in compaction history for a given table/partition.

hive.compactor.history.reaper.interval: Default=2m; A valid value is a time with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.; Determines how often the compaction history reaper runs.