Cloudera Docs

How compaction impacts Hive ACID table replication

Compaction helps the Hive ACID replication process to work efficiently with the help of ChangeManager. You must consider a few guidelines when you enable compaction for Hive ACID replication. This topic discusses how compaction works and how it benefits Hive ACID table replication.

How compaction works

Compaction is a background process that merges multiple delta and delete delta files into a larger file, and discards the obsolete versions and deleted records. The compaction process is performed at table-level for an unpartitioned table, and at partition-level for a partitioned table. During the compaction process, the "read requests" use the old delta files to capture the "write requests" in new delta files. After the compaction process is complete, the read requests use the new merged file. The old delta files are discarded after the existing read requests are complete.

When there is a high write throughput, the disks’ write bandwidth and resources are shared between the delta file writes. A new database has the bandwidth for these write operations but as the database grows, more bandwidth is required. In this scenario, if the compaction values are not configured carefully, the compaction process cannot keep up with the incoming writes and the unmerged delete file count grows, which in turn affects the disk space and read performance.

By default, compaction is not enabled. You must enable the compaction process using the hive.compactor.initiator.on = true key-value which is applicable to all the Hive tables. On a table-level, you can turn off compaction with the table parameter NO_AUTO_COMPACTION using ALTER TABLE [***TABLE NAME***] SET TBLPROPERTIES ("no_auto_compaction" = "false"). Compaction runs in the background after you enable it depending on the other configuration parameters as discussed in Table 1.

For information about enabling the compaction process, see Initiate automatic compaction in Cloudera Manager.

Table 1. Table 1: Understanding major and minor compactions
Major compactions Minor compactions
Runs automatically when the ratio of the combined size of the delta files to the base file surpasses the hive.compactor.delta.pct.threshold property value, or if the number of aborted transactions exceed the hive.compactor.abortedtxn.threshold value.

You can also trigger it on-demand using the ALTER TABLE command.

Runs automatically when the number of delta directories in a table or partition surpasses the hive.compactor.delta.num.threshold property value.

You can also trigger it on-demand using the ALTER TABLE command.
Produces a single base file after the compaction is complete.

The metastores’ transaction table removes the aborted transactions after the table/partitions related to it are compacted.

 Merges multiple delta files into a single merged delta file. The base file remains intact.

High-level compaction process steps

The following high-level steps show how compaction works in the background when the compaction conditions are met:

  1. The initiator thread identifies the compaction candidate, and places it in the compaction queue table COMPACTION_QUEUE.
  2. After the hive.compactor.check.interval value (configured in seconds) exceeds, the worker thread initiates the major compaction process depending on the queue, and then initiates the minor compaction process.

    When a compaction job remains in the ‘in-progress’ state for a long time, or the worker thread for it is no longer available, the job is moved to the compaction queue table.

  3. When the execution engine is MapReduce, map-only compaction jobs are generated for major and minor compactions where:
    1. the minor compaction job reads all the input files and writes to a single output file, and
    2. the major compaction job reads all the records from the input files, retains the latest version of each record, and then writes to a file.
  4. The cleaner thread initiates the clean up operation after the compaction process is complete. In this process, the thread:
    1. identifies the old delta files using the WriteIDs in the directory names, and deletes them.
    2. deletes the TXN_COMPONENTS and TXNS tables for the partition or table that were compacted during the major compaction process.

Compaction and Hive ACID replication

Hive ACID replication is not a continuous process. It involves an initial bootstrap replication and subsequent incremental replications. ChangeManager plays a big role to maintain data consistency and also ensures efficient replication. You must enable ChangeManager manually. For more information, see Enable ChangeManager.

The replication payload is not self-contained, that is the 'REPL LOAD operation' to copy the data files to the target cluster might not happen immediately after the 'REPL DUMP operation' is complete. There might be an interval between these operations.

The following list discusses the different scenarios when major compaction gets initiated because the required conditions were met during the interval (that is, between the REPL DUMP and REPL LOAD operations), and how ChangeManager manages to preserve data integrity:

  • Major compaction occurs after the REPL DUMP operation is complete on the source.

    In this scenario, the ChangeManager preserves the original files in the REPL DUMP operation and facilitates the copying of data files during the subsequent REPL LOAD operation.

  • Major compaction occurs after a bunch of inserts, updates, and deletes operations are completed on the source during the interval between the REPL DUMP and REPL LOAD operations.

    During the REPL LOAD operation, the delta files in the ChangeManager are considered and not the compacted base file in the actual storage location. Therefore, the REPL LOAD operation completes successfully.

  • Major compaction occurs before the REPL LOAD operation and no files in the source have been modified.

    In this scenario, there is no impact on the replicated data.

For more information about compaction, see data compaction, starting compaction manually, and compactor properties.

Guidelines to ensure the compaction process assists the replication process efficiently

  • The REPL DUMP operation must occur during the replication logs’ retention period.

    Replication logs have a retention period. Older logs that are not in the retention window get purged. For information about retention of replication logs, see Retaining logs for Replication Manager.

  • The REPL LOAD operation must occur during the ChangeManager retention period.

    Add the hive.repl.cm.retain parameter to set the ChangeManager retention period in the Cloudera Manager > Clusters > Hive-on-Tez service > Configuration > Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml property. For more information, see Hive configuration parameters for Hive ACID table replication policies.