org.apache.hadoop.hbase.regionserver.compactions

Class Compactor<T extends Compactor.CellSink>

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.Compactor<T>

Direct Known Subclasses:

AbstractMultiOutputCompactor, DefaultCompactor

@InterfaceAudience.Private
public abstract class Compactor<T extends Compactor.CellSink>
extends java.lang.Object

A compactor is a compaction algorithm associated a given policy. Base class also contains reusable parts for implementing compactors (what is common and what isn't is evolving).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Compactor.CellSink
protected static interface	Compactor.CellSinkFactory<S>
protected static class	Compactor.FileDetails The sole reason this class exists is that java has no ref/out/pointer parameters.
protected static interface	Compactor.InternalScannerFactory

Field Summary

Fields

Modifier and Type	Field and Description
protected Compression.Algorithm	compactionCompression
protected int	compactionKVMax
protected Configuration	conf
protected Compactor.InternalScannerFactory	defaultScannerFactory
protected int	keepSeqIdPeriod specify how many days to keep MVCC values during major compaction
protected CompactionProgress	progress
protected Store	store

Method Summary

Modifier and Type	Method and Description
protected abstract void	abortWriter(T writer)
protected void	appendMetadataAndCloseWriter(StoreFile.Writer writer, Compactor.FileDetails fd, boolean isMajor) Appends the metadata and closes the writer.
protected abstract java.util.List<Path>	commitWriter(T writer, Compactor.FileDetails fd, CompactionRequest request)
protected java.util.List<Path>	compact(CompactionRequest request, Compactor.InternalScannerFactory scannerFactory, Compactor.CellSinkFactory<T> sinkFactory, CompactionThroughputController throughputController, User user)
protected java.util.List<StoreFileScanner>	createFileScanners(java.util.Collection<StoreFile> filesToCompact, long smallestReadPoint) Creates file scanners for compaction.
protected InternalScanner	createScanner(Store store, java.util.List<StoreFileScanner> scanners, long smallestReadPoint, long earliestPutTs, byte[] dropDeletesFromRow, byte[] dropDeletesToRow)
protected InternalScanner	createScanner(Store store, java.util.List<StoreFileScanner> scanners, ScanType scanType, long smallestReadPoint, long earliestPutTs)
protected StoreFile.Writer	createTmpWriter(Compactor.FileDetails fd, boolean shouldDropBehind) Creates a writer for a new file in a temporary directory.
protected Compactor.FileDetails	getFileDetails(java.util.Collection<StoreFile> filesToCompact, boolean allFiles) Extracts some details about the files to compact that are commonly needed by compactors.
CompactionProgress	getProgress()
protected long	getSmallestReadPoint()
protected boolean	performCompaction(Compactor.FileDetails fd, InternalScanner scanner, Compactor.CellSink writer, long smallestReadPoint, boolean cleanSeqId, CompactionThroughputController throughputController, boolean major) Performs the compaction.
protected InternalScanner	postCreateCoprocScanner(CompactionRequest request, ScanType scanType, InternalScanner scanner, User user) Calls coprocessor, if any, to create scanners - after normal scanner creation.
protected InternalScanner	preCreateCoprocScanner(CompactionRequest request, ScanType scanType, long earliestPutTs, java.util.List<StoreFileScanner> scanners) Calls coprocessor, if any, to create compaction scanner - before normal scanner creation.
protected InternalScanner	preCreateCoprocScanner(CompactionRequest request, ScanType scanType, long earliestPutTs, java.util.List<StoreFileScanner> scanners, User user)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

progress

protected volatile CompactionProgress progress

conf

protected final Configuration conf

store

protected final Store store

compactionKVMax

protected final int compactionKVMax

compactionCompression

protected final Compression.Algorithm compactionCompression

keepSeqIdPeriod

protected int keepSeqIdPeriod

specify how many days to keep MVCC values during major compaction

defaultScannerFactory

protected final Compactor.InternalScannerFactory defaultScannerFactory

Method Detail

getProgress

public CompactionProgress getProgress()

getFileDetails

protected Compactor.FileDetails getFileDetails(java.util.Collection<StoreFile> filesToCompact,
                                               boolean allFiles)
                                        throws java.io.IOException

Extracts some details about the files to compact that are commonly needed by compactors.

Parameters:

filesToCompact - Files.

allFiles - Whether all files are included for compaction

Returns:

The result.

Throws:

java.io.IOException

createFileScanners

protected java.util.List<StoreFileScanner> createFileScanners(java.util.Collection<StoreFile> filesToCompact,
                                                              long smallestReadPoint)
                                                       throws java.io.IOException

Creates file scanners for compaction.

Parameters:

filesToCompact - Files.

Returns:

Scanners.

Throws:

java.io.IOException

getSmallestReadPoint

protected long getSmallestReadPoint()

createTmpWriter

protected StoreFile.Writer createTmpWriter(Compactor.FileDetails fd,
                                           boolean shouldDropBehind)
                                    throws java.io.IOException

Creates a writer for a new file in a temporary directory.

Parameters:

fd - The file details.

Returns:

Writer for a new StoreFile in the tmp dir.

Throws:

java.io.IOException - if creation failed

compact

protected java.util.List<Path> compact(CompactionRequest request,
                                       Compactor.InternalScannerFactory scannerFactory,
                                       Compactor.CellSinkFactory<T> sinkFactory,
                                       CompactionThroughputController throughputController,
                                       User user)
                                throws java.io.IOException

Throws:

java.io.IOException

commitWriter

protected abstract java.util.List<Path> commitWriter(T writer,
                                                     Compactor.FileDetails fd,
                                                     CompactionRequest request)
                                              throws java.io.IOException

Throws:

java.io.IOException

abortWriter

protected abstract void abortWriter(T writer)
                             throws java.io.IOException

Throws:

java.io.IOException

preCreateCoprocScanner

protected InternalScanner preCreateCoprocScanner(CompactionRequest request,
                                                 ScanType scanType,
                                                 long earliestPutTs,
                                                 java.util.List<StoreFileScanner> scanners)
                                          throws java.io.IOException

Calls coprocessor, if any, to create compaction scanner - before normal scanner creation.

Parameters:

request - Compaction request.

scanType - Scan type.

earliestPutTs - Earliest put ts.

scanners - File scanners for compaction files.

Returns:

Scanner override by coprocessor; null if not overriding.

Throws:

java.io.IOException

preCreateCoprocScanner

protected InternalScanner preCreateCoprocScanner(CompactionRequest request,
                                                 ScanType scanType,
                                                 long earliestPutTs,
                                                 java.util.List<StoreFileScanner> scanners,
                                                 User user)
                                          throws java.io.IOException

Throws:

java.io.IOException

postCreateCoprocScanner

protected InternalScanner postCreateCoprocScanner(CompactionRequest request,
                                                  ScanType scanType,
                                                  InternalScanner scanner,
                                                  User user)
                                           throws java.io.IOException

Calls coprocessor, if any, to create scanners - after normal scanner creation.

Parameters:

request - Compaction request.

scanType - Scan type.

scanner - The default scanner created for compaction.

Returns:

Scanner scanner to use (usually the default); null if compaction should not proceed.

Throws:

java.io.IOException

performCompaction

protected boolean performCompaction(Compactor.FileDetails fd,
                                    InternalScanner scanner,
                                    Compactor.CellSink writer,
                                    long smallestReadPoint,
                                    boolean cleanSeqId,
                                    CompactionThroughputController throughputController,
                                    boolean major)
                             throws java.io.IOException

Performs the compaction.

Parameters:

fd - FileDetails of cell sink writer

scanner - Where to read from.

writer - Where to write to.

smallestReadPoint - Smallest read point.

cleanSeqId - When true, remove seqId(used to be mvcc) value which is <= smallestReadPoint

major - Is a major compaction.

Returns:

Whether compaction ended; false if it was interrupted for some reason.

Throws:

java.io.IOException

createScanner

protected InternalScanner createScanner(Store store,
                                        java.util.List<StoreFileScanner> scanners,
                                        ScanType scanType,
                                        long smallestReadPoint,
                                        long earliestPutTs)
                                 throws java.io.IOException

Parameters:

store - store

scanners - Store file scanners.

scanType - Scan type.

smallestReadPoint - Smallest MVCC read point.

earliestPutTs - Earliest put across all files.

Returns:

A compaction scanner.

Throws:

java.io.IOException

createScanner

protected InternalScanner createScanner(Store store,
                                        java.util.List<StoreFileScanner> scanners,
                                        long smallestReadPoint,
                                        long earliestPutTs,
                                        byte[] dropDeletesFromRow,
                                        byte[] dropDeletesToRow)
                                 throws java.io.IOException

Parameters:

store - The store.

scanners - Store file scanners.

smallestReadPoint - Smallest MVCC read point.

earliestPutTs - Earliest put across all files.

dropDeletesFromRow - Drop deletes starting with this row, inclusive. Can be null.

dropDeletesToRow - Drop deletes ending with this row, exclusive. Can be null.

Returns:

A compaction scanner.

Throws:

java.io.IOException

appendMetadataAndCloseWriter

protected void appendMetadataAndCloseWriter(StoreFile.Writer writer,
                                            Compactor.FileDetails fd,
                                            boolean isMajor)
                                     throws java.io.IOException

Appends the metadata and closes the writer.

Parameters:

writer - The current store writer.

fd - The file details.

isMajor - Is a major compaction.

Throws:

java.io.IOException