public abstract class AbstractFSWAL<W extends WriterBase>
extends java.lang.Object
WAL
to go against FileSystem
; i.e. keep WALs in HDFS. Only one
WAL is ever being written at a time. When a WAL hits a configured maximum size, it is rolled.
This is done internal to the implementation.
As data is flushed from the MemStore to other on-disk structures (files sorted by key, hfiles), a WAL becomes obsolete. We can let go of all the log edits/entries for a given HRegion-sequence id. A bunch of work in the below is done keeping account of these region sequence ids -- what is flushed out to hfiles, and what is yet in WAL and in memory only.
It is only practical to delete entire files. Thus, we delete an entire on-disk file
F
when all of the edits in F
have a log-sequence-id that's older
(smaller) than the most-recent flush.
To read an WAL, call
WALFactory#createReader(org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path)
. *
Modifier and Type | Field and Description |
---|---|
protected long |
blocksize
Block size to use writing files.
|
protected boolean |
closed |
protected Configuration |
conf
conf object
|
protected WALCoprocessorHost |
coprocessorHost |
protected static int |
DEFAULT_SLOW_SYNC_TIME_MS |
protected java.util.concurrent.atomic.AtomicLong |
filenum |
protected FileSystem |
fs
file system instance
|
protected java.util.concurrent.atomic.AtomicLong |
highestSyncedTxid
Updated to the transaction id of the last successful sync call.
|
protected long |
highestUnsyncedTxid
The highest known outstanding unsync'd WALEdit transaction id.
|
protected java.lang.String |
implClassName
The class name of the runtime implementation, used as prefix for logging/tracing.
|
protected java.util.List<WALActionsListener> |
listeners
Listeners that are called on WAL events.
|
protected long |
logrollsize |
protected int |
maxLogs |
protected java.util.concurrent.atomic.AtomicInteger |
numEntries |
protected PathFilter |
ourFiles
Matches just those wal files that belong to this wal instance.
|
protected java.lang.String |
prefixPathStr
Prefix used when checking for wal membership.
|
protected java.util.concurrent.locks.ReentrantLock |
rollWriterLock
This lock makes sure only one log roll runs at a time.
|
protected org.apache.hadoop.hbase.regionserver.wal.SequenceIdAccounting |
sequenceIdAccounting
Class that does accounting of sequenceids in WAL subsystem.
|
protected java.util.concurrent.atomic.AtomicBoolean |
shutdown |
protected long |
slowSyncNs |
protected java.util.concurrent.atomic.AtomicLong |
totalLogSize
The total size of wal
|
protected Path |
walArchiveDir
dir path where old logs are kept.
|
protected Path |
walDir
WAL directory, where all WAL files would be placed.
|
protected java.util.concurrent.ConcurrentNavigableMap<Path,org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.WalProps> |
walFile2Props
Map of WAL log file to properties.
|
protected java.lang.String |
walFilePrefix
Prefix of a WAL file, usually the region server name it is hosted on.
|
protected java.lang.String |
walFileSuffix
Suffix included on generated wal file names
|
Modifier | Constructor and Description |
---|---|
protected |
AbstractFSWAL(FileSystem fs,
Path rootDir,
java.lang.String logDir,
java.lang.String archiveDir,
Configuration conf,
java.util.List<WALActionsListener> listeners,
boolean failIfWALExists,
java.lang.String prefix,
java.lang.String suffix) |
Modifier and Type | Method and Description |
---|---|
void |
abortCacheFlush(byte[] encodedRegionName) |
abstract long |
append(RegionInfo info,
WALKeyImpl key,
WALEdit edits,
boolean inMemstore)
NOTE: This append, at a time that is usually after this call returns, starts an mvcc
transaction by calling 'begin' wherein which we assign this update a sequenceid.
|
protected boolean |
append(W writer,
org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry) |
protected void |
blockOnSync(org.apache.hadoop.hbase.regionserver.wal.SyncFuture syncFuture) |
void |
checkLogLowReplication(long checkInterval) |
void |
close() |
void |
completeCacheFlush(byte[] encodedRegionName) |
protected Path |
computeFilename(long filenum)
This is a convenience method that computes a new filename with a given file-number.
|
protected abstract W |
createWriterInstance(Path path) |
protected abstract void |
doAppend(W writer,
org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry) |
protected abstract boolean |
doCheckLogLowReplication() |
protected abstract void |
doReplaceWriter(Path oldPath,
Path newPath,
W nextWriter) |
protected abstract void |
doShutdown() |
WALCoprocessorHost |
getCoprocessorHost() |
Path |
getCurrentFileName()
This is a convenience method that computes a new filename with a given using the current WAL
file-number
|
long |
getEarliestMemStoreSeqNum(byte[] encodedRegionName) |
long |
getEarliestMemStoreSeqNum(byte[] encodedRegionName,
byte[] familyName) |
long |
getFilenum() |
protected long |
getFileNumFromFileName(Path fileName)
A log file has a creation timestamp (in ms) in its file name (
filenum . |
long |
getLogFileSize() |
java.util.OptionalLong |
getLogFileSizeIfBeingWritten(Path path)
if the given
path is being written currently, then return its length. |
int |
getNumLogFiles() |
int |
getNumRolledLogFiles() |
protected int |
getPreallocatedEventCount() |
protected org.apache.hadoop.hbase.regionserver.wal.SyncFuture |
getSyncFuture(long sequence) |
static Path |
getWALArchivePath(Path archiveDir,
Path p) |
protected void |
logRollAndSetupWalProps(Path oldPath,
Path newPath,
long oldFileLen) |
static void |
main(java.lang.String[] args)
Pass one or more log file names and it will either dump out a text version on
stdout or split the specified log files. |
protected void |
postSync(long timeInNanos,
int handlerSyncs) |
void |
registerWALActionsListener(WALActionsListener listener) |
void |
requestLogRoll() |
protected void |
requestLogRoll(boolean tooFewReplicas) |
byte[][] |
rollWriter() |
byte[][] |
rollWriter(boolean force) |
void |
shutdown() |
protected long |
stampSequenceIdAndPublishToRingBuffer(RegionInfo hri,
WALKeyImpl key,
WALEdit edits,
boolean inMemstore,
<any> ringBuffer) |
java.lang.Long |
startCacheFlush(byte[] encodedRegionName,
java.util.Map<byte[],java.lang.Long> familyToSeq) |
java.lang.Long |
startCacheFlush(byte[] encodedRegionName,
java.util.Set<byte[]> families) |
java.lang.String |
toString() |
boolean |
unregisterWALActionsListener(WALActionsListener listener) |
void |
updateStore(byte[] encodedRegionName,
byte[] familyName,
java.lang.Long sequenceid,
boolean onlyIfGreater)
updates the sequence number of a specific store.
|
protected static final int DEFAULT_SLOW_SYNC_TIME_MS
protected final FileSystem fs
protected final Path walDir
protected final Path walArchiveDir
protected final PathFilter ourFiles
protected final java.lang.String walFilePrefix
protected final java.lang.String walFileSuffix
protected final java.lang.String prefixPathStr
protected final WALCoprocessorHost coprocessorHost
protected final Configuration conf
protected final java.util.List<WALActionsListener> listeners
protected final org.apache.hadoop.hbase.regionserver.wal.SequenceIdAccounting sequenceIdAccounting
protected final long slowSyncNs
protected final long logrollsize
protected final long blocksize
protected final int maxLogs
protected final java.util.concurrent.locks.ReentrantLock rollWriterLock
protected final java.util.concurrent.atomic.AtomicLong filenum
protected final java.util.concurrent.atomic.AtomicInteger numEntries
protected volatile long highestUnsyncedTxid
protected final java.util.concurrent.atomic.AtomicLong highestSyncedTxid
highestUnsyncedTxid
for case where we have an append where a sync has not yet come in
for it.protected final java.util.concurrent.atomic.AtomicLong totalLogSize
protected volatile boolean closed
protected final java.util.concurrent.atomic.AtomicBoolean shutdown
protected java.util.concurrent.ConcurrentNavigableMap<Path,org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.WalProps> walFile2Props
protected final java.lang.String implClassName
Performance testing shows getClass().getSimpleName() might be a bottleneck so we store it here, refer to HBASE-17676 for more details
protected AbstractFSWAL(FileSystem fs, Path rootDir, java.lang.String logDir, java.lang.String archiveDir, Configuration conf, java.util.List<WALActionsListener> listeners, boolean failIfWALExists, java.lang.String prefix, java.lang.String suffix) throws FailedLogCloseException, java.io.IOException
FailedLogCloseException
java.io.IOException
public long getFilenum()
protected long getFileNumFromFileName(Path fileName)
filenum
. This helper
method returns the creation timestamp from a given log file. It extracts the timestamp assuming
the filename is created with the computeFilename(long filenum)
method.protected final int getPreallocatedEventCount()
public void registerWALActionsListener(WALActionsListener listener)
public boolean unregisterWALActionsListener(WALActionsListener listener)
public WALCoprocessorHost getCoprocessorHost()
public java.lang.Long startCacheFlush(byte[] encodedRegionName, java.util.Set<byte[]> families)
public java.lang.Long startCacheFlush(byte[] encodedRegionName, java.util.Map<byte[],java.lang.Long> familyToSeq)
public void completeCacheFlush(byte[] encodedRegionName)
public void abortCacheFlush(byte[] encodedRegionName)
public long getEarliestMemStoreSeqNum(byte[] encodedRegionName)
public long getEarliestMemStoreSeqNum(byte[] encodedRegionName, byte[] familyName)
public byte[][] rollWriter() throws FailedLogCloseException, java.io.IOException
FailedLogCloseException
java.io.IOException
protected Path computeFilename(long filenum)
filenum
- to usepublic Path getCurrentFileName()
public int getNumRolledLogFiles()
public int getNumLogFiles()
public static Path getWALArchivePath(Path archiveDir, Path p)
protected final void logRollAndSetupWalProps(Path oldPath, Path newPath, long oldFileLen)
protected final void blockOnSync(org.apache.hadoop.hbase.regionserver.wal.SyncFuture syncFuture) throws java.io.IOException
java.io.IOException
public byte[][] rollWriter(boolean force) throws FailedLogCloseException, java.io.IOException
FailedLogCloseException
java.io.IOException
public long getLogFileSize()
public void requestLogRoll()
public void shutdown() throws java.io.IOException
java.io.IOException
public void close() throws java.io.IOException
java.io.IOException
public void updateStore(byte[] encodedRegionName, byte[] familyName, java.lang.Long sequenceid, boolean onlyIfGreater)
encodedRegionName
- familyName
- sequenceid
- onlyIfGreater
- protected final org.apache.hadoop.hbase.regionserver.wal.SyncFuture getSyncFuture(long sequence)
protected final void requestLogRoll(boolean tooFewReplicas)
protected final boolean append(W writer, org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry) throws java.io.IOException
java.io.IOException
protected final void postSync(long timeInNanos, int handlerSyncs)
protected final long stampSequenceIdAndPublishToRingBuffer(RegionInfo hri, WALKeyImpl key, WALEdit edits, boolean inMemstore, <any> ringBuffer) throws java.io.IOException
java.io.IOException
public java.lang.String toString()
toString
in class java.lang.Object
public java.util.OptionalLong getLogFileSizeIfBeingWritten(Path path)
path
is being written currently, then return its length.
This is used by replication to prevent replicating unacked log entries. See https://issues.apache.org/jira/browse/HBASE-14004 for more details.
public abstract long append(RegionInfo info, WALKeyImpl key, WALEdit edits, boolean inMemstore) throws java.io.IOException
walKey
parameter. Be warned that the WriteEntry is not
immediately available on return from this method. It WILL be available subsequent to a sync of
this append; otherwise, you will just have to wait on the WriteEntry to get filled in.java.io.IOException
protected abstract void doAppend(W writer, org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry) throws java.io.IOException
java.io.IOException
protected abstract W createWriterInstance(Path path) throws java.io.IOException, CommonFSUtils.StreamLacksCapabilityException
java.io.IOException
CommonFSUtils.StreamLacksCapabilityException
protected abstract void doReplaceWriter(Path oldPath, Path newPath, W nextWriter) throws java.io.IOException
java.io.IOException
protected abstract void doShutdown() throws java.io.IOException
java.io.IOException
protected abstract boolean doCheckLogLowReplication()
public void checkLogLowReplication(long checkInterval)
public static void main(java.lang.String[] args) throws java.io.IOException
stdout
or split the specified log files.java.io.IOException