org.apache.hadoop.hbase.mapreduce

Class LoadIncrementalHFiles

java.lang.Object

Configured

org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class LoadIncrementalHFiles
extends Configured

Tool to load the output of HFileOutputFormat into an existing table.

See Also:

usage()

Field Summary

Fields

Modifier and Type	Field and Description
static String	CREATE_TABLE_CONF_KEY
static String	MAX_FILES_PER_REGION_PER_FAMILY
static String	NAME

Constructor Summary

Constructors

Constructor and Description
LoadIncrementalHFiles(Configuration conf)

Method Summary

Modifier and Type	Method and Description
protected void	bulkLoadPhase(Table table, Connection conn, ExecutorService pool, Deque<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> queue, <any> regionGroups) This takes the LQI's grouped by likely regions and attempts to bulk load them.
void	doBulkLoad(Path hfofDir, Admin admin, Table table, RegionLocator regionLocator) Perform a bulk load of the given directory into the given pre-existing table.
void	doBulkLoad(Path hfofDir, HTable table) Perform a bulk load of the given directory into the given pre-existing table.
protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem>	groupOrSplit(<any> regionGroups, org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem item, Table table, Pair<byte[][],byte[][]> startEndKeys) Attempt to assign the given load queue item into its target region group.
static byte[][]	inferBoundaries(TreeMap<byte[],Integer> bdryMap)
static void	main(String[] args)
int	run(String[] args)
protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem>	splitStoreFile(org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem item, Table table, byte[] startKey, byte[] splitKey)
protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(Connection conn, TableName tableName, byte[] first, Collection<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> lqis) Attempts to do an atomic load of many hfiles into a region.
protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(HConnection conn, byte[] tableName, byte[] first, Collection<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> lqis) Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use tryAtomicRegionLoad(Connection, TableName, byte[], Collection).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final String NAME

See Also:

Constant Field Values

MAX_FILES_PER_REGION_PER_FAMILY

public static final String MAX_FILES_PER_REGION_PER_FAMILY

See Also:

Constant Field Values

CREATE_TABLE_CONF_KEY

public static final String CREATE_TABLE_CONF_KEY

See Also:

Constant Field Values

Constructor Detail

LoadIncrementalHFiles

public LoadIncrementalHFiles(Configuration conf)
                      throws Exception

Throws:

Exception

Method Detail

doBulkLoad

public void doBulkLoad(Path hfofDir,
                       HTable table)
                throws TableNotFoundException,
                       IOException

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

table - the table to load into

Throws:

TableNotFoundException - if table does not yet exist

IOException

doBulkLoad

public void doBulkLoad(Path hfofDir,
                       Admin admin,
                       Table table,
                       RegionLocator regionLocator)
                throws TableNotFoundException,
                       IOException

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

table - the table to load into

Throws:

TableNotFoundException - if table does not yet exist

IOException

bulkLoadPhase

protected void bulkLoadPhase(Table table,
                             Connection conn,
                             ExecutorService pool,
                             Deque<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> queue,
                             <any> regionGroups)
                      throws IOException

This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

Throws:

IOException

splitStoreFile

protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> splitStoreFile(org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem item,
                                                                                                     Table table,
                                                                                                     byte[] startKey,
                                                                                                     byte[] splitKey)
                                                                                              throws IOException

Throws:

IOException

groupOrSplit

protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> groupOrSplit(<any> regionGroups,
                                                                                                   org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem item,
                                                                                                   Table table,
                                                                                                   Pair<byte[][],byte[][]> startEndKeys)
                                                                                            throws IOException

Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles. protected for testing

Throws:

IOException

tryAtomicRegionLoad

@Deprecated
protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(HConnection conn,
                                                                                                                      byte[] tableName,
                                                                                                                      byte[] first,
                                                                                                                      Collection<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                                                               throws IOException

Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use tryAtomicRegionLoad(Connection, TableName, byte[], Collection).

Throws:

IOException

tryAtomicRegionLoad

protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(Connection conn,
                                                                                                          TableName tableName,
                                                                                                          byte[] first,
                                                                                                          Collection<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                                                   throws IOException

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list. NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically. Protected for testing.

Returns:

empty list if success, list of items to retry on recoverable failure

Throws:

IOException

inferBoundaries

public static byte[][] inferBoundaries(TreeMap<byte[],Integer> bdryMap)

run

public int run(String[] args)
        throws Exception

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception