org.apache.hadoop.hbase.util

Class LoadTestTool

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.util.LoadTestTool

@InterfaceAudience.LimitedPrivate(value="Tools")
public class LoadTestTool
extends AbstractHBaseTool

A command-line utility that reads, writes, and verifies data. Unlike PerformanceEvaluation, this tool validates the data written, and supports simultaneously writing and reading the same set of keys.

Field Summary

Fields

Modifier and Type	Field and Description
protected BloomType	bloomType
protected Cipher	cipher
protected CommandLine	cmd This will be removed as we factor out the dependency on command line
protected Compression.Algorithm	compressAlgo
protected DataBlockEncoding	dataBlockEncodingAlgo
static byte[][]	DEFAULT_COLUMN_FAMILIES Column families used by the test
static byte[]	DEFAULT_COLUMN_FAMILY Column family used by the test
protected static int	DEFAULT_DATA_SIZE The default data size if not specified
static int	DEFAULT_NUM_REGIONS_PER_SERVER
protected static int	DEFAULT_NUM_THREADS The number of reader/writer threads if not specified
protected static long	DEFAULT_START_KEY
protected static String	DEFAULT_TABLE_NAME Table name to use of not overridden on the command line
protected boolean	deferredLogFlush
protected long	endKey
protected boolean	ignoreConflicts
protected boolean	isBatchUpdate
protected boolean	isInitOnly
protected boolean	isMultiPut
protected boolean	isRead
protected boolean	isSkipInit
protected boolean	isUpdate
protected boolean	isWrite
protected int	maxColDataSize
protected int	maxColsPerKey
protected int	minColDataSize
protected int	minColsPerKey
protected static String	NUM_TABLES
protected int	numUpdaterThreads
protected int	numWriterThreads
protected static String	OPT_BATCHUPDATE
static String	OPT_BLOOM
static String	OPT_COLUMN_FAMILIES
static String	OPT_COMPRESSION
static String	OPT_DATA_BLOCK_ENCODING
static String	OPT_DATA_BLOCK_ENCODING_USAGE
static String	OPT_DEFERRED_LOG_FLUSH
static String	OPT_DEFERRED_LOG_FLUSH_USAGE
static String	OPT_ENCRYPTION
protected static String	OPT_ENCRYPTION_USAGE
static String	OPT_GENERATOR
static String	OPT_GENERATOR_USAGE
protected static String	OPT_INIT_ONLY
static String	OPT_INMEMORY
protected static String	OPT_KEY_WINDOW
protected static String	OPT_MAX_READ_ERRORS
static String	OPT_MULTIGET
static String	OPT_MULTIPUT
protected static String	OPT_NUM_KEYS
static String	OPT_NUM_REGIONS_PER_SERVER
protected static String	OPT_NUM_REGIONS_PER_SERVER_USAGE
protected static String	OPT_READ
static String	OPT_READER
static String	OPT_READER_USAGE
static String	OPT_REGION_REPLICA_ID
protected static String	OPT_REGION_REPLICA_ID_USAGE
static String	OPT_REGION_REPLICATION
protected static String	OPT_REGION_REPLICATION_USAGE
protected static String	OPT_REGIONS_PER_SERVER
protected static String	OPT_SKIP_INIT
protected static String	OPT_START_KEY
static String	OPT_TABLE_NAME
protected static String	OPT_UPDATE
static String	OPT_UPDATER
static String	OPT_UPDATER_USAGE
protected static String	OPT_USAGE_BLOOM
protected static String	OPT_USAGE_COMPRESSION
static String	OPT_USAGE_IN_MEMORY
protected static String	OPT_USAGE_LOAD Usage string for the load option
protected static String	OPT_USAGE_READ Usage string for the read option
protected static String	OPT_USAGE_UPDATE Usage string for the update option
protected static String	OPT_WRITE
static String	OPT_WRITER
static String	OPT_WRITER_USAGE
protected static String	OPT_ZK_PARENT_NODE
protected static String	OPT_ZK_QUORUM
protected MultiThreadedReader	readerThreads
protected long	startKey
protected AtomicReference<Throwable>	thrown
protected int	updatePercent
protected MultiThreadedUpdater	updaterThreads
protected MultiThreadedWriter	writerThreads

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS

Constructor Summary

Constructors

Constructor and Description
LoadTestTool()

Method Summary

Modifier and Type	Method and Description
protected void	addOptions() Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.
protected void	applyColumnFamilyOptions(TableName tableName, byte[][] columnFamilies) Apply column family options such as Bloom filters, compression, and data block encoding.
protected int	doWork() The "main function" of the tool
static byte[]	generateData(Random r, int length)
byte[][]	getColumnFamilies()
protected int	getNumThreads(String numThreadsStr)
void	initTestTable()
protected int	loadTable()
static UserGroupInformation	loginAndReturnUGI(Configuration conf, String username)
static void	main(String[] args)
protected void	processOptions(CommandLine cmd) This method is called to process the options after they have been parsed.
protected String[]	splitColonSeparated(String option, int minNumCols, int maxNumCols)

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getConf, parseArgs, parseInt, parseLong, printUsage, printUsage, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_TABLE_NAME

protected static final String DEFAULT_TABLE_NAME

Table name to use of not overridden on the command line

See Also:

Constant Field Values

DEFAULT_COLUMN_FAMILY

public static byte[] DEFAULT_COLUMN_FAMILY

Column family used by the test

DEFAULT_COLUMN_FAMILIES

public static final byte[][] DEFAULT_COLUMN_FAMILIES

Column families used by the test

DEFAULT_DATA_SIZE

protected static final int DEFAULT_DATA_SIZE

The default data size if not specified

See Also:

Constant Field Values

DEFAULT_NUM_THREADS

protected static final int DEFAULT_NUM_THREADS

The number of reader/writer threads if not specified

See Also:

Constant Field Values

OPT_USAGE_LOAD

protected static final String OPT_USAGE_LOAD

Usage string for the load option

See Also:

Constant Field Values

OPT_USAGE_READ

protected static final String OPT_USAGE_READ

Usage string for the read option

See Also:

Constant Field Values

OPT_USAGE_UPDATE

protected static final String OPT_USAGE_UPDATE

Usage string for the update option

See Also:

Constant Field Values

OPT_USAGE_BLOOM

protected static final String OPT_USAGE_BLOOM

OPT_USAGE_COMPRESSION

protected static final String OPT_USAGE_COMPRESSION

OPT_DATA_BLOCK_ENCODING_USAGE

public static final String OPT_DATA_BLOCK_ENCODING_USAGE

OPT_BLOOM

public static final String OPT_BLOOM

See Also:

Constant Field Values

OPT_COMPRESSION

public static final String OPT_COMPRESSION

See Also:

Constant Field Values

OPT_DEFERRED_LOG_FLUSH

public static final String OPT_DEFERRED_LOG_FLUSH

See Also:

Constant Field Values

OPT_DEFERRED_LOG_FLUSH_USAGE

public static final String OPT_DEFERRED_LOG_FLUSH_USAGE

See Also:

Constant Field Values

OPT_DATA_BLOCK_ENCODING

public static final String OPT_DATA_BLOCK_ENCODING

OPT_INMEMORY

public static final String OPT_INMEMORY

See Also:

Constant Field Values

OPT_USAGE_IN_MEMORY

public static final String OPT_USAGE_IN_MEMORY

See Also:

Constant Field Values

OPT_GENERATOR

public static final String OPT_GENERATOR

See Also:

Constant Field Values

OPT_GENERATOR_USAGE

public static final String OPT_GENERATOR_USAGE

See Also:

Constant Field Values

OPT_WRITER

public static final String OPT_WRITER

See Also:

Constant Field Values

OPT_WRITER_USAGE

public static final String OPT_WRITER_USAGE

See Also:

Constant Field Values

OPT_UPDATER

public static final String OPT_UPDATER

See Also:

Constant Field Values

OPT_UPDATER_USAGE

public static final String OPT_UPDATER_USAGE

See Also:

Constant Field Values

OPT_READER

public static final String OPT_READER

See Also:

Constant Field Values

OPT_READER_USAGE

public static final String OPT_READER_USAGE

See Also:

Constant Field Values

OPT_KEY_WINDOW

protected static final String OPT_KEY_WINDOW

See Also:

Constant Field Values

OPT_WRITE

protected static final String OPT_WRITE

See Also:

Constant Field Values

OPT_MAX_READ_ERRORS

protected static final String OPT_MAX_READ_ERRORS

See Also:

Constant Field Values

OPT_MULTIPUT

public static final String OPT_MULTIPUT

See Also:

Constant Field Values

OPT_MULTIGET

public static final String OPT_MULTIGET

See Also:

Constant Field Values

OPT_NUM_KEYS

protected static final String OPT_NUM_KEYS

See Also:

Constant Field Values

OPT_READ

protected static final String OPT_READ

See Also:

Constant Field Values

OPT_START_KEY

protected static final String OPT_START_KEY

See Also:

Constant Field Values

OPT_TABLE_NAME

public static final String OPT_TABLE_NAME

See Also:

Constant Field Values

OPT_COLUMN_FAMILIES

public static final String OPT_COLUMN_FAMILIES

See Also:

Constant Field Values

OPT_ZK_QUORUM

protected static final String OPT_ZK_QUORUM

See Also:

Constant Field Values

OPT_ZK_PARENT_NODE

protected static final String OPT_ZK_PARENT_NODE

See Also:

Constant Field Values

OPT_SKIP_INIT

protected static final String OPT_SKIP_INIT

See Also:

Constant Field Values

OPT_INIT_ONLY

protected static final String OPT_INIT_ONLY

See Also:

Constant Field Values

NUM_TABLES

protected static final String NUM_TABLES

See Also:

Constant Field Values

OPT_REGIONS_PER_SERVER

protected static final String OPT_REGIONS_PER_SERVER

See Also:

Constant Field Values

OPT_BATCHUPDATE

protected static final String OPT_BATCHUPDATE

See Also:

Constant Field Values

OPT_UPDATE

protected static final String OPT_UPDATE

See Also:

Constant Field Values

OPT_ENCRYPTION

public static final String OPT_ENCRYPTION

See Also:

Constant Field Values

OPT_ENCRYPTION_USAGE

protected static final String OPT_ENCRYPTION_USAGE

OPT_NUM_REGIONS_PER_SERVER

public static final String OPT_NUM_REGIONS_PER_SERVER

See Also:

Constant Field Values

OPT_NUM_REGIONS_PER_SERVER_USAGE

protected static final String OPT_NUM_REGIONS_PER_SERVER_USAGE

See Also:

Constant Field Values

DEFAULT_NUM_REGIONS_PER_SERVER

public static int DEFAULT_NUM_REGIONS_PER_SERVER

OPT_REGION_REPLICATION

public static final String OPT_REGION_REPLICATION

See Also:

Constant Field Values

OPT_REGION_REPLICATION_USAGE

protected static final String OPT_REGION_REPLICATION_USAGE

See Also:

Constant Field Values

OPT_REGION_REPLICA_ID

public static final String OPT_REGION_REPLICA_ID

See Also:

Constant Field Values

OPT_REGION_REPLICA_ID_USAGE

protected static final String OPT_REGION_REPLICA_ID_USAGE

See Also:

Constant Field Values

DEFAULT_START_KEY

protected static final long DEFAULT_START_KEY

See Also:

Constant Field Values

cmd

protected CommandLine cmd

This will be removed as we factor out the dependency on command line

writerThreads

protected MultiThreadedWriter writerThreads

readerThreads

protected MultiThreadedReader readerThreads

updaterThreads

protected MultiThreadedUpdater updaterThreads

startKey

protected long startKey

endKey

protected long endKey

isWrite

protected boolean isWrite

isRead

protected boolean isRead

isUpdate

protected boolean isUpdate

deferredLogFlush

protected boolean deferredLogFlush

dataBlockEncodingAlgo

protected DataBlockEncoding dataBlockEncodingAlgo

compressAlgo

protected Compression.Algorithm compressAlgo

bloomType

protected BloomType bloomType

numWriterThreads

protected int numWriterThreads

minColsPerKey

protected int minColsPerKey

maxColsPerKey

protected int maxColsPerKey

minColDataSize

protected int minColDataSize

maxColDataSize

protected int maxColDataSize

isMultiPut

protected boolean isMultiPut

numUpdaterThreads

protected int numUpdaterThreads

updatePercent

protected int updatePercent

ignoreConflicts

protected boolean ignoreConflicts

isBatchUpdate

protected boolean isBatchUpdate

isSkipInit

protected boolean isSkipInit

isInitOnly

protected boolean isInitOnly

cipher

protected Cipher cipher

thrown

protected AtomicReference<Throwable> thrown

Constructor Detail

LoadTestTool

public LoadTestTool()

Method Detail

splitColonSeparated

protected String[] splitColonSeparated(String option,
                                       int minNumCols,
                                       int maxNumCols)

getNumThreads

protected int getNumThreads(String numThreadsStr)

getColumnFamilies

public byte[][] getColumnFamilies()

applyColumnFamilyOptions

protected void applyColumnFamilyOptions(TableName tableName,
                                        byte[][] columnFamilies)
                                 throws IOException

Apply column family options such as Bloom filters, compression, and data block encoding.

Throws:

IOException

addOptions

protected void addOptions()

Description copied from class: AbstractHBaseTool

Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.

Specified by:

addOptions in class AbstractHBaseTool

processOptions

protected void processOptions(CommandLine cmd)

Description copied from class: AbstractHBaseTool

This method is called to process the options after they have been parsed.

Specified by:

processOptions in class AbstractHBaseTool

initTestTable

public void initTestTable()
                   throws IOException

Throws:

IOException

doWork

protected int doWork()
              throws IOException

Description copied from class: AbstractHBaseTool

The "main function" of the tool

Specified by:

doWork in class AbstractHBaseTool

Throws:

IOException

loadTable

protected int loadTable()
                 throws IOException

Throws:

IOException

generateData

public static byte[] generateData(Random r,
                                  int length)

main

public static void main(String[] args)

loginAndReturnUGI

public static UserGroupInformation loginAndReturnUGI(Configuration conf,
                                                     String username)
                                              throws IOException

Throws:

IOException