public class IntegrationTestSparkBulkLoad extends IntegrationTestBase
IntegrationTestBulkLoad
in mapreduce.
Usage on cluster:
First add hbase related jars and hbase-spark.jar into spark classpath.
spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad
HBASE_HOME/lib/hbase-spark-it-XXX-tests.jar -m slowDeterministic -Dhbase.spark.bulkload.chainlength=300Modifier and Type | Class and Description |
---|---|
static class |
IntegrationTestSparkBulkLoad.createCombinerFunc |
static class |
IntegrationTestSparkBulkLoad.LinkedListCheckingFlatMapFunc
PairFlatMapFunction used to transfer
|
static class |
IntegrationTestSparkBulkLoad.LinkedListCheckingForeachFunc
Sort all LinkChain for one LinkKey, and test List
|
static class |
IntegrationTestSparkBulkLoad.LinkedListCreationMapper |
static class |
IntegrationTestSparkBulkLoad.ListToKeyValueFunc |
static class |
IntegrationTestSparkBulkLoad.mergeCombinersFunc |
static class |
IntegrationTestSparkBulkLoad.mergeValueFunc |
static class |
IntegrationTestSparkBulkLoad.NaturalKeyPartitioner
Class to figure out what partition to send a link in the chain to.
|
static class |
IntegrationTestSparkBulkLoad.SparkLinkChain
Writable used as the value emitted from a pass over the hbase table.
|
static class |
IntegrationTestSparkBulkLoad.SparkLinkKey
Writable class used as the key to group links in the linked list.
|
CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util
Constructor and Description |
---|
IntegrationTestSparkBulkLoad() |
Modifier and Type | Method and Description |
---|---|
protected void |
addOptions() |
protected java.util.Set<java.lang.String> |
getColumnFamilies()
Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
|
TableName |
getTablename()
Provides the name of the table that is protected from random Chaos monkey activity
|
static TableName |
getTableName(Configuration conf) |
static void |
main(java.lang.String[] args) |
protected void |
processOptions(CommandLine cmd) |
void |
runCheck()
After adding data to the table start a mr job to
|
void |
runLinkedListSparkJob(int iteration)
Running spark job to create LinkedList for testing
|
void |
runLoad()
Running spark job to load data into hbase table
|
int |
runTestFromCommandLine() |
void |
setUpCluster() |
void |
testBulkLoad() |
cleanUp, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey
public void runLoad() throws java.lang.Exception
java.lang.Exception
public void runLinkedListSparkJob(int iteration) throws java.lang.Exception
iteration
- iteration th of this jobjava.lang.Exception
public void runCheck() throws java.lang.Exception
java.io.IOException
java.lang.ClassNotFoundException
java.lang.InterruptedException
java.lang.Exception
public void testBulkLoad() throws java.lang.Exception
java.lang.Exception
public void setUpCluster() throws java.lang.Exception
setUpCluster
in class IntegrationTestBase
java.lang.Exception
protected void addOptions()
addOptions
in class IntegrationTestBase
protected void processOptions(CommandLine cmd)
processOptions
in class IntegrationTestBase
public int runTestFromCommandLine() throws java.lang.Exception
runTestFromCommandLine
in class IntegrationTestBase
java.lang.Exception
public TableName getTablename()
IntegrationTestBase
getTablename
in class IntegrationTestBase
public static TableName getTableName(Configuration conf)
protected java.util.Set<java.lang.String> getColumnFamilies()
IntegrationTestBase
getColumnFamilies
in class IntegrationTestBase
public static void main(java.lang.String[] args) throws java.lang.Exception
java.lang.Exception