HashTable/SyncTable tool configuration

Using the batchsize option

You can define the amount of cell data for a given region that is hashed together in a single hash value using the batchsize option, which sets the batchsize property. Sizing this property has a direct impact on the synchronization efficiency. If the batch size is increased, larger chunks are hashed.

If only a few differences are expected between the two tables, using a bit larger batch size can be beneficial, as less scans are executed by mapper tasks of SyncTable.

However, if relatively frequent differences are expected between the tables, using a large batch size can cause frequent mismatches of hash values, as the probability of finding at least one mismatch in a batch is increased.

$ hbase org.apache.hadoop.hbase.mapreduce.HashTable --batchsize=32000 --numhashfiles=50 --starttime=1265875194289 --endtime=1265878794289 --families=cf2,cf3 TestTableA /hashes/testTable

Creating a read-only report

You can use the dryrun option in the second, SyncTable, step to create a read only report. It produces only COUNTERS indicating the differences between the two tables, but does not perform any actual changes. It can be used as an alternative of the VerifyReplication tool.

$ hbase org.apache.hadoop.hbase.mapreduce.SyncTable --dryrun=true --sourcezkcluster=zk1.example.com,zk2.example.com,zk3.example.com:2181:/hbase hdfs://nn:8020/hashes/testTable testTableA testTableB

Handling missing cells

By default, SyncTable changes the target table to make it an exact copy of the source table, at least for the specified startrow-stoprow or starttime-endtime.

Setting doDeletes to false modifies the default behaviour to not delete target cells that are missing on the source table. Similarly, setting doPuts to false modifies the default behaviour to not add missing cells to target table. If you set both doDeletes and doPuts to false, the result will be the same as setting dryrun to true.

In the case of two-way replication or other scenarios where both source and target clusters can have data ingested, Cloudera recommends to set doDeletes to false. Otherwise any additional cells inserted on the SyncTable target cluster and not yet replicated to the source cluster would be deleted, and potentially lost permanently.