Initial Configuration and Tuning
HBase administrators typically use the following methods to initially configure the cluster:
Increase the request handler thread count
Configure the size and number of WAL files
Configure compactions
Pre-split tables
Tune JVM garbage collection
Increase the Request Handler Thread Count
Administrators who expect their HBase cluster to experience a high volume request pattern should increase the number of listeners generated by the RegionServers. Use the hbase.regionserver.handler.count
property in the hbase-site.xml
configuration file to set the number higher than the default value of 30
.
Configure the Size and Number of WAL Files
HBase uses the Write Ahead Log, or WAL, to recover memstore data not yet flushed to disk if a RegionServer crashes. Administrators should configure these WAL files to be slightly smaller than the HDFS block size. By default, an HDFS block is 64 MB and a WAL is approximately 60 MB. Hortonworks recommends that administrators ensure that enough WAL files are allocated to contain the total capacity of the memstores. Use the following formula to determine the number of WAL files needed:
(regionserver_heap_size * memstore fraction) / (default_WAL_size)
For example, assume the following HBase cluster configuration:
16 GB RegionServer heap
0.4 memstore fraction
60 MB default WAL size
The formula for this configuration looks as follows:
(16384 MB * 0.4 / 60 MB = approximately 109 WAL files
Use the following properties in the hbase-site.xml
configuration file to configure the size and number of WAL files:
Configuration Property | Description | Default |
---|---|---|
hbase.regionserver.maxlogs | Sets the maximum number of WAL files. | 32 |
hbase.regionserver.logroll.multiplier | Multiplier of HDFS block size. | 0.95 |
hbase.regionserver.hlog.blocksize | Optional override of HDFS block size. | Value assigned to actual HDFS block size. |
Note | |
---|---|
If recovery from failure takes longer than expected, try reducing the number of WAL files to improve performance. |