System Administration
Also available as:
PDF

Increase Memstore Size for region server

Usage of the RegionServer's memstore largely determines the maximum number of regions for the region server. Each region has its own memstores, one for each column family, which grow to a configurable size, usually between 128 and 256 Mb. Administrators specify this size with the hbase.hregion.memstore.flush.size property in the hbase-site.xml configuration file. The region server dedicates some fraction of total memory to region memstores based on the value of the hbase.regionserver.global.memstore.size configuration property. If usage exceeds this configurable size, HBase may become unresponsive or compaction storms might occur.

Use the following formula to estimate the number of Regions for a RegionServer:

(regionserver_memory_size) * (memstore_fraction) / ((memstore_size) * (num_column_families))

For example, assume the following configuration:

  • region server with 16 Gb RAM (or 16384 Mb)

  • Memstore fraction of .4

  • Memstore with 128 Mb RAM

  • 1 column family in table

The formula for this configuration would look as follows:

(16384 Mb * .4) / ((128 Mb * 1) = approximately 51 regions

The easiest way to decrease the number of regions for this example RegionServer is increase the RAM of the memstore to 256 Mb. The reconfigured RegionServer would then have approximately 25 regions, and the HBase cluster will run more smoothly if the reconfiguration is applied to all RegionServers in the cluster. The formula can be used for multiple tables with the same configuration by using the total number of column families in all the tables.

[Note]Note

The formula assumes all regions are filled at approximately the same rate. If a fraction of the cluster's regions are written to, divide the result by this fraction.

If the data request pattern is dominated by write operations rather than read operations, increase the memstore fraction. However, this increase negatively impacts the block cache.