Using Apache HBase to store and access data
Also available as:
PDF
loading table of contents...

Prerequisites to configure Off-Heap Memory (BucketCache)

Note
Note

Before configuring off-heap memory, complete the tasks in the previous "Configuring BlockCache" section.

To prepare for BucketCache configuration, compare the figure and table below before proceeding to the "Configuring BucketCache" steps.

Figure 1. Diagram of Configuring BucketCache


In the following table:

  • The first column refers to the elements in the figure.

  • The second column describes each element and, if applicable, its associated variable or property name.

  • The third column contains values and formulas.

  • The fourth column computes values based on the following sample configuration parameters:

    • 128 GB for the RegionServer process (there is additional memory available for other HDP processes)

    • A workload of 75% reads, 25% writes

    • HBASE_HEAPSIZE = 20 GB (20480 MB)

Note
Note

Most of the following values are specified in megabytes; three are proportions.

Item

Description

Value or Formula

Example

A

Total physical memory for RegionServer operations: on-heap plus off-heap ("direct") memory (MB)

(hardware dependent)

131072

B

The HBASE_HEAPSIZE (-Xmx) property: Maximum size of JVM heap (MB)

This value was set when the BlockCache was configured.

Recommendation: 20480

20480

C

The -XX: MaxDirectMemorySize option: Amount of off-heap ("direct") memory to allocate to HBase (MB)

A - B

131072 - 20480 = 110592

Dp

The hfile.block.cache.size property: Proportion of maximum JVM heap size (HBASE_HEAPSIZE, -Xmx) to allocate to BlockCache. The sum of this value plus the hbase.regionserver. global.memstore.size must not exceed 0.8.

This value was set when the BlockCache was configured.

(proportion of reads) * 0.8

0.75 * 0.8 = 0.6

Dm

Maximum amount of JVM heap to allocate to BlockCache (MB)

B * Dp

20480 * 0.6 = 12288

Ep

The hbase.regionserver. global.memstore.size property: Proportion of maximum JVM heap size (HBASE_HEAPSIZE, -Xmx) to allocate to MemStore. The sum of this value plus hfile.block.cache.size must be less than or equal to 0.8.

0.8 - Dp

0.8 - 0.6 = 0.2

F

Amount of off-heap memory to reserve for other uses (DFSClient; MB)

Recommendation: 1024 to 2048

2048

G

Amount of off-heap memory to allocate to BucketCache (MB)

C - F

110592 - 2048 = 108544

The hbase.bucketcache.size property: Total amount of memory to allocate to the off-heap BucketCache (MB)

G

108544