Automated heap memory tuner

for HBase

HBase is a distributed, fault-tolerant, column-oriented key value store based on Google BigTable and built on top of Hadoop HDFS. HBase is widely used at FB. This memory

tuner has been implemented in open-source Apache HBase.

What is HBase ?

Basic Architecture

As with any database, optimized I/O is a critical concern to HBase. When possible, the priority is to not perform any I/O at all. This means that memory utilization and caching structures are of utmost importance. To this end, HBase maintains two cache structures: the “memory store” and the “block cache”.

JVM Heap

When something is written to HBase, it is first written to an in-memory store (memstore), once this memstore reaches a certain size, it is flushed to disk into a store file (everything is also written immediately to a log file for durability). The store files created on disk are immutable.

Memstore

Block cache keeps data blocks resident in memory after they’re read.

Servicing reads from the Block Cache is the primary mechanism through which HBase is able to serve random reads with millisecond latency.

Block Cache

In simple terms we should give more memory to memstore if server is write heavy and we should give more memory to block cache when it is read heavy.

But unfortunately this distribution is hard coded and is 40-40 by default. Which leads to wastage of servers memory.

Current distribution and the problem

Implemented a default Heap Memory Tuner.

There is a configurable time period,after this time interval tuner will regularly check the condition and decide in which direction to shift the memory.

Solution

● Number of flushes● Number of cache misses● Number of evictions● Current sizes of memstore and block cache● Their current max allowed limit ● All the tuner decisions over past few periods

Parameters used

● Warming up period● Feedback based● Tuner direction● Lookup over past periods stats● Varying step size

Tuning process overview

● min/max range on memstore size● min/max range on block cache size● Tuner period ● Lookup periods● Warming up period

Configurable Parameters

There are 13 unit tests that check that tuner works as expected in some common workload scenarios.

They thoroughly test that tuner is on only when it is supposed to.

Unit tests

● Read Only● Write Only● Read heavy● Read heavy with small working set size● 50-50 workload

Testing on actual workloads

Test result from cluster for write heavy job followed by read heavy job.

Two write heavy jobs running One write heavy job runningOne write heavy job + one read heavy job

20 initial tuner periods were ignored.

One read heavy job running

Insert throughput/latency comparison on write heavy job.

What does it improve ?

Default Tuned

Avg. Throughput 12704.01 ops 13706.87 ops

Avg. Latency 2354.81 us 2183.29 us

99th Percentile Latency 3 ms 2 ms

QUESTIONS ? ?

THANK YOU ! !