What is HBase?
HBase is an open source, distributed, sorted map modeled after Google’s Big Table
NoSQL solution built atop Apache Hadoop
Top level Apache Project
CAP Theorem
Consistency (all nodes see the same data at the same time)
Availability (a guarantee that every request receives a response about whether it was successful or failed)
Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system)
According to the theorem, a distributed system can satisfy any two of these guarantees at the same time, but not all three.
Usage Scenarios
Lots of Data - 100s of Gigs to Petabytes
High Throughput – 1000’s of records/sec
Scalable cache capacity – Adding several nodes adds to available cache
Data Layout – Excels at key lookup and no penalty for sparse columns
Column Oriented Databases
HBase belongs to family of databases called as Column-oriented
Column-oriented databases save their data grouped by columns.
The reason to store values on a per-column basis instead is based on the assumption that, for specific queries, not all of the values are needed.
Reduced I/O is one of the primary reasons for this new layout
Specialized algorithms—for example, delta and/or prefix compression—selected based on the type of the column (i.e., on the data stored) can yield huge improvements in compression ratios. Better ratios result in more efficient bandwidth usage.
HBase as a Column Oriented Database
HBase is not a column-oriented database in the typical RDBMS sense, but utilizes an on-disk column storage format.
This is also where the majority of similarities end, because although HBase stores data on disk in a column-oriented format, it is distinctly different from traditional columnar databases.
Whereas columnar databases excel at providing real-time analytical access to data, HBase excels at providing key-based access to a specific cell of data, or a sequential range of cells.
HBase and Hadoop
Hadoop excels at storing data of arbitrary, semi-, or even unstructured formats, since it lets you decide how to interpret the data at analysis time, allowing you to change the way you classify the data at any time: once you have updated the algorithms, you simply run the analysis again.
HBase sits atop Hadoop using all the best features of HDFS such as scalability and data replication
HBase UnUsage
When data access patterns are unknown – HBase follows a data centric model rather than relationship centric, Hence it does not make sense doing an ERD model for HBase
Small amounts of data – Just use an RDBMS
Limited/No random reads and writes – Just use HDFS directly
HBase Use Cases - Facebook
One of the earliest and largest users of HBase
Facebook messaging platform built atop HBase in 2010
Chosen because of the high write throughput and low latency random reads
Other features such as Horizontal Scalability, Strong Consistency and High Availability via Automatic Failover.
HBase Use Cases - Facebook
In addition to online transaction processing workloads like messages, it is also used for online analytic processing workloads where large data scans are prevalent.
Also used in production by other Facebook services, including the internal monitoring system, the recently launched Nearby Friends feature, search indexing, streaming data analysis, and data scraping for their internal data warehouses.
Seek vs. Transfer
One of the fundamental differences between typical RDBMS and nosql ones is the use of B or B+ trees and Log Structured Merge Trees(LSM) which was the basis of Google’s BigTable
B+ Trees
B+ Trees allow for efficient insertion, lookup and deletion of records that are identified by keys.
Represent dynamic, multilevel indexes with lower and upper bounds per segment or page
This allows for higher fanout compared to binary trees resulting in lower number of I/O operations
Range scans are also very efficient
LSM Trees
Incoming data first stored in logfile, completely sequentially
Once the log has modification saved, updates an in-memory store.
Once enough updates are accrued in the in-memory store, it flushes a sorted list of key->record pairs to disks creating store files.
At this point all updates to log can be deleted since modifications have been persisted.
Fundamental Difference
Disk Drives
Too Many Modifications force costly optimizations.
More Data at random locations cause faster fragmentation
Updates and deletes are done at disk seek rates rather than disk transfer rates
Fundamental Difference(Contd)
Works at disk transfer rates
Scales better to handle large amounts of data.
Guarantees consistent insert rate
Transform random writes into sequential writes using logfiles plus in-memory store
Reads independent from writes so no contention between the two
HBase Basics
When data is added to HBase, it is first written to the WAL(Write ahead log) called HLog.
Once the write is done, it is then written to an in memory called MemStore
Once the memory exceeds a certain threshold, it flushes to disk as an HFile
Over time HBase merges smaller HFiles into larger ones. This process is called compaction
Ref: https://www.altamiracorp.com/blog/employee-posts/handling-big-data-with-hbase-part-3-architecture-overview
Facebook-Hydrabase
In HBase, when a regionserver fails, all regions hosted by that regionserver are moved to another regionserver.
Depending on HBase has been setup, this typically entails splitting and replaying the WAL files which could take time and lengthens the failover
Hydrabase differs from HBase in this. Instead of having a region by a single region server, it is hosted by a set of regionservers.
When a regionserver fails, there are standby regionservers ready to take over
Facebook-Hydrabase
The standby region servers can be spread across different racks or even data centers, providing availability.
The set of region servers serving each region form a quorum. Each quorum has a leader that services read and write requests from the client.
HydraBase uses the RAFT consensus protocol to ensure consistency across the quorum.
With a quorum of 2F+1, HydraBase can tolerate up to F failures.
Increases reliability from 99.99% to 99.999% ~ 5 mins downtime/year.
HBase Users - Flurry
Mobile analytics, monetization and advertising company founded in 2005
Recently acquired by Yahoo
2 data centers with 2 clusters each, bi directional replication
1000 slave nodes per cluster – 32 GB RAM, 4 drives(1 or 2 TB), 1 Gig E, Dual Quad Core processors *2 HT = 16 procs
~30 tables, 250k regions, 430TB(after LZO compression)
2 big tables are approx 90% of that, 1 wide table with 3 CF, 4 billion rows with 1 MM cells per row. The other tall table with 1 CF, 1 trillion rows and 1 cell per row
HBase Security – 0.98
Cell Tags – All values in HBase are now written in cells, can also carry arbitrary no. of tags such as metadata
Cell ACLs – enables the checking of (R)ead, (W)rite, E(X)excute, (A)dmin & (C)reate
Cell Labels – Visibility expression support via new security coprocessor
Transparent Encryption – data is encrypted on disk – HFiles are encrypted when written and decrypted when read
RBAC – Uses Hadoop Group Mapping Service and ACL’s to implement
Apache Phoenix
SQL layer atop HBase – Has a query engine, metadata repository & embedded JDBC driver, top level apache project, currently only for HBase
Fastest way to access HBase data – HBase specific push down, compiles queries into native, direct HBase calls(no map-reduce), executes scans in parallel
Integrates with Pig, Flume & Sqoop
Phoenix maps HBase data model to relational world
Open TSDB 2.0
Distributed, Scalable Time Series Database on top of HBase
Time Series – data points for identity over time.
Stores trillions of data points, never loses precision, scales using HBase
Good for system monitoring & measurement – servers & networks, Sensor data – The internet of things, SCADA, Financial data, Results of Scientific experiments, etc.
Open TSDB 2.0
Users – OVH(3rd largest cloud/hosting provider) to monitor everything from networking, temperature, voltage to resource utilization, etc.
Yahoo uses it to monitor application performance & statistics
Arista networking uses it for high performance networking
Other users such as Pinterest, Ebay, Box, etc.
Apache Slider(Incubator)
YARN application to deploy existing distributed applications on YARN, monitor them and make them larger or smaller as desired -even while the application is running.
Incubator Apache Project; Similar to Tez for Hive/Pig
Applications can be stopped, "frozen" and restarted, "thawed" later; It allows users to create and run multiple instances of applications, even with different application versions if needed
Applications such as HBase, Accumulo & Storm can run atop it