Introduction of HBase Reporter: Hu Yi 2009-3-11. Overview HBase is an Apache open source project...

transcript

Introduction of HBase

Reporter: Hu Yi

2009-3-11

Overview

HBase is an Apache open source project whose goal is to provide storage for the Hadoop Distributed Computing Environment.

Data is logically organized into tables, rows and columns.

Outline

Data Model Architecture and Implementation Examples & Tests

Conceptual View

A data row has a sortable row key and an arbitrary number of columns.

A Time Stamp is designated automatically if not artificially.

Row keyTime

Column“contents:

”Column “anchor:”

“com.apache.www”

t12 “<html>…”

t11 “<html>…”

t10“anchor:apache.

com”“APACHE

“com.cnn.www”

t15“anchor:cnnsi.co

m”“CNN”

t13“anchor:my.look.c

a”“CNN.co

t6 “<html>…”

t5 “<html>…”

t3 “<html>…”

Physical Storage View

Physically, tables are stored on a per-column family basis.

Empty cells are not stored in a column-oriented storage format.

Each column family is managed by an HStore.

Row key TSColumn

“contents:”

t12 “<html>…”

t11 “<html>…”

“com.cn.www”

t6 “<html>…”

t5 “<html>…”

t3 “<html>…”

Row key TS Column “anchor:”

“com.apache.www” t10

“anchor:apache.com”

“APACHE”

com.cn.www”

t9“anchor:

cnnsi.com”“CNN”

t8“anchor:

my.look.ca”“CNN.co

HStore

Data MapFile

Index MapFile

Key/Value

Index key

HStore

Memcache

Row Ranges: Regions

Row key/ Column ascending, Timestamp descending

Physically, tables are broken into row ranges contain rows from start-key to end-key

Row keyTime

StampColumn

“contents:”Column “anchor:”

t15 anchor:cc value

t13 ba

t12 bb

t11 anchor:cd value

t10 bc

aaab t14

aaac anchor:be value

aaad anchor:ad value

aaaet5 ae

Outline

Three major components

The HBaseMaster

The HRegionServer

The HBase client

HBaseMaster

Assign regions to HRegionServers.

1. ROOT region locates all the META regions.

2. META region maps a number of user regions.

3. Assign user regions to the HRegionServers.

Enable/Disable table and change table schema

Monitor the health of each Server

ROOT Regi on

META Regi on

USER Regi on

ROOT/META Table

Each row in the ROOT and META tables is approximately 1KB in size. At the default size of 256MB.

ROOTtable METAregions

USERregions

KB bytes

HRegionServer

Write Requests Read Requests Cache Flushes Compactions Region Splits

Hstore1 Hstore2

Memcache1

Row keyTimeStam

Column“contents

:”Column “anchor:”

“com.apache.ww

t12“<html>…”

t11“<html>…”

com”“APACH

“com.cnn.www”

m”“CNN”

t8“anchor:my.look.

ca”“CNN.co

t6“<html>

…”

t5“<html>

…”

t3“<html>

…”

Memcache2

Mapfile1.1

Mapfile1.2

HRegionServer

Hstore1

Memcache1

Mapfile1.1

Mapfile1.2

Row keyTimeStam

Column“contents:

“com.apache.ww

t12“<html>…”

t11“<html>…”

com”“APACHE

“com.cnn.www”

m”“CNN”

ca”“CNN.co

t6“<html>

…”

t5“<html>

…”

t3“<html>

…”

HRegionServer

Cache Flushes

Hstore1

Memcache1

Mapfile1.1

Mapfile1.2

Row keyTimeStam

Column“contents:

“com.apache.ww

t12“<html>…”

t11“<html>…”

com”“APACHE

“com.cnn.www”

m”“CNN”

ca”“CNN.co

t6“<html>

…”

t5“<html>

…”

t3“<html>

…”

Mapfile1.1

Mapfile1.2

Mapfile1.3

HRegionServer

Compactions

Hstore1

Memcache1

Mapfile1.1

Mapfile1.2Mapfile1

Row keyTimeStam

Column“contents:

“com.apache.ww

t12“<html>…”

t11“<html>…”

com”“APACHE

“com.cnn.www”

m”“CNN”

ca”“CNN.co

t6“<html>

…”

t5“<html>

…”

t3“<html>

…”

HRegionServer

Region Splits

Hstore1

Memcache1

Mapfile1

Row keyTimeStam

Column“contents

:”Column “anchor:”

“com.apache.ww

t12“<html>…”

t11“<html>…”

com”“APACH

“com.cnn.www”

m”“CNN”

ca”“CNN.co

t6“<html>

…”

t5“<html>

…”

t3“<html>

…”

HBase Client

HBase Client ROOT Region

HBase Client

META Region

HBase Client User Region

Information cached

Outline

Create MyTable

HBaseAdmin admin= new HBaseAdmin(config);HColumnDescriptor []column;column= new HColumnDescriptor[2];column[0]=new HColumnDescriptor("columnFamily1:");column[1]=new HColumnDescriptor("columnFamily2:");HTableDescriptor desc= new HTableDescriptor(Bytes.toByt

es("MyTable"));desc.addFamily(column[0]);desc.addFamily(column[1]);admin.createTable(desc);

Row Key

Timestamp

columnFamily1:

columnFamily2:

Insert Values

BatchUpdate batchUpdate = new BatchUpdate("myRow",timestamp);

batchUpdate.put("columnFamily1:labela",Bytes.toBytes("labela value"));

batchUpdate.put("columnFamily1:labelb",Bytes.toBytes(“labelb value"));

table.commit(batchUpdate);

Row Key

Timestamp columnFamily1:

myRow　

ts1 labela labela value

ts2labelb

　 labelb value

I nsert

100000

120000

140000

160000

100000 10000 1000 100 10 1

1 10 100 1000 10000 100000

Insert

100000

1000000

10 100

Row*10 Column=1

HbaseMySQL

Search

Row keyTime

StampColumn “anchor:”

t10 “anchor:apache.com” “APACHE”

“com.cnn.www”

t9 “anchor:cnnsi.com” “CNN”

t8 “anchor:my.look.ca” “CNN.com”

Select value from table where key=‘com.apache.www’ AND label=‘anchor:apache.com’

Search ScannerSelect value from table where anchor=‘cnnsi.com’

Row keyTime

StampColumn “anchor:”

t10 “anchor:apache.com” “APACHE”

“com.cnn.www”

t9 “anchor:cnnsi.com” “CNN”

t8 “anchor:my.look.ca” “CNN.com”

Summary

Column-oriented modification more flexible.

Higher performance on row key clusters.

Future work

More test work

Optimization on search

Thank you

Introduction of HBase Reporter: Hu Yi 2009-3-11. Overview HBase is an Apache open source project...

Documents