© Hortonworks Inc. 2011

HBase Dev Meetup 09/11/2012

Enis Soztutarenis [@] apache [dot] org#enissoz

Page 1

© Hortonworks Inc. 2011

Integration Testing

• Hbase Integration Tests plan– HBASE-6201

1. Create hbase-it module2. Ability to run integration tests on a given cluster

mvn verifybin/hbase org.apache.hadoop.hbase.IntegrationTestsDriver

3. Port candidate tests 4. Add more long running and fault injection tests5. (Most of) Integration tests should be able to run over a mini cluster or

a 100+ node cluster. 6. Integration tests should run in mini mode as nightlies (apache jenkins)7. Our plan is to run the whole suite of Integration tests nightly/weekly

on medium-sized clusters. Every org. should be able to do this easily.

Page 2Architecting the Future of Big Data

© Hortonworks Inc. 2011

Current Status

• hbase-it module committed, HBASE-6203• HBaseCluster interface for interacting with the cluster from system tests

(HBASE-6241)– Patch near to being committed

• Subtask jiras for porting each test or adding a new test• Hbase book documentation available in patch for HBASE-6302• Candidate tests to be converted:

Page 3Architecting the Future of Big Data

• LoadTestTool• BigTop TestLoadAndVerify• Goraci (https://github.com/keith-

turner/goraci)• TestAcidGuarantees /

TestAtomicOperation• TestRegionBalancing (HBASE-6053)• TestFullLogReconstruction• TestMultiVersions / TestKeepDeletes• TestFromClientSide

• TestShell and src/test/ruby• TestRollingRestart• TestMasterFailover• TestImportExport• Test**OnCluster• Balancer tests

© Hortonworks Inc. 2011

HBASE-6241 Overview

• New junit @Category(IntegrationTests.class)• Introduces HBaseCluster

– public abstract ClusterStatus getClusterStatus();– public abstract void startRegionServer(String hostname);– Start/stop/kill RS / Master– MiniHBaseCluster and DistributedHBaseCluster implements HBaseCluster

• Introduces DistributedHBaseCluster– Keeps state about a distributed cluster – Does not know about environment. Uses ClusterManager to interact with the

cluster• ClusterManager is an Api to manage services

– public abstract void start(ServiceType service, String hostname)– Pluggable, specific implementations for every environment

• HBaseClusterManager extends ClusterManager– Knows about environment (HBASE_HOME, HBASE_CONF_DIR,

HBASE_SSH_OPTS )– Uses SSH and remotely executes ps, kill -9, bin/hbase-daemon.sh start/stop

Page 4Architecting the Future of Big Data

© Hortonworks Inc. 2011

HBASE-6241 Overview (cont.)

• Introduces HBaseIntegrationTestingUtility extends HBaseTestingUtility– Integration tests uses this

• Introduces 2 ways to run integration tests – bin/hbase o.a.h.h.IntegrationTestsDriver –config <hbase_conf_dir>

– mvn verify (runs tests using mini cluster)– mvn verify –Dhbase.test.cluster.distributed=true –Dhbase.conf.dir=<hbase_conf_dir> (further patch)

• Introduces ChaosMonkey– A utility to injects faults in a running cluster.– Does not know about environment, uses HBaseIntegrationTestingUtility– Actions (RestartActiveMaster, RestartRandomRs, RestartRsHoldingMeta,

BatchRestartRs, RollingBatchRestartRs, etc )– Policies (PeriodicRandomActionPolicy), predefined named policies

(EVERY_MINUTE_RANDOM_ACTION_POLICY)• Introduces IntegrationTestDataIngestWithChaosMonkey

– Runs LoadTestTool and ChaosMonkey for configurable running time– Adjusts data load based on the cluster size (also works with mini cluster)

Page 5Architecting the Future of Big Data

© Hortonworks Inc. 2011

Open Issues

•0.94 porting?•Hadoop NN HA testing•BIGTOP-635•Community interest•Different environments (no SSH, etc)•???

Page 6Architecting the Future of Big Data

© Hortonworks Inc. 2011

HBase on Windows

• WHY? –Because we can : )–More platforms = more users + more development

• WHO–MS + HW

• WHEN?–Patches will start rolling shortly

• WHERE?–Apache HBase Trunk–0.94? 0.92?

• WHAT– Hadoop on windows (HADOOP-8079)– Test fixes, file path name related fixes, timing issues, network related changes,

build changes, bin scripts, etc

Page 7Architecting the Future of Big Data

© Hortonworks Inc. 2011

ThanksGo bananas!

Architecting the Future of Big Data Page 8

Taken from http://99designs.com/t-shirt-design/contests/design-chaos-monkey-t-shirt-70909/entries?entriespage=1#contest-header