Breaking a riak cluster

transcript

Consistency or BustBreaking a Riak Cluster

Sunday, July 31, 11

Consistency or Bust: Breaking a Riak Cluster

Jeff Kirkellemail: jeff.kirkell@gmail.comtwitter: @jeffkirkell

Sunday, July 31, 11

NoSQL Primer

Sunday, July 31, 11

Riak Sales Pitch Dynamo inspired AP of CAP key / value store scales consistently fault tolerance true high availability built-in replication lot’s of clients simple to implement low $$$ to use multiple license options

Sunday, July 31, 11

Riak Deep Dive

Sunday, July 31, 11

Cluster Defined cluster # nodes 64 / # nodes vnodes

Sunday, July 31, 11

Important Bits gossiping hintful handoff n # copies of data r # nodes that must respond for read w # nodes that must respond for write dw # replicas written on write for success rw # replicas deleted for success

Sunday, July 31, 11

Tools of the Trade terminal curl rebar rekon

Sunday, July 31, 11

The Workshopsetup instructions for cluster

https://github.com/jkirkell/riak_workshop

Sunday, July 31, 11

Start Your Engines$ ./dev/dev1/bin/riak start$ ./dev/dev2/bin/riak start$ ./dev/dev3/bin/riak start$ ./dev/dev4/bin/riak start$ ./dev/dev5/bin/riak start$ps aux | grep erl

Sunday, July 31, 11

$ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1$ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1$ ./dev/dev4/bin/riak-admin join dev1@127.0.0.1$ ./dev/dev5/bin/riak-admin join dev1@127.0.0.1$ ./dev/dev3/bin/riak-admin status

Come Together

Sunday, July 31, 11

Default Settings

quorum = floor(5 / 2 + 1) = 3

Sunday, July 31, 11

Add Data$ curl -v -d 'inserted data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/default

Sunday, July 31, 11

Confirm Data Exists$ curl -i http://127.0.0.1:8091/riak/default?keys=true

Sunday, July 31, 11

Find Your Data$ find ./dev/dev*/data/bitcask/*/*.data

Sunday, July 31, 11

Stop Primary Node$ dev[n]/bin/riak stop

$ curl -i http://127.0.0.1:8091/riak/default/[key]

Sunday, July 31, 11

Stop Another Primary Node$ dev[n]/bin/riak stop

Sunday, July 31, 11

Query Again$ curl -i http://127.0.0.1:8091/riak/default/[key]

Sunday, July 31, 11

Check What’s Happening$find ./dev/dev*/data/bitcask/*/*.data

$ ./dev/dev[n]/bin/riak-admin transfers

Sunday, July 31, 11

Update Data$ curl -v -d 'updated data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/defaults/[key]

Sunday, July 31, 11

Start Node $ dev[n]/bin/riak start

Sunday, July 31, 11

Review Logs$ less ./dev2/log/*.1

$ less ./dev5/log/*.1

Sunday, July 31, 11

Start Other Node$ dev[n]/bin/riak start

Sunday, July 31, 11

Summary consistently fail 1st read with 2 node down 2 node outage max. hintful-handoff takes time default settings good for general use

Sunday, July 31, 11

Clean Your Cluster$ dev1/bin/riak stop$ dev2/bin/riak stop$ dev3/bin/riak stop$ dev4/bin/riak stop$ dev5/bin/riak stop$ cd .. && rm -rf ./dev/*$ make devrel$ dev1/bin/riak start$ dev2/bin/riak start$ dev3/bin/riak start$ dev4/bin/riak start$ dev5/bin/riak start

Sunday, July 31, 11

Low Value Data

n = 2, r = 1, w = 1

Sunday, July 31, 11

Set Bucket Properties$ curl -v -X PUT -H "Content-Type: application/json" -d '{"props":{"n_val":2,"r":1,"w":1}}' http://127.0.0.1:8091/riak/low_value

Sunday, July 31, 11

Add Data$ curl -v -d 'inserted data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/low_value

Sunday, July 31, 11

Confirm Data Exists$ curl -i http://127.0.0.1:8091/riak/low_value?keys=true

Sunday, July 31, 11

Stop Primary Node$ dev[n]/bin/riak stop

Sunday, July 31, 11

Find Your Data Again$ find ./dev/dev*/data/bitcask/*/*.data

Sunday, July 31, 11

Stop Another Primary Node$ dev[n]/bin/riak stop

Sunday, July 31, 11

Query Again$ curl -i http://127.0.0.1:8091/riak/default/[key]

Sunday, July 31, 11

Add More Data$ curl -v -d 'new data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/low_value

Sunday, July 31, 11

Start Down Nodes$ dev[n]/bin/riak start

$ dev[n]/bin/riak start

$ ./dev/dev1/bin/riak-admin transfers

Sunday, July 31, 11

Summary hintful-handoff does not happen * 1 node outage max for reads 4 node outage max for writes great for fire and forget data like logs

Look at the logs for the nodes you brought down. It will show data transfer from hand-off even though the available nodes do not serve the read requests. This may or may not happen randomly due to Eventual Consistency. One other note is that R + W = N cannot guarantee consistency.

Sunday, July 31, 11

Clean Your Cluster$ dev1/bin/riak stop$ dev2/bin/riak stop$ dev3/bin/riak stop$ dev4/bin/riak stop$ dev5/bin/riak stop$ cd .. && rm -rf ./dev/*$ make devrel$ dev1/bin/riak start$ dev2/bin/riak start$ dev3/bin/riak start$ dev4/bin/riak start$ dev5/bin/riak start

Sunday, July 31, 11

Mission Critical Data

n = 5, r = 1, w = 5, dw = 5

Sunday, July 31, 11

Set Bucket Properties

Sunday, July 31, 11

Set Bucket Properties$ curl -v -X PUT -H "Content-Type: application/json" -d '{"props":{"n_val":5,"r":1,"w":5,"dw":5}}' http://127.0.0.1:8091/riak/high_value

Sunday, July 31, 11

Add Data$ curl -v -d 'inserted data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value

Sunday, July 31, 11

Confirm Data Exists$ curl -i http://127.0.0.1:8091/riak/high_value?keys=true

Sunday, July 31, 11

Stop Any 2 Nodes$ dev[n]/bin/riak stop

$ dev[n]/bin/riak stop

$ curl -i http://127.0.0.1:8091/riak/high_value/[key]

Sunday, July 31, 11

Stop Another Node$ dev[n]/bin/riak stop

$ curl -i http://127.0.0.1:8091/riak/high_value/[key]

Sunday, July 31, 11

Update Data$ curl -v -d 'updated data' -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value/[key]

Sunday, July 31, 11

Query Data$ curl -i http://127.0.0.1:8091/riak/high_value/[key]

Sunday, July 31, 11

Start Down Nodes$ dev[n]/bin/riak start

$ dev[n]/bin/riak start

$ ./dev/dev1/bin/riak-admin transfers

Sunday, July 31, 11

Confirm Data Exists$ curl -i http://127.0.0.1:8091/riak/high_value?keys=true

Sunday, July 31, 11

Summary data will duplicate on available nodes to ensure 5 copies 4 node outage max for reads 4 node outage max for writes slower due to all data writes great for high availability data

Sunday, July 31, 11

Total Destruction

Sunday, July 31, 11

Total Destruction

Sunday, July 31, 11

Summary easy to get up and going scales as you need it flexible configuration for consistency sake eventual consistency doesn’t always do what you expect just scratched the surface

Sunday, July 31, 11

Great Reading http://www.basho.com http://bit.ly/eG08u0 http://bit.ly/eFfcjb http://bit.ly/ejCClJ http://slidesha.re/g3hQfg http://bit.ly/n6hWMK

Sunday, July 31, 11

Breaking a riak cluster

Technology