Introduction to Riak and Ripple (KC.rb)

transcript

Introduction toRiak & Ripple

Sean CribbsDeveloper Advocatebasho

Introduction toRiak & Ripple

Sean CribbsDeveloper Advocate

Rubyist, Evangelist, Support Monkey

Riak @10,000 ft.

Riak @10,000 ft.•Amazon Dynamo-inspired

replicated, distributed, fault-tolerant, masterless, scalable, operations-friendly

•Key-Value / Document

• Schema-less, content-agnostic

•Web-friendlyHTTP, JSON, Javascript

Dynamo-like Scalability

•To get...

•more storage,

•To get...

•more storage,

•more throughput,

•To get...

•more storage,

•more throughput,

•lower latency...

•To get...

•more storage,

•more throughput,

•lower latency...

•...add more machines.aka horizontal & linear

Key-Value++

Key-Value++•Data objects are identified by keys,

and have metadata

•Keys are grouped in buckets

and have metadata

•Links enable lightweight relationships

and have metadata

•Links enable lightweight relationships

•Query with MapReduce

Sans-Schema

Sans-Schema•Buckets created on the fly

•Values are opaque

•Content-Type matters

•The application defines the semantics: more flexibility, more responsibility

Web-Friendly

Web-Friendly•HTTP is primary interface

•JSON is used for structured data

•Javascript is used for MapReduce functions

•Plays well with Varnish, Squid, HAProxy, F5, etc.

•Plays well with Varnish, Squid, HAProxy, F5, etc.[see also Webmachine]

Use Cases

Use Cases•Document storage

(sparse structure)

•Distributed file storage (small size, large soon)

(sparse structure)

•Session storage

(sparse structure)

•Session storage

•Distributed cache

(sparse structure)

•Session storage

•Distributed cache

•Browser-only/Mobile Apps

Getting Started

Getting Started•Download a package from

http://downloads.basho.com

•Set the node name, IPs

•Start up the node

•Join a cluster

•Start storing/retrieving values

Riak in Ruby

Riak in Ruby•Three gems

•riak-client (basic ops)

•ripple (ODM)

•riak-sessions (Rack/Rails session stores)

•ripple (ODM)

•riak-sessions (Rack/Rails session stores)

•All HTTP - Protobuffs coming

require ‘riak’

require ‘riak’• Make a client object

client = Riak::Client.new

• Get a bucketbucket = client.bucket(‘foo’) # Riak::Bucket

• Get an object from the bucketobj = bucket.get(‘bar’) # Riak::RObject

• Initialize a new objectobj = bucket.new(‘baz’)

Riak::RObject

Riak::RObject•Get/set object key

obj.key = “bar”

•Get/set content-typeobj.content_type = ‘application/json’

obj.key = “bar”

•Get/set the object body dataobj.data = {“name” => “Sean”}

obj.key = “bar”

•Get/set the object body dataobj.data = {“name” => “Sean”}

•Store the objectobj.store

More RObject

More RObject•Get object’s bucket

obj.bucket

•Delete the objectobj.delete # freezes the object

obj.bucket

•Detect/extract siblings (more later)obj.conflict? && obj.siblings# Array<RObject>

obj.bucket

•Detect/extract siblings (more later)obj.conflict? && obj.siblings# Array<RObject>

•Follow/traverse links (more later)obj.walk(:tag => “friend”)

Riak::Bucket

Riak::Bucket•Set the replication factor

bucket.n_val = 5

•Set default request quorumsbucket.r = 3 # w,dw,rw # number or one/all/quorum

bucket.n_val = 5

•List keysbucket.keys # pass block to stream

bucket.n_val = 5

•List keysbucket.keys # pass block to stream

•Set consistency flag (allow sibling generation)obj.allow_mult = true

Links: LightweightRelationships

Link Header

Link: </riak/demo/test1>; riaktag=”userinfo”

Link Header

bucket

Link Header

bucket

Link Header

bucket

Links in Ruby API

Links in Ruby API•Create a link

Riak::Link.new(“/riak/bucket/key”, “tag”)

•Read an object’s linksobj.links # Set<Riak::Link>

•Convert an object to a linkobj.links << obj2.to_link(“next”)obj.store

Link-Walking

Link-Walking•Asks Riak to traverse a sequence of

links via special URL

•Filter by bucket/tag

•Can return results from intermediate traversals

•Response is nested multipart/mixed(riak-client handles this for you)

L/W Example

L/W ExampleGET /riak/demo/test1/_,friend,1

Start at demo/test1, follow all links tagged “friend” and return the objects

obj = client[‘demo’][‘test1’]obj.walk(:tag => “friend”,:keep => true)

L/W ExampleGET /riak/demo/test1/_,friend,1

Start at demo/test1, follow all links tagged “friend” and return the objects

obj = client[‘demo’][‘test1’]obj.walk(:tag => “friend”,:keep => true)

L/W Example

L/W ExampleGET /riak/demo/test1/_,_,_/_,_,1

Start at demo/test1, follow all links, follow all links from those, return everything from the last set

obj.walk({},{:keep => true})

Map-Reduce

list of keys

Map-Reduce

list of keys

map map map map map

Map-Reduce

list of keys

map map map map map

reduce

Map-Reduce

list of keys

map map map map map

reduce

results

function(value, keyData, arg) { var object = Riak.mapValuesJson(value)[0]; for(field in arg) { if(object[field] != arg[field]) { return []; } } return [object];}

in Javascript

Riak object

in Javascript

Riak object

{ bucket:”foo”, key:”bar”, vclock:”...”, values:[ {metadata:{...}, data:”....”} ]}

in Javascript

in JavascriptKey-specific

in Javascript

in JavascriptPhase-

specific data

in Javascript

Parse JSON data

in Javascript

Check field equality

in Javascript

Return nothing if unequal

in Javascript

Return the object if all equal

in Javascript

Reduce

function(values,arg) { return values.sort(function(a,b){ return a[arg] - b[arg]; });}

in Javascript

Reduce

in Javascript

Map results

Reduce

in Javascript

Reduce

in Javascript

Phase-specific data

Reduce

in Javascript

Reduce

in Javascript

Custom sort on

Reduce

in Javascript

Reduce

in Javascript

Reduce potentially called multiple times!!

Example Query

Example Query{“inputs”: “goog”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.findHighGreater”, “arg”: 600.0, “keep”: false}, {“reduce”:{“language”:”javascript”, “name”: “Riak.reduceMax”, “keep”: true}]}

Riak::MapReduce.new(c).add(‘goog’).map(‘App.findHighGreater’, :arg => 600.0).reduce(“Riak.reduceMax”, :keep => true).run

Built-in Functions

Built-in Functions•Riak.mapValues

•Riak.mapValuesJson

•Riak.mapByFields

•Riak.reduceSum

•Riak.mapByFields

•Riak.reduceSum

•Riak.reduceSort

•Riak.mapByFields

•Riak.reduceSum

•Riak.reduceSort

•Riak.reduceMin/reduceMax

Ripple - ODM

Document Modelsclass Person include Ripple::Document

property :name, String, :presence => true many :addresses many :friends, :class => Personend

class Address include Ripple::EmbeddedDocument

property :street, Stringend

Rails Setup

# Gemfilegem 'curb' # Faster HTTPgem 'yajl-ruby' # Faster JSONgem 'ripple'

Rails Setup

$ gem install curb yajl-ruby ripple

Rails Setup (cont.)

# config/ripple.ymldevelopment: host: 127.0.0.1 port: 8098

Rails Setup (cont.)

# config/ripple.ymldevelopment: host: 127.0.0.1 port: 8098

# config/application.rbrequire 'ripple/railtie'

Ripple Roadmap

Ripple Roadmap•Testing server (edge on Github)

•Protocol Buffers support

•Streaming MapReduce

•Ripple-specific built-ins

•Better ActionView support (form_for)

Happy FunDemo Time

PlugInterested in learning about support, consulting, or Enterprise features?

PlugInterested in learning about support, consulting, or Enterprise features?Email info@basho.com or go to http://www.basho.com/contact.html to talk with us.

www.basho.com

sean@basho.com@seancribbs

Questions

Introduction to Riak and Ripple (KC.rb)

Technology