Top five questions to ask when choosing a big data solution

transcript

Five factors to consider when choosing a big data solution!Jonathan EllisCTO, DataStaxProject Chair, Apache Cassandra

how do I

modelmy application?

Popular options• Key/value

• Tabular

• Document

• Graph?

Schema is your friend

{ "id": "e451dd42-ece3-11e1-a0a3-34159e154f4c", "name": "jbellis", "state": "TX", "birthdate": "1/1/1976", "email_addresses": ["jbellis@gmail", "jbellis@datastax.com"],}

SQL can be your friend too

CREATE TABLE users ( id uuid PRIMARY KEY, name text, state text, birth_date date);

CREATE INDEX ON users(state);

SELECT * FROM usersWHERE state=‘Texas’ AND birth_date > ‘1950-01-01’;

CREATE TABLE users_addresses ( user_id uuid REFERENCES users, email text);

SELECT *FROM users NATURAL JOIN users_addresses;

Collections

CREATE TABLE users_addresses ( user_id uuid REFERENCES users, email text);

SELECT *FROM users NATURAL JOIN users_addresses;

Collections

CREATE TABLE users ( id uuid PRIMARY KEY, name text, state text, birth_date date, email_addresses set<text>);

UPDATE usersSET email_addresses = email_addresses + {‘jbellis@gmail.com’, ‘jbellis@datastax.com’};

Collections

Joins don’t scale• No joins

• No subqueries

• No aggregation functions* or GROUP BY

• ORDER BY?

SELECT * FROM tweetsWHERE user_id IN (SELECT follower FROM followers WHERE user_id = ’driftx’)

followers

tweets

CREATE TABLE timeline ( user_id uuid, tweet_id timeuuid, tweet_author uuid, tweet_body text, PRIMARY KEY (user_id, tweet_id));

Clustering in Cassandrauser_id tweet_id _author _body

jbellis 3290f9da.. rbranson loremjbellis 3895411a.. tjake ipsum

... ... ...

driftx 3290f9da.. rbranson loremdriftx 71b46a84.. yzhang dolor

... ... ...

yukim 3290f9da.. rbranson loremyukim e451dd42.. tjake amet

... ... ...

CREATE TABLE timeline ( user_id uuid, tweet_id timeuuid, tweet_author uuid, tweet_body text, PRIMARY KEY (user_id, tweet_id));

Clustering in Cassandrauser_id tweet_id _author _body

jbellis 3290f9da.. rbranson loremjbellis 3895411a.. tjake ipsum

... ... ...

driftx 3290f9da.. rbranson loremdriftx 71b46a84.. yzhang dolor

... ... ...

yukim 3290f9da.. rbranson loremyukim e451dd42.. tjake amet

... ... ...

SELECT * FROM timelineWHERE user_id = ’driftx’;

how does it

perform?

Larger than memory datasets

Locking

Efficiency

UPDATE usersSET email_addresses = email_addresses + {...}WHERE user_id = ‘jbellis’;

Durability

C* storage engine very briefly

Memory

Hard drive

Memtable

write( , )k1 c1:v1

Commit log

Memory

Hard drive

Memtable

write( , )k1 c1:v1

Commit log

k1 c1:v1

Memory

Hard drive

write( , )k1 c2:v2

k1 c1:v1

k1 c2:v2

Memory

Hard drive

k1 c1:v1

k1 c2:v2

write( , )k2 c1:v1 c2:v2

k2 c1:v1 c2:v2

Memory

Hard drive

k1 c1:v1

k1 c1:v4

k1 c2:v2

write( , )k1 c1:v4 c3:v3

k2 c1:v1 c2:v2

k1 c1:v4 c3:v3

Memory

Hard drive

SSTable

k1 c1:v4 c2:v2

k2 c1:v1 c2:v2

cleanup

No random writes

Cassandra 0.6

Cassandra 1.0

reads/s writes/s

how does it handle

failure?

Classic partitioning with SPOFpartition 1 partition 2 partition 3 partition 4

router

client

Availability• “High availability implies that a single fault will not bring

down your system. Not ‘we’ll recover quickly.’” -- Ben Coverston: DataStax

• “The biggest problem with failover is that you're almost never using it until it really hurts. It's like backups that you never test.” -- Rick Branson: Instagram

Fully distributed, no SPOFclient

Multiple datacenters

how does it

scale?

Scaling antipatterns• Metadata servers

• Router bottlenecks

• Overloading existing nodes when adding capacity

flexibleis it?

Data model: Realtime

Portfolios

StockHist

stock lastGOOG $95.52AAPL $186.10AMZN $112.98

LiveStocks

stock date priceGOOG 2011-01-01 $8.23GOOG 2011-01-02 $6.14GOOG 2011-001-03 $7.78

user stock sharesjbellis GOOG 80jbellis LNKD 20yukim AMZN 100

Data model: Analytics

worst_date loss2011-07-23 -$34.812011-03-11 -$11432.242011-05-21 -$1476.93

Portfolio1

HistLoss

Portfolio2Portfolio3

Data model: Analyticsstock rdate returnGOOG 2011-07-25 $8.23GOOG 2011-07-24 $6.14GOOG 2011-07-23 $7.78AAPL 2011-07-25 $15.32AAPL 2011-07-24 $12.68

10dayreturns

INSERT OVERWRITE TABLE 10dayreturnsSELECT a.stock, b.date as rdate, b.price - a.priceFROM StockHist a JOIN StockHist b ON (a.stock = b.stock AND date_add(a.date, 10) = b.date);

portfolio rdate preturnPortfolio1 2011-07-25 $118.21Portfolio1 2011-07-24 $60.78Portfolio1 2011-07-23 -$34.81Portfolio2 2011-07-25 $2143.92Portfolio3 2011-07-24 -$10.19

portfolio_returns

INSERT OVERWRITE TABLE portfolio_returnsSELECT portfolio, rdate, SUM(b.return)FROM portfolios a JOIN 10dayreturns b ON (a.stock = b.stock)GROUP BY portfolio, rdate;

INSERT OVERWRITE TABLE HistLossSELECT a.portfolio, rdate, minpFROM ( SELECT portfolio, min(preturn) as minp FROM portfolio_returns GROUP BY portfolio) a JOIN portfolio_returns b ON (a.portfolio = b.portfolio and a.minp = b.preturn);

worst_date loss2011-07-23 -$34.812011-03-11 -$11432.242011-05-21 -$1476.93

Portfolio1

HistLoss

Portfolio2Portfolio3

Some Cassandra users

Questions?

• http://www.!ickr.com/photos/26817893@N05/2573006312/

• http://www.!ickr.com/photos/rowanbank/7686239548

• http://www.!ickr.com/photos/mervtheswerve/6081933265

• http://www.!ickr.com/photos/dg_pics/2526208830

• http://www.!ickr.com/photos/wainwright/351684037

• http://www.!ickr.com/photos/mikeneilson/1606662529

• http://www.!ickr.com/photos/sbisson/3852905534

• http://www.!ickr.com/photos/breadnbadger/2674928517

Image credits

Top five questions to ask when choosing a big data solution

Technology