Why relationships are cool but join sucks - Big Data & Graphs in Rome

transcript

(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License Page 1www.orientechnologies.com

Luca Garulli – Founder and CEO@Orient Technologies LtdAuthor of OrientDB

www.twitter.com/lgarulli

Why Relationships

are coolbut the “JOIN”

BigData & GraphsIn Rome

(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License Page 2

1979First Relational DBMS available as product

2009NoSQL movement

1979First Relational DBMS available as product

2009NoSQL movement

Hey, 30 years in the IT field is so huge!

Before 2009 teams of developersalways fought to select:

Operative SystemProgramming Language

Middleware (App-Servers)

What about the Database?

One of the main resistances ofRDBMS users to pass to a NoSQL product

are related to thecomplexity of the model:

Ok, NoSQL products are super forBigData and BigScale

but...

...what about the model?

What is the NoSQL answer about managing complex domains?

Key-Value stores ?Column-Based ?

Document database ?Graph database !

No Relationshipssupport

Why most of NoSQL

productsdon’t supportRelationship

Between entities?

To understand why,let’s see how

Relational DBMSmanaged them

Domain: the super minimal “Selling App”

Customer Address

Order Stock

Registry system

Order system

Registry system

Domain: the super minimal “Selling App”

Order system

Customer Address

How doesRelational DBMS

manage relationships?

Relational World: 1-1 Relationships

JOIN Customer.Address -> Address.Id

CustomerId Name Address

10 Luca 34

11 Jill 44

34 John 54

56 Mark 66

88 Steve 68

AddressId Location

34 Rome

44 London

54 Moscow

66 New Mexico

68 Palo Alto

Foreign key

Primary keyPrimary key

Relational World: 1-N Relationships

Inverse JOIN Address.Customer -> Customer.Id

CustomerId Name

10 Luca

11 Jill

34 John

56 Mark

88 Steve

AddressId Customer Location

24 10 Rome

33 10 London

44 34 Moscow

66 56 Cologne

68 88 Palo Alto

Relational World: N-M Relationships

Additional table with 2 JOINs(1) CustomerAddress.Id -> Customer.Id and(2) CustomerAddress.Address -> Address.Id

CustomerId Name

10 Luca

11 Jill

34 John

56 Mark

88 Steve

AddressId Location

24 Rome

33 London

44 Moscow

66 Cologne

68 Palo Alto

CustomerAddressId Address

What’s wrong with theRelational Model?

These are all JOINs executedeverytime you traverse a

relationship

The JOIN is the evil!Customer

Id Name

10 Luca

11 Jill

34 John

56 Mark

88 Steve

AddressId Location

24 Rome

33 London

44 Moscow

66 Cologne

68 Palo Alto

relationship

relationship!

CustomerAddressId Address

A JOIN means searching for a key inanother table

The first rule to improve performanceis indexing all the keys

Index speeds up searches, but slows downinsert, updates and deletes

So in the best case a JOIN is a lookupinto an index

This is done per single join!

If you traverse hundreds of relationshipsyou’re executing hundreds of JOINs

Index Lookupis it really that fast?

Index Lookup: how does it works?A-Z

A-L M-Z

Think to an Address Book

where we have to find the Luca’s phone

number

A-L M-Z

A-D E-L

M-R S-Z

Index algorithms are all similar and based on

balanced trees

A-L M-Z

A-D E-L

M-R S-Z

A-B C-D

E-G H-L

A-L M-Z

A-D E-L

M-R S-Z

A-B C-D

E-G H-L

H-J K-L

A-L M-Z

A-D E-L

M-R S-Z

A-B C-D

E-G H-L

H-J K-L

Found! This lookup took 5 steps and grows up with the index

Can you imaginehow many steps a

Lookup operation does into anIndex with Millions or Billions

of records?

And this JOIN is executedforeach involved table,

multiplied foreach scanned records

Querying more tables can easilyproduce millions of JOINs/Lookups!

Here the rule: more entries= more lookup steps = slower JOIN

Oh! This is whyperformance of my database

drops down whenit becomes bigger,

and bigger,and bigger!

What aboutDocument Databases

like MongoDB?

How MongoDB manages relationships:

{ “_id” : “292846512”, “type” : “Order”, “number” : 1223, “customer” : “123456789”}

MongoDB uses the same approach:

it stores the _id of the connecteddocuments. At run-time it lookups upfor the _id by using an index.

Is there a better way tomanage relationships?

“A graph database is anystorage system

that providesindex-free adjacency”

- Marko Rodriguez (author of TinkerPop Blueprints)

How does GraphDB manageindex-free relationships?

Every developer knowsthe Relational Model,but who knows the

Graph one?

Back to school:Graph Theory crash course

Basic Graph

Luca NoSQLDay

Property Graph Model*

Lucaname: Luca

surname: Garullicompany: Orient Tech

NoSQLDay

date: Nov 15° 2013

since: 2013

Vertices and Edges can have propertiesVertices and Edges can have propertiesVertices and Edges can have properties

Vertices are directed

* https://github.com/tinkerpop/blueprints/wiki/Property-Graph-Model

Property Graph Model

Luca NoSQLDay

Likessince: 2013

Speakstitle: «Switching...»

abstract: «This talk presents...»An Edge connects 2

vertices: use multiple edges to represents 1-N and N-M

relationships

Property Graph Model

Daniel

Organizes

FriendOfNoSQL

located

Studies

Compliments, this is your diploma in«Graph Theory»

The Graph theoryis so simple to be so

powerful

Let’s go backto the Graph Stuff

How does OrientDBmanage relationships?

Luca(vertex)

OrientDB: traverse a relationship

label : ‘Customer’name : ‘Luca’

RID = #13:35 RID = #13:100

label = ‘Address’name = ‘Rome’

The Record ID (RID)is the physical position

Rome(vertex)

The Record ID (RID)is the physical position

OrientDB: traverse a relationship

out : [#14:54]label : ‘Customer’name : ‘Luca’

out: [#13:35]in: [#13:100]Label : ‘Lives’

RID = #13:35 RID = #13:100

in: [#14:54]label = ‘Address’name = ‘Rome’

The Edge’s RID is saved inside both vertices, as

«out» and «in»

The Edge’s RID is saved inside both vertices, as

«out» and «in»

RID = #14:54

Luca(vertex)

Rome(vertex)

LucaLives

OrientDB: traverse -> outgoing

RID = #13:35RID = #14:54

RID = #13:100

LucaLives

OrientDB: traverse <- incoming

RID = #13:35RID = #14:54

RID = #13:100

GraphDB handles relationships as aphysical LINK to the record

assigned when the edge is created

on the other side

RDBMS computes therelationship every time you query a database

Is not that crazy?!

This means jumping from aO(log N) algorithm to a near O(1)

traversing cost is not more affectedby database size!

This is huge in the BigData age

an Open Source (Apache licensed)document-graph NoSQL dbms

OrientDB in the Blueprints micro-benchmark,on common hw, with a hot cache,

traverses 29,6 Millionsof records in less than 5 seconds

about 6 Millions of nodes traversed per sec!

*unless you live in the Google’s server farm

Do not try this at home with a RDBMS*!

Create the graph in SQL$luca> cd bin$luca> ./console.shOrientDB console v.1.6.1 (www.orientdb.org) Type 'help' to display all the commands supported.

orientdb> create vertex Customer set name = ‘Luca’Created vertex #13:35 in 0.03 secs

orientdb> create vertex Address set name = ‘Rome’Created vertex #13:100 in 0.02 secs

orientdb> create edge Lives from #13:35 to #13:100Created edge #14:54 in 0.02 secs

Create the graph in JavaGraph graph = new OrientGraph("local:/tmp/db/graph”);

Vertex luca = graph.addVertex( “class:Customer” );luca.setProperty( “name", “Luca” );

Vertex rome = graph.addVertex ( “class:Address” );rome.setProperty( “name", “Rome” );

Edge edge = luca.addEdge( “Lives”, rome );

graph.shutdown();

Query the graph in SQL

orientdb> select in(‘Lives’) from Address where name = ‘Rome’---+------+---------|--------------------+--------------------+--------+ #| RID |@class |label |out_Lives |in |---+------+---------+--------------------+--------------------+--------+ 0| 13:35|Customer |Luca |[#14:54] | |---+------+---------+--------------------+--------------------+--------+1 item(s) found. Query executed in 0.007 sec(s).

Incoming vertices

More on query powerorientdb> select sum( out(‘Order’).total ) from Customer where name = ‘Luca’

orientdb> traverse both(‘Friend’) from Customer while $depth <= 7

orientdb> select from ( traverse both(‘Friend’) from Customer while $depth <= 7 ) where @class=‘Customer’ and city.name = ‘Udine’

Query vs traversalOnce you’ve a well connected database

in the form of a Super Graph you cancross records instead of query them!

All you need is a few“Root Vertices”where to start traversing

Query vs traversalCustomers

Luca Mark Jill

Order2332

Order8834

WhiteSoap

StocksSpecialCustomers

This is aroot

vertex

Root Vertices can be enriched byMeta Graphs

to decorate Graphs withadditional informationand make easier/faster

the retrieval

Temporal based Meta Graph

Order2333

Order2334

Calendar

Hour9/4/2013

Order2332

Day9/4/2013

MonthApril 2013

Year2013

Location based Meta Graph

Order2333

Order2334

Location

CityRome

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Mix & Merge graphs

Order2333

Order2334

Location

CityRome

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Order2333

Order2334

Location

CityRome

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Get all the orderssold in “Fiumicino” cityon 9/4/2013 at 10:00

Start from Calendar, look for Hour 10:00

Order2333

Order2334

Location

CityRome

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Order2333

Order2334

Location

CityRome

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Found 2 Orders, filter by incoming

edges<

Found 2 Orders, now filter by

incoming edges

Order2333

Location

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Order2334

Only “Order 2333” has incoming

connections with “Fiumicino”

CityRome

Order2333

Location

CityFiumicino

Order2332

StateRM

RegionLazio

CountryItaly

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Order2334

CityRome

Or start from Location, look for Fiumicino

Order2333

Order2332

Calendar

Hour9/4/2013

Day9/4/2013

MonthApril 2013

Year2013

Order2334

Start from Location, look for Fiumicino

Location

CityRome

CityFiumicino

StateRM

RegionLazio

CountryItaly

Recommendation system

Enrico

Friend

Da Carlone

Enrico

LaMediterranea

Meridionale

Friend

EatsEats

EaitalyEa

Da Carlone

LucaJill

Enrico

LaMediterranea

Meridionale

Friend

EatsEats

Eaitaly

select both(‘Friend’) from Person where name = ‘Luca’

Da Carlone

LucaJill

Enrico

LaMediterranea

Meridionale

Friend

EatsEats

Eaitaly

select both(‘Friend’).out(‘Eats’) from Person where name = ‘Luca’

Da Carlone

LucaJill

Enrico

LaMediterranea

Meridionale

Friend

EatsEats

Eaitaly

select both(‘Friend’).out(‘Eats’) from Person where name = ‘Luca’

This is your database

Get last customer bought ‘Barolo’select last(out(‘Order’).in(‘Customer)) from Stock where name = ‘Barolo’

#34:22

Get his’s country

select out(‘City’) from #34:22Udine, Italy

#55:12

Get orders from that country

select in(‘Customer’) from #55:12

Let’s move like aSpider

on the web

OrientDB = {flexibility of Document databases+ complexity of the Graph model

+ Object Oriented concepts+ super fast Index

+ powerful SQL dialect+ multi-master replication and sharding}

Ø configdownload, unzip, run!

cut & paste the db

150,000records per second

(flat records, no index, on commodity hw)

Schema-lessschema is not mandatory, relaxed model,collect heterogeneous documents all together

Schema-fullschema with constraints on fields and validation rules

Customer.age > 17Customer.address not null

Customer.surname is mandatoryCustomer.email matches '\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b'

Schema-mixedschema with mandatory and optional fields + constraints

the best of schema-less and schema-full modes

ACID Transactionsdb.begin();try{ // your code ... db.commit();

} catch( Exception e ) { db.rollback();}

SQLselect * from employee where name like '%Jay%' and status=0

Why reinventyet another language when

the 100% of developers alreadyknow SQL?

OrientDB begins from SQLbut improves it with new

operators for graph manipulation

For the most of the querieseveryday a programmer needs

SQL is simpler,more readable and

compact thenScripting (Map/Reduce)

SQL & relationshipsselect from Account where address.city.country.name = 'Italy'

select from Account where addresses contains (city.country.name = 'Italy')

SQL & trees/graphsselect out('friend’) from V where name = 'Luca' and surname = 'Garulli'

select out[@class='knows’] from V where name = 'Jay' and surname = 'Miner'

traverse friends from #13:55 where $depth < 7

SQL sub queriesselect from ( traverse friends from Profile where $depth < 7 ) where home.city.name = ‘Cologne’

SQL & stringsselect from Profile where name.toUpperCase() = 'LUCA'

select from City where country.name.substring(1,3).toUpperCase() = 'TAL'

select from Agenda where phones contains ( number.indexOf( '+39' ) > -1 )

select from Agenda where email matches '\bA-Z0-9._%+-?+@A-Z0-9.-?+\.A-Z?{2,4}\b'

SQL & schema-lessselect from Profile where any() like '%Jay%'

select from Stock where all() is not null

SQL & collectionsselect from Tree where children contains ( married = true )

select from Tree where children containsAll ( married = true )

select from User where roles containsKey 'shutdown'

select from Graph where edges.size() > 0

Native JSONODocument = new ODocument().fromJSON( " { '@rid' = '26:10', '@class' = 'Developer', 'name' : 'Luca', 'surname' : 'Garulli', 'out' : [ #10:33, #10:232 ] }“ );

Always FreeOpen Source Apache 2 license

free for any purposes,even commercials

Some clients

Kondoot

Scenari

(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License Page 97www.orientechnologies.com

Thanks!Luca Garulli – Founder and CEO

www.twitter.com/lgarulli

Why relationships are cool but join sucks - Big Data & Graphs in Rome

Software