Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala

Alberto Perdomo@albertoperdomo

FOSDEM 2013Graph Processing DevRoom

About me

About me

Co-Founder of Aentos

About me

Co-Founder of Aentos

Living my dream* (CNY)

About me

Co-Founder of Aentos

Living my dream* (CNY)

Web developer

d

Features

Features

★ Extensive movies & TV-shows data

Features

★ Extensive movies & TV-shows data

★ Info & price on streaming & rent services:

iTunes, Amazon, Hulu, Netflix...

Features

★ Extensive movies & TV-shows data

★ Info & price on streaming & rent services:

iTunes, Amazon, Hulu, Netflix...

★ Social features (contacts, comments)

Features

★ Extensive movies & TV-shows data

★ Info & price on streaming & rent services:

iTunes, Amazon, Hulu, Netflix...

★ Social features (contacts, comments)

★ Ratings

Features

★ Extensive movies & TV-shows data

★ Info & price on streaming & rent services:

iTunes, Amazon, Hulu, Netflix...

★ Social features (contacts, comments)

★ Ratings

★ Suggestions by users

Features

★ Extensive movies & TV-shows data

★ Info & price on streaming & rent services:

iTunes, Amazon, Hulu, Netflix...

★ Social features (contacts, comments)

★ Ratings

★ Suggestions by users

★ System recommendations

UnconventionalSystem

Architecture*

Ingredients

✔ MongoDB

Ingredients

✔ MongoDB

✔ Neo4j

Ingredients

✔ MongoDB

✔ Neo4j

✔ Ruby

Ingredients

✔ MongoDB

✔ Neo4j

✔ Ruby

✔ Scala

Ingredients

⚛ System Overview ⚛iPad

Client

Main API(Ruby)

Recommend. Engine(Scala)

MongoDB Neo4jUsersRatingsMovies and TV shows

All data

API: Ruby + MongoDB

API: Ruby + MongoDB

⚛ REST+JSON

API: Ruby + MongoDB

⚛ REST+JSON

⚛ Accounts & sessions

API: Ruby + MongoDB

⚛ REST+JSON

⚛ Accounts & sessions

⚛ Movies & TV-Shows

API: Ruby + MongoDB

⚛ REST+JSON

⚛ Accounts & sessions

⚛ Movies & TV-Shows

⚛ Contacts

API: Ruby + MongoDB

⚛ REST+JSON

⚛ Accounts & sessions

⚛ Movies & TV-Shows

⚛ Contacts

⚛ Ratings

API: Ruby + MongoDB

⚛ REST+JSON

⚛ Accounts & sessions

⚛ Movies & TV-Shows

⚛ Contacts

⚛ Ratings

⚛ User suggestions & recommendations

Recommendations: Scala + Neo4j

Recommendations: Scala + Neo4j

⚛ Movies & TV-shows (IDs & rel.)

Recommendations: Scala + Neo4j

⚛ Movies & TV-shows (IDs & rel.)

⚛ Users

Recommendations: Scala + Neo4j

⚛ Movies & TV-shows (IDs & rel.)

⚛ Users

⚛ Ratings

Communication (RabbitMQ)

iPad Client

Main API(Ruby)

Recommend. Engine(Scala)

MongoDB Neo4j

RabbitMQ + JSON

JSON array of IDs

GET /suggestions

Query params in JSON

UsersRatingsMovies and TV shows

JSON array of data

Retrieve datafor response Collect IDs of suggestions

All dataGraph reset*

Issues

Issues

☠ API server waiting for query in Neo4j

Issues

☠ API server waiting for query in Neo4j

☠ Query MongoDB for add. data

Issues

☠ API server waiting for query in Neo4j

☠ Query MongoDB for add. data

☠ Transform data

Issues

☠ API server waiting for query in Neo4j

☠ Query MongoDB for add. data

☠ Transform data

☠ Ruby server is blocking

Communication (DB)

iPad Client

Main API(Ruby)

Recommend. Engine(Scala)

MongoDB Neo4j

GET /suggestions

UsersRatingsMovies and TV shows

Redirect to Recomm.Eng. w/ hashed params

Retrieve datafor query params Collect IDs of suggestions

All data

JSON array of suggested results

Retrieve data for response

Follow redirect link

Graph reset*

Improvements

Improvements

✌ No more long blocking requests in Ruby server

Improvements

✌ No more long blocking requests in Ruby server

✌ Move heavy lifting to Spray/Scala (async)

Improvements

✌ No more long blocking requests in Ruby server

✌ Move heavy lifting to Spray/Scala (async)

✌ Add cache layer

Improvements

✌ No more long blocking requests in Ruby server

✌ Move heavy lifting to Spray/Scala (async)

✌ Add cache layer

Additional Cache Layer

Additional Cache Layer

* Read-Through

Additional Cache Layer

* Read-Through

* Stores transformed data

Additional Cache Layer

* Read-Through

* Stores transformed data

* Implemented using spray-caching

Total Response Time

40s → <1s

Delta Updates

Delta Updates

* API: New ratings in queue collection (MongoDB)

Delta Updates

* API: New ratings in queue collection (MongoDB)

* Recommendations:Backend checks for updates in queue and applies them in graph

Delta Updates

iPad Client

Main API(Ruby)

Recommend. Engine(Scala)

MongoDB Neo4j

POST /rating

UsersRatingsMovies and TV shows

Store rating +Put rating in queue Collect IDs of suggestions

Check if updatesin queue

GET /suggestions

Apply updatesif pending

Retrieve data for response

(Re)loading data in graph

(Re)loading data in graph

1. Create new Neo4j instance

(Re)loading data in graph

1. Create new Neo4j instance

2. Import data from MongoDB

(Re)loading data in graph

1. Create new Neo4j instance

2. Import data from MongoDB

3. Hot swap

(Re)loading data in graph

1. Create new Neo4j instance

2. Import data from MongoDB

3. Hot swap

★ Allows for massive updates in the primary

database

(Re)loading data in graph

1. Create new Neo4j instance

2. Import data from MongoDB

3. Hot swap

★ Allows for massive updates in the primary

database

★ Used only for content-related data and user

Graph DB as an index for relations*

Graph DB as an index for relations*powerful queries on relations

Graph DB as an index for relations*powerful queries on relations

Graph DB as an index for relations*powerful queries on relations

⁂ like full text search for strings

Design decisions: Graph

Design decisions: Graph

✍ Copy subsets of data in Neo4j

Design decisions: Graph

✍ Copy subsets of data in Neo4j

✍ Rebuild graph with no downtime*

Design decisions: Graph

✍ Copy subsets of data in Neo4j

✍ Rebuild graph with no downtime*

✍ Massive updates blazingly fast

Design decisions: Graph

✍ Copy subsets of data in Neo4j

✍ Rebuild graph with no downtime*

✍ Massive updates blazingly fast

✍ Less critical pieces

The right tools, for us! ✌

API in Ruby & MongoDB

The right tools, for us! ✌

API in Ruby & MongoDBMaintanability, flexibility, libraries,

The right tools, for us! ✌

API in Ruby & MongoDBMaintanability, flexibility, libraries,

schema-free!, development speed

The right tools, for us! ✌

Recommendations

The right tools, for us! ✌

RecommendationsScala: Performance + good interface for Neo4j

The right tools, for us! ✌

RecommendationsScala: Performance + good interface for Neo4j

Neo4j: Graph problem

Thanks! ☃

Alberto Perdomo@albertoperdomo

❤ Credits ❤

Entering Hyperspace by Éole Wind