Introduction to Polyglot Persistence

Antonios Giannopoulos Database Administrator at ObjectRocket by Rackspace

FOSSCOMM 2016

Background

- 14 years in databases and system engineering

- NoSQL DBA @ ObjectRocket by Rackspace

- Passionate about MongoDB & Cassandra

What is Polyglot Persistence?

A set of applications that use several core database technologies

ApplicationLayer

Key/Value Store

Column-family

Database

Relational

Database

What is Polyglot Persistence?

Using the right tool for the right use case

Let there be RDBMS“Monoglot” was (and still is) fine for simple application (one type of workload)

But… applications become complex

A simple E-commerce platform must have:- Session data (Add to Basket)- Search Engine (Search for products)- Recommendation engine (Customers Who Bought This Item Also Bought)- Payment platform - Geo Location service

Applications growing rapidly

A RDBMS taleIn the good old days “monoglot” = RDBMS

Once upon a time there was an RDBMS with performance issues- Vertical scaling - Secondary indexes- Partitioning- Denormalize- Read-Only Slaves- Shading - Start separating workloads

But… The more we scale the more features we miss

The 3V EraVolume: Amount of dataVelocity: Speed of data processingVariety: Number of types of data

New databases introduced for Big Data

General-purpose DB is no longer on-trend

Per use-case Datastores becoming more popular

The raise of Polyglot Persistence

CAP theorem

Picking the right tools – Data Structure

- Relational databases (Oracle, MySQL)

- Key-value stores (Redis, Riak)

- Column Family stores (Cassandra, Hbase)

- Document databases (MongoDB, CouchDB)

- Graph databases (Neo4J)

Relational Databases

Based on the relational model (Codd)Data are organized in tables (Row and columns)Using SQL (Structured Query Language)

Relational Databases

Use when:- Your dataset is relational- Strong consistency needed- Access patterns are unknown

But … Doesn’t scale well horizontally

Use Cases:- Due to early adoption are everywhere- Payment systems

Key/Value stores

A big hash map associative array- Very simple, One key <-> One value- Very fast read/write- No secondary indexes

{“Key”: (VRN)} => {value (car facts)}

[make#Ford {YYY0000} => model#Fiesta year#2010]

Key/Value stores

Use when:- Operations are based on the key- Data is not highly related- Basic CRUD needed

But… Complex queries are painful

Use cases:- Session Data- User Profile/Preferences- Shopping cards

Document Databases

Nested structures of keys and their values- Very flexible schema (JSON, XML)- One key One value but value is visible to queries- Supports hierarchical data- Supports secondary indexes

{“id”(VRN)} => {“document” (car facts)}

{ “make”: Ford”,{YYY0000} => “model”:”Fiesta”, “year”:2010 }

Document Databases

Use when:- You don’t know much for the schema - Unstructured and Heterogeneous data

But … Joins and references are trickyDe-normalization requires more space

Use cases:- Product Catalog- CMS- Event logging from different sources

Column family

In a table, data of the same column is stored together (K-V that V is K-V)- Data organized as columns- Great for sparse tables- Very fast column operation including aggregation

{“id”(VRN)} => {“column families” (car facts)}

{ “car”:{“make”: Ford”, “model”:”Fiesta” …},{YYY0000} => “parts”:{…}, “service”: {…} }

Column family

Use when:- Big Data (Huge write volumes)- Versioning (Time-series data) But… Know your statements in advanceSchema design is not trivial

Use cases:- Time series data- Bidding platforms- Playlists

Graph Databases

Inspired by the graph theoryNodes hold Data/Entities Each node connect to others using attribute(s)

Graph Databases

Use when:- Highly interconnected data- Define explicit relationships and need traversal queries

But … Doesn’t scale well horizontally Use Cases:- Where 3rd degree (or higher) relationship needed - Social Media - Queries like friend of a friend

What about Hadoop?

Framework for distributed storing and processing large sets of data

Use for:- ETL - Read raw data, apply filters create structured summary- Exploration engine / Data discovery- Data archive / Massive Storage

But…Do not use Hadoop as a database replacement

Picking the right tools - Questions

Data structure:- Does it have a natural structure? Is unstructured?- How it is connected to each other?- How is it distributed?- How much data?

Access Patterns:- Read/Write ratio?- Uniform or random?- What is more important?

Picking the right tools - Questions

Organization needs:- Do I need Authentication? What type?- Do I need Encryption?- Do I need Backups?- Do I need a Disaster Site?- What level of Monitoring?- Drivers, Languages?

Tools- Third Party tools- Add-ons/Plugins

Picking the right tools - Questions

Maturity- How long is in the market?

Documentation - Books, tutorials- Training

Type of Support - Community - Commercial support

Challenges

Define the architecture- Decide which datastore will use to store certain data - Wrong decision can lead to painful migration(s)

Deployment complexity- Provision different type of HW, OS , patches …- Backup/Restore - Control configuration changes- Monitor all different components

Challenges

Application complexity- Different connection per datastore- Handle different type of errors- Map different results on the application layer- Keep Datastores in-sync (cross-database consistency)

- Active/Passive topology- Active/Active topology

Training for Devs and Ops- Develop new skills for your teams- Support for a period of time

Don’ts and anti-patterns

Over engineering- Keep it simple- Remove pieces that don’t add value

Conform to stereotypes- Use cases are general guidelines - Benchmarks are indicators

Stay static- Try new technologies- Being on-trend but cautious

Use Case: Mongo – ElasticSearch

MongoDB is using B-Tree indexes for everything

B-Tree is great but not excel for all use-cases

Start getting hard limits for certain use cases like FullText search

Lucene engine is better option for Full text search (using inverted indexes)

We already had the ElasticSearch to our portfolio so we just “connected” the two products

Use Case: Mongo – Elastic Search

- Connector is using the Active/Passive model

- Writes must go through MongoDB and then propagated to ElasticSearch

- Flexible, user can pick what to propagate:- Database(s)- Collection(s)- Field mapping- Indexing/analyzing

Use Case: Mongo – Elastic Search

- An initial sync is needed (bulk API)

- Connector is using a tailable cursor that reads MongoDB oplog and propagate the changes

- Similar to Extract, Transform, Load (ETL)

- Application is responsible to properly direct the read requests to the most suitable datastore

Call me Polyglot , Call me Multi-Model

MongoDB (version 3 or higher)- MMAPv1- WiredTiger- In-Memory

Percona Server for MongoDB- MMAPv1- WiredTiger- RocksDB- PerconaFT

The future(?) Multi Model databases

Support different models within the engine -OR-Offer different layers on top of the engine

OrientDB supporting graph, document and key/value modelsRelationships are managed as in graph databases with direct connections between records

FoundationDB feature layers on top of a key-value store

Keep in touch

: I iamantonios antonios.giannopoulos@rackspace.co.uk

We are hiring!!! Data Engineers, DevOps, DBAs and morehttp://objectrocket.com/careershttps://www.rackspace.com/talent/

Questions?

Thank you!!!