NoSQL Riak MongoDB Elasticsearch - All The Same?

transcript

MongoDB, Elasticsearch, Riak – all the same?

Eberhard Wolff Freelancer

Head Technology Advisory Board adesso AG

http://ewolff.com

Eberhard Wolff - @ewolff

Leseprobe: http://bit.ly/CD-Buch

Modeling: Relational

Databases vs. JSON

Financial System •  Different financial products

•  Mapping objects / database

•  Inheritance

E/R Model

Stock Zero Bond Option

Country > 20 database tables Up to 25 attributes

Currency

JOINs L

Get all asset with interest

rate x

Asset Type ID

Zero Bond

Interest Rate

Fixed Rate Bond

Interest Rate

Stock Option

Preferred Underlying asset

Country Price Country

Currency

{ "ID" : "42", "type" : "Fixed Rate Bond", "Country" : "DE", "Currency" : "EUR", "ISIN" : "DE0001141562", "Interest Rate" : "2.5" }

All stores in this presentation

support JSON

Scaling Relational Databases

Larger Server

DB Server DB Server Expensive

Server Limited

Common Storage

DB Server Expensive

Storage

Limited

DB Server

DB Server DB Server e.g. Oracle

Replication Cheap Server

Almost unlimited DB Server

DB Server DB Server DB Server

Inconsistent data

Conflict resolution or Read only

Replication

DB Server

DB Server DB Server DB Server

MySQL Master-Slave

Oracle Advanced

Replication

Network Failure •  Either Answer & provide outdated data •  or Don’t answer i.e. always provide up to date data

CAP •  Consistency •  Availability •  Network Partition Tolerance

•  If network fails provide a potentially incorrect answer or no at all?

BASE •  Basically Available •  Soft State •  Eventually (= in the end) consistent

•  i.e. give potentially incorrect answer

BASE and Relational DBs •  Very limited

•  Stand by •  Read only replica

•  No truly distributed DB

Relational & BASE •  Most relational operations cover

multiple tables

•  Needs locks across multiple servers

•  Not realistically possible

NoSQL & BASE •  Typical operation covers one data

structure •  …that contains more information

•  No complex locking

•  More sophisticated BASE

Naïve View on NoSQL

Key / Value Stores •  Map Key to Value •  For simple data structure •  Retrieval only by key

•  Easy scalability •  Only for simple

applications

Key Value 42 Some

Document Oriented •  Documents

e.g. JSON •  Complex

structures & queries

•  Still great scalability •  For more complex

applications

{ "author":{ "name":"Eberhard Wolff", "email":"eberhard.wolff@gmail.com" }, "title": "Continuous Delivery”, }

Graph, Column

Oriented…

Educated View on NoSQL

Key / value Document-based

Search engine All the same?

MongoDB

elasticsearch

MongoDB

elasticsearch

•  Key / value •  Truly distributed database

What is Riak?

Riak: Technologies •  Erlang

•  Open Source (Apache 2.0)

•  Company: Basho

•  Allows secondary indices

•  Riak Search 2.0: Solr integration •  Solr: Lucene based search engine •  API compatible to Solr

•  Key / value or document based?

More indices

•  Map/reduce •  Scans all datasets

•  Can store large binary objects

More Features

Scaling Riak •  Based on the Dynamo paper

•  Well understood •  …and battle proofed at Amazon

Scaling Riak Server A

Shard1 Shard3

Shard4

Server B Shard2 Shard1

Shard4

Server D Shard4 Shard2

Shard3

Server C Shard3 Shard2

Shard1

Shard1 Shard3

Shard4

Shard3

Shard1

Shard1 Shard3

Shard4

Shard3

Shard1

New Server

Tuning BASE •  N node with replica •  R nodes read from •  W nodes written to •  Trade off

Is it bullet proof?

Jepsen •  Test suite for network failures etc •  https://aphyr.com/tags/jepsen

•  Riak succeeds •  …if tuned correctly •  …might still need to merge versions •  https://aphyr.com/posts/285-call-me-

maybe-riak

MongoDB

elasticsearch

MongoDB

elasticsearch

MongoDB

elasticsearch

•  Document-oriented •  MMAPv1

Memory-mapped files + journal •  New in 3.0: WiredTiger for complex

loads Humongous

What is MongoDB?

MongoDB: Technologies •  C++

•  Open Source (AGPL)

•  Company: MongoDB, Inc.

•  Can store large binary objects

•  Its own full text search

More Features

More Features •  Map / Reduce

•  JavaScript

•  Aggregation framework

Scaling MongoDB

Replica 1

Shard 1

Replica 2

Replica 3

Shard 2

Replica 1

Replica 2

Replica 3

Availability

Replica 1

Shard 1

Replica 2

Replica 3

Shard 2

Replica 1

Replica 2

Replica 3

Scaling MongoDB

Replica 1

Shard 1

Replica 2

Replica 3

Replica 1

Shard 2

Replica 2

Replica 3

Replica 1

Shard 3

Replica 2

Replica 3

Scaling MongoDB

Replica 1

Shard 1

Replica 2

Replica 3

Shard 2

Replica 1

Replica 2

Replica 3

Tuning BASE •  Write concerns •  How many nodes should

acknowledge the write?

•  Read from primary •  …or also secondaries

Jepsen •  Mongo loses writes •  A bug – might still be there •  Also: non-acknowledge writes might

still survive •  …and overwrite other data

•  https://aphyr.com/posts/284-call-me-maybe-mongodb

MongoDB

elasticsearch

MongoDB

elasticsearch

MongoDB

elasticsearch

Database =Storage + Search

elasticsearch =Storage + Search

What is elasticsearch? •  Search Engine

•  Also stores original documents

•  Based on Lucene Search Libray

•  Easy scaling

elasticsearch: Technologies •  Java

•  REST

•  Open Source (Apache)

•  Backed by company elasticsearch

elasticsearch Internals •  Append only file •  Many benefits

•  But not too great for updates

Scaling elasticsearch

Server Server Server

Shard 1 Replica 1

Replica 2 Shard 2

Replica 3 Shard 3

Tuning BASE •  Write acknowledge: 1, majority, all •  Including indexing

•  Read from primary •  …or also secondaries

Jepsen •  Loses data even if just one node is

partioned (June 2014) •  Actively worked on •  It’s a search engine…

•  https://aphyr.com/posts/317-call-me-maybe-elasticsearch

•  http://www.elasticsearch.org/guide/en/elasticsearch/resiliency/current/

Scenarios

elasticsearch

Search •  Powerful query language

•  Configurable index

•  Text analysis •  Stop words •  Stemming

Facets •  Number of hits by category

•  Useful for statistics •  & Big Data

•  Statistical facet (+ computation)

•  Range facets etc.

MongoDB

elasticsearch

MongoDB

elasticsearch

Conclusion •  Relational databases might be

BASE •  NoSQL embraces BASE better •  Key / Value, Document stores and

search engine: very similar features •  Care about scaling •  Care about resilience

Thank You!