INNOVATION DANK DEVOPSJesper Richter-Reichhelm (@jrirei)

BackendClient

BackendClient

BackendClient

BackendClient

DevOps

Some say DevOps is about

• Agile admins • Faster releases • Virtualization • Automation tools

I say it’s about

• Big Picture Mindset • Reality Feedback • Learning together

Classic way

DevOps

Wooga way

Dev Ops

Dev Ops

DevOps is about

Monster World

SQL

app

lb

SQL

Browser

sql sql

appapp

Things we did

Many small optimisations

Bigger databases…

… many, many more of them

app

slave

master master

slave

1. create new DBs

2. setup replication

3. start using masters

4. cut replication

5. truncate data

BA

AB

AB

AB

AB

app

slave

master master

slave slave

master master

slave

AC

AC

BD

BD

AC

AC

BD

BD

Stateless approach

1+ million users / day

10+ billion requests / month

~10 DB writes / request

> 100,000 DB writes / second

http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/

Things we did

Many small optimisations

Bigger databases…

… many, many more of them

… and finally the right thing!

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

SQL

EBS EBS EBS EBS

lb lblblb

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

EBSSQLBrowserlb

lb

lb

lb

app

app

app

app

app

app

app app

SQL SQL

sql sql

SQL SQL

sql sql

SQL SQL

sql

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

Rd Rd

rd rd

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

app appapp app app app app app app app app appapp app app

SQL

EBS EBS EBS EBS

lb lblblb

38 single points of failure

flickr.com/photos/qubodup/9244593352

Innovation

Wooga way

FE Dev

BE Dev

Art

Product

“You build it, you run it. - Werner Vogels

Freedom

Monster World

Monster World

Magic LandMonster World

Magic LandMonster World

Magic Land

Server Database

Server Database

One Game Session

Stateful approach

1+ million users / day

4 sessions / user / day

1 DB write / session

~50 DB writes / second

Server

sessionsessionsessionsession

Server

sessionsessionsessionsession

Server

sessionsessionsessionsession

S3

http://www.flickr.com/photos/aigle_dore/

Knowledge Exchange

DevOpsDevOps ???

Wooga way

FE Dev

BE Dev

Art

Product

Exchanging knowledge

flickr.com/photos/wili/233621595

Being transparent

flickr.com/photos/marcomagrini/698692268

flickr.com/groups/bestfavoriterelayrace/

Rotating jobs

2009

Ruby

LAMP

statelessBB BI

MW

Redis HH

LAMP

statelessBB BI

Ruby MW MW 2

Ruby MW MW 2

2010

LAMP

statelessBB BI

Redis HH

Ruby MW MW 2

2010

Erlang

statefulML

2011

Redis

Ruby

LAMP

statelessBB BI

MW

HH

MW 2

Erlang

statefulML

offline

DD

Erlang

statefulML

2011

Redis

Ruby

Riak

Javascript

ev. Ruby

1 doc

LAMP

statelessBB BI

MW

HH

MW 2

PI

PP

MWM

2012

Erlang

Redis

Ruby

stateful

Riak

Javascript

offline

ev. Ruby

1 doc

JRuby

shared

LAMP

multiplay

statelessBB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

GX

FT

KB

BI 2

2013

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

…

2014

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

… …

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services

BB BI

MW

HH

MW 2

ML

DD

PI

PP

MWM PV

SBS

DYN

GX

FT

KB

BI 2

DIV

… …

…

…

…2015

Erlang

Redis

Ruby

stateful

Riak

Javascriptrealtime

offline

ev. Ruby

1 doc

JRuby Scala

shared

Go

Elixir

hybrid

LAMP

generic

multiplay

stateless

services2015

MySQLPython

Python MySQL

Erlang

RedisRuby stateful

Riak

Javascript

realtime

offline

ev. Ruby

1 doc

JRuby

Scala

shared

Go

Elixir

hybrid

generic

multiplay

stateless

services

Languages DBs State mngt. Misc.

Python MySQL

Erlang

RedisRuby stateful

Riak

Javascript

realtime

offline

ev. Ruby

1 doc

JRuby

Scala

shared

Go

Elixir

hybrid

generic

multiplay

stateless

services

Languages DBs State mngt. Misc.

Wooga’stoolbox

Having trust

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

- Agile Manifesto

Innovation

Thank you

Questions?

Jesper Richter-Reichhelm (@jrirei)

Questions?

Jesper Richter-Reichhelm (@jrirei)