Date post: | 23-Jan-2015 |
Category: |
Documents |
Upload: | larryleaseweb |
View: | 370 times |
Download: | 0 times |
GAMES FOR THE MASSESSCALING RAILS TO THE EXTREME
EuRuKo 2011 in Berlin
Who is that guy?
Jesper Richter-‐Reichhelm
@jrirei
Berlin, Germany
Head of Engineering @ wooga
Wooga develops social games
Wooga has dedicated game teams
PHPSQLCloud
RubySQL + Redis
Cloud
RubyRedis
Bare Metal
Wooga has dedicated game teams
PHPSQLCloud
RubySQL + Redis
Cloud
RubyRedis
Bare Metal
Both games are similar in their setup
Flash client Ruby backend
Both games have similar problems
High traffic
• 200+ million h=p requests / day
High DB throughput
• 100.000 DB operaDons / second
Hardest problems to solve
• DB migraDons, DB failover, DB performance, ...
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
Big cloud vs. bare metal
Big cloud: Amazon EC2
• new server in 5 minutes
• automated deployment
• load based setup
Big cloud vs. bare metal
Big cloud: Amazon EC2
• new server in 5 minutes
• automated deployment
• load based setup
Bare metal: Hetzner
• new server in 2-‐14 days
• automated deployment
What kind of app server are available?
XL CPU instance
• 8 virtual cores
• 8 GB RAM
Bare metal servers are more powerful
XL CPU instance
• 8 virtual cores
• 8 GB RAM
EQ10 server
• 12 “processing threads”
• 24 GB RAM
Bare metal servers are more powerful
XL CPU instance
• 8 virtual cores
• 8 GB RAM
=> 390,-‐ € / month
EQ10 server
• 12 “processing threads”
• 24 GB RAM
=> 200,-‐ € / month
And how is the network?
min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms
Want a fast network? Get your own!
min ping: 0.2 -‐ 0.3 ms
max ping: 7.8 -‐ 59 ms
avg ping: 0.8 -‐ 4.4 ms
min ping: 0.2 ms
max ping: 0.3 ms
avg ping: 0.2 ms
Want a fast network? Get your own!
min ping: 0.2 -‐ 0.3 ms
max ping: 7.8 -‐ 59 ms
avg ping: 0.8 -‐ 4.4 ms
22 DB calls per hfp calls
min ping: 0.2 ms
max ping: 0.3 ms
avg ping: 0.2 ms
saved 13 -‐ 92 ms latency
What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
Bare metal is great if you
• want to save money
• need stable performance
What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
Say: “Don’t worry”
Bare metal is great if you
• want to save money
• need stable performance
Say: “Think ahead”
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis)
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
Redis only
If MySQL is a truck, ...
Fast enough for reads
Can store on disk
Robust replicakon
h=p://www.flickr.com/photos/erix/245657047/
If MySQL is a truck, Redis is a Ferrari
Fast enough for reads
Can store on disk
Robust replicakon
Super fast reads/writes
Out of memory => dead
Fragile replicakon
h=p://www.flickr.com/photos/erix/245657047/
Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis)
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
Redis only
Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis)
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
MySQL has its hard limits
Redis only
Works fine if done right
How many tables to store a user in?
22 MySQL tables
7 Redis hashes
How many tables to store a user in?
22 MySQL tables
7 Redis hashes
1 Redis Hash
How many tables to store a user in?
22 MySQL tables
7 Redis hashes
Easy to implement
Faster for small changes
1 Redis Hash
Less top level keys
Export and import is easy
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis used vm
Dumps to disk impossible
• Own system to store JSON files to disk
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis used vm
Dumps to disk impossible
• Own system to store JSON files to disk
Redis became a cache only
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis now uses diskstore
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis now uses diskstore
• Experimental branch
• Stores async. to disk
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis now uses diskstore
• Experimental branch
• Stores async. to disk
Redis became a DB again
There are many inackve users
hourly
daily
monthly
lifetime
0M 2M 4M 6M
6,0M
3,0M
0,4M
0,1Mhourly
daily
monthly
lifetime
0M 8M 16M 24M
20,8M
8,0M
1,3M
0,3M
Not all users play at the same day
In-‐memory
100% of users in RAM
DB hierarchy
• Redis in-‐memory
• Redis diskstore on disk
3% of users in RAM
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
Let’s dig a bit deeper
1,300,000 daily users
5000 hfp calls/s peak
100 ms response kme
35-‐70 servers
400,000 daily users
2500 hfp calls/s peak
10 ms response kme
11 servers
Hoskng costs / DAU compared
HosDng
5x higher
There are 2 main reasons for higher cost
Databases Other
DB: 20x higher
Other: 3x higher
There are 2 main reasons for higher cost
Databases Other
DB: 20x higher(both RAM and cost)
Other: 3x higher
No DB hierarchy!
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
depends ...
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
depends ...
never
again!
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
depends ...
never, ever,
ever again!
Next game will do things befer
Start in cloud and then go metal?
Redis only
DB hierarchy
BTW, wooga is hiring!
Q & A
slideshare.net/wooga
Jesper Richter-‐Reichhelm@jrirei
wooga.com/jobs
BTW, wooga is hiring!
Thank you
slideshare.net/wooga
Jesper Richter-‐Reichhelm@jrirei
wooga.com/jobs