Let's Chat about Chat - RICON 2014

Post on 21-Jun-2015

265 views 4 download

Tags:

description

In League of Legends, just as in any competitive team game, communication is essential to success. Therefore, when building Chat for the game we had to make sure that the new service would be absolutely rock solid in every respect. This includes not only guaranteed message delivery and consistent presence propagation across the system, but also maintenance of the created social network graph. In this talk I would like to present how we achieved linear scalability for Chat, improved its overall fault tolerance, and got ready for the new features we wanted to ship. I will also discuss in detail why we migrated our data from MySQL to Riak and how we used CRDTs to deal with conflicting object updates.

transcript

ABOUT CHATLET’S CHAT

Michal Ptaszek, @michalptaszekRiot Games

WHAT’S PLANNED

1 2 3

GAME CHAT RIAK LESSONSLEARNED

4 5

Q&A

WHAT IS LEAGUE OF LEGENDS?

2009LAUNCH

TEAMORIENTED

100+CHAMPS

MODERNFANTASY

MESSAGING SERVICE

PRESENCE SERVICE

SOCIAL GRAPH SERVICE

WHAT IS IT?CHAT

WHAT IS IT?CHAT

CHAT BY THE NUMBERS

67 million monthly players

27 million daily

players

7.5 million concurrent

players

1 billion events

routed per server, per

day

CHAT AT 10K FEETSTABLE, SCALABLE CHAT SERVICE

PROTOCOL DATA STORESERVER

STABLE, SCALABLE CHAT SERVICE

DATA STORESERVERPROTOCOL

CHAT AT 10K FEET

PROTOCOL: XMPP

Decentralized Architecture

Openness

Extensibility

Availability of Client

Libraries

Security Wide Adoption

STABLE, SCALABLE CHAT SERVICE

CHAT AT 10K FEET

DATA STOREPROTOCOL SERVER

SERVER: EJABBERD

Which gives us...

TECHNOLOGY: ERLANG/OTPErlang is...

A functional language

Built with concurrency and distribution in mind

Able to scale extremely well

Capable of reloading code on the fly

A declarative style of programming

An easier way to build our distributed applications

More time to focus on coding

Less downtime

SERVER: EJABBERD - PHILOSOPHY

ARCHITECTURE

LET ITCRASH

FAULTTOLERANCE

SERVER: EJABBERD - ARCHITECTURE

SecondaryRiak Cluster

Ejabberd Ejabberd LB

Riak Riak

STABLE, SCALABLE CHAT SERVICE

CHAT AT 10K FEET

SERVERPROTOCOL DATA STORE

‣‣

RIAK

RIAK: ARCHITECTURE

SCALE Linearly scalable

No growth headaches

FAULTTOLERANCE No SPoF Higher

uptime

SCHEMA-LESSFaster feature

iterations

More shipped features

ARCHITECTURE

DATA STORE: RIAK

Ejabberd Ejabberd

Riak Riak

Ejabberd

Riak‣

CRDT

RIAK: CONFLICT RESOLUTION

CRDT

RIAK: CONFLICT RESOLUTION

FRIENDS LISTS

RIAK: DATA TYPES

○○○○

IGNORE LISTS

RIAK: DATA TYPES

OFFLINE MESSAGES

RIAK: DATA TYPES

LAST ONLINE TS

RIAK: DATA TYPES

Source cluster #1

MDC REPLICATION

DATA STORE: RIAK

Sinkcluster

Source cluster #2

Source cluster #3

RIAK STATS

RIAK: MONITORING

RIAK CLUSTER

RIAK: MONITORING

KEEP YOUR OBJECTS SMALL

LESSONS LEARNED

AVOID SIBLINGS CREATION

LESSONS LEARNED

ALWAYS LOAD TEST YOUR CODE

LESSONS LEARNED

THINGS WILL FAIL

LESSONS LEARNED

CHAT IS DOING GREAT!

SCALE AND PERFORMANCE

CHAT IS EVOLVING

SITUATIONCURRENT

THANK YOU!ANY QUESTIONS?