BATTLELOG//Building scalable web sites with tight game integrationJohan Mjönes & Joakim Bodin

What is battlelog?

› Battlefield 10 year anniversary soon› Battlelog becoming the community hangout› Integrate game launching for PC players

BATTLELOG

Features

› Friend-centric› Easy communication› Current stats and history› Activity log› PC Menu› News / Forums

BATTLELOG

SCALING

TECHNOLOGY

TECHNOLOGYESN PLANET

TECHNOLOGYESN PLANET OVERVIEW

Languages

› Developing Battlelog means developing in Python, Java, C++› The Battlelog Web is written primarily using Python› Plugin, game components in C++› Python allows for rapid web development (e.g. remote access console)

TECHNOLOGY

TECHNOLOGY

Horizontal (sharding)

1-7 7-14 ...

Battlelog

User DBs(slices and shards)

......

Vertical

Battlelog

User Lab Feature

Different processes

Partitioning

Persistence & Index

› MySQL› No joins› Used as indexed KVS

› Apache Solr› Full text search for forums› Other uses in the future› Fast!

TECHNOLOGY

Server to client push

› Uses ESN Beaconpush› Allows delivery of messages from web server to client browser.› Uses long polling or websockets (or flash)

› AV / anti-malware software + push = › Blocks all websockets› Injects Javascript into the DOM› Etc...

TECHNOLOGY

Caching with Memcached

› Great framework support› Service methods cached via annotations

› Invalidation using method signature instead of key

› Allows populating cache directly after modification› Dog pile prevention

TECHNOLOGY

EA Services

› Nucleus› Users

› No user info in Battlelog DB! Success!

› Personas› Soldiers

› Entitlements› Licenses

› Blaze› Game Servers reports to Blaze› Events to Battlelog

› Game Server information› Asynchronous responses (e.g. matchmaking)

› REST› Stats

TECHNOLOGY

Nucleus Details

TECHNOLOGY

Nucleus

Accounts

Entitlements

Web Backend

Web

Soldiers

Licenses

HTTP

TECHNOLOGYBlaze Details

› Uses Web Access Layer (WAL)› WAL client generated from TDF

› TDF is a API definition language

› Blaze events (XML over HTTP)

Blaze

DB

Web BackendWeb Frontend

Blaze

Game Servers

WAL

Real-time Events

Thrift

Events

AJAXWeb Browser

BATTLELOG FEATURES

Friends

› Started out normalized› Unmanagable amount of rows› Ended up as one blob per user

› Packed user ids

› A lot easier to cache properly

BATTLELOG FEATURES

Server Browser

› Custom search server (Java)› Custom query language, minimal message overhead› Fast update, fast search

BATTLELOG FEATURES

Joining a game

BATTLELOG FEATURES

BATTLELOG FEATURES

Joining a game: plugin details

BATTLELOG FEATURES

Web Browser

Game ClientPluginGame UI JS

Pipe

Battle Reports

› Game report from Blaze (HTTP XML event)› Parse and divide per player

› One report per player via internal message queue› Look for unlocks, rankups, medals, awards etc

› Add to feed, send real time updates

› Invalidate player stats caches

› Compile Battlereport (& pre-cache it)› Send notification to involved (logged in) users

BATTLELOG FEATURES

LOAD TESTING

Background

› Load tested using Locust› Open Source (MIT License)› https://github.com/cgbystrom/locust› Built by ESN (& others)

› Battlelog tested with 2.7 million PSU with 36 million players› That’s 40,000 requests per second› The tests was successful

LOAD TESTING

Pre-pass

› Planet dev bar› Timing› Queries

› No point in load testing if there are apparent issues with a single user

LOAD TESTING

Realistic load testing

› Some testing tools simply hammer certain URLs› Locust allows for realistic usage scenarios

› Many players go away for 20 minutes while playing, receiving notifications

› Updated tests using actual usage data from alpha trial / open beta

› Coordinated testing› Same user spans

› Overpopulate tables to simulate fragmented MySQL indices

LOAD TESTING

What did we find?

› Bugs› Bottlenecks

› Never assume where your bottlenecks are› Test to find out

› Network related› Saturated network, not CPU› Keep the traffic within the rack!

› Software related› Best practices be damned, cheat if neccessary› Lazy loading data in controllers› Client side rendering

› Surface + History API + JSON =

LOAD TESTING

GAME + WEB DEVELOPMENT

Being a part of the pipeline

› Data driven games› Battlelog has its own pipeline

› Written in Python› Imports data from the game pipeline

› Weapons, vehicles, levels, dog tags, etc› Assets include images, text, data

› Resolving unlocks and dependencies› Generates python structs and javascript structs

› Generates many helper maps. Some might only be used once.

› DLC friendly!› Result of many iterations

GAME + WEB DEVELOPMENT

User Interface› Separate UI from consoles / in game menu

› Having the same wouldn’t make sense, different UX› Adapt to different devices

› Web features› Easily accessible, e.g. Sharing› Extendability with browser extensions, user styles etc› Access on any platform (play on console, have Battlelog on iPad)

› Easy to iterate over UI changes

GAME + WEB DEVELOPMENT

THE FUTURE

Battlelog will evolve

› Agile: Easy to update› Big things:

› DLC› Public APIs› Other things we can’t talk about

› Small things› Improvements based on community feedback

THE FUTURE

YOUR TEAM WON

QUESTIONS?

Johan Mjönes@nollbit

Joakim Bodin@jbripley

The Battlelog team is hiring!

http://dice.se/hiring.aspjobs@dice.se

Technical DirectorDevelopment DirectorBackendFrontend