Post on 17-Feb-2017
transcript
It's in the game: The path to micro-services at Electronic Arts with Docker
Andrew HatelyIBM CTO Cloud Architecture
Scott PorterEA Cloudcell Server LeadMelbourne Firemonkeys
GAMING MARKET GROWING ACROSS ALL PLATFORMS AND REGIONS
Mobile PC/F2P Console
NORTH AMERICA LATAM EUROPE ASIA/ROW
GROWTH YoY 12%
REVENUES 2015est.
21BGROWTH YoY 19%
REVENUES 2015est.
4BGROWTH YoY 7%
REVENUES 2015est.
17BGROWTH YoY 9%
REVENUES 2015est.
40B
17% 46% 37%41% 16% 43%
195M 182M 553M 817M
33% 24% 43% 51% 44% 5%
2015 est. Revenues Consist of Software Gaming Market
MOBILE FUELING INDUSTRY GROWTH
2009 2010 2011 2012 2013 2014 2015$3.3 $4.3 $6.1
$9.9
$17.0
$27.8
$34.0
Mobile games industry has grown 10x in less than a decade
Mobile Gaming Revenue, WW $B
Mobile Game Server architecture/requirements• Mobile Gaming Back-end is a very typical Web application• Frequent updates to game content - infrequent updates to game
engine• Daily peak users 150,000 to 250,000 simultaneous users and 1-2
million daily • Need to scale-up/down around release and content pack releases• Stateless front-ends with memory caching and both SQL and no-
SQL DB• 50TB per day of data - Increasing use of analytics on user
experience, recorded game telemetry, replays, etc.
Firew
all
Load
Ba
lanc
er
NGNI
X
Game Server REDIS
Memcache
MySQL cluster
PHPPHPPHPPHPPHP
Object Storage
Demo – Docker for Gaming
• Our current Ops complexity
• Partnership with IBM
• Why we believe Docker can help
• How we integrated Docker
Scott PorterCloudcell Server LeadEA Melbourne Firemonkeys
scott@ea.com@sl_porter
EA MELBOURNE FIREMONKEYS
• Moble studio based in Melbourne, Australia
• Formed in 2012 from two EA Mobile studios, Iron Monkey and Firemint
• Focused on developing the highest quality mobile titles
• Firemonkeys in house Client/Server Tech Stack
• Mobile focused connected features
• Originally developed by Firemint for Flight Control
Integrated into all Firemonkeys live titles
• Live Service Games
• Bi monthly client releases
• No downtime
What do we do?
• 13.5k Peak Requests/s
• ~900M Requests/day
• ~430k Peak Concurrent Players
• 5M – 8M Daily Active Players
• > 100 TB Binary Data
• 170 Hosts (60 Live Hosts)
• Team of 7
Current Scale
• Baremetal Hosts
• Nginx, PHP, Memcache, MySQL, Redis
• Batched client requests, short-lived, stateless
• Requests can touch any datastore
Architecture
Game Logic Data StorageLoad Balancing
Architecture
External Services
StatsD / Diamond / Graphite / Graphana
Object Storage
Datadog
• Automated Development / Manual Live process
• In house database migration and code deployment system
• Bash & rsync
• Rollbacks are handled via code duplication
Current Deployment Methodology
• Asynchronous processes (workers) manually configured ( services, cron jobs )
• Brittle.
• SaltStack is awesome!
• But still slow, complex, no HA
Current Orchestration Methodology
• Moved binary data to SoftLayer Object Storage (Open Stack Swift)
• Currently provision for Peak
• Shard live environments per game title
• Interested in transitioning to microservices
Current Scalability
• Improved deployments
• Growing ecosystem
• Scalable
• Easy transition, run side by side with existing stack
Why Docker?
• Focused on the Game Logic Layer
• Decomposition• Nginx• Admin• Application• Workers
Packaging Cloudcell
• Generic Containers
• Run multiple containers, not processes
• Tradeoff considerations
Packaging Cloudcell
Code
Building Images
Jenkins
Docker Registry
Base Images
nginx
php-fpm
php-worker
Built Images
php-fpm-nfs-0001
php-fpm-nfs-0002
php-worker-a-1
nfs-0002.tar
worker-a-1.tar
nfs-0001.tar
Dev Env
Jenkins
Application Code (SVN)
• Fleet
• Unit files per service• Nginx• Admin• Application• Each worker type
Orchestration
Load Testing
Host
Baremetal
nginx
Application
Worker A Load Generation
LTG LTG
LTG LTG
LTG
LTG
Host
Docker
Docker
nginx
php-fpm-nfs-0001
php-worker-a-1
etcd
Fleet
• Overall 5 – 10% overhead running Load Test with Docker
Load Testing Results
PLACEHOLDER NEED A REAL GRAPH HERE!
• Docker Private Registry
• Jenkins• Studio Jenkins ships code to build machine• Build machine builds and commits images to
Registry• Generates Fleet configuration from templates• Executes Fleet deployments
Deployments
• Config Editor• Bridges Cloudcell Application deployments to
environment state• Provides a simple UI for building, deployments,
and state via Jenkins and Fleet
Deployments
Deployments
Docker Build Machine Host
Docker
etcd
Jenkins
Docker Registry
Config Editor nginx
php-fpm-nfs-0001
php-worker-a-1
Fleet
Docker enhanced architecture on SoftLayerCentOS 7 host
Discovery/Config
Fleet managed Docker services
nginx-confd.service
NGINX-confd container
Orchestration Docker
php-fpm@1.servic
eea/php-fpm:latest
1
php-fpm@2.servic
eea/php-fpm:latest
2php-fpm-discovery
@2.service
php-worker-A@2.service
ea/php-worker:latest
2
php-worker-B@1.service
ea/php-worker:latest
1
php-worker-A-discovery@2.service
php-worker-B-discovery@1.service
php-worker-A@1.service
ea/php-worker:latest
1php-worker-A-discovery
@1.service
php-fpm-discovery@1.service
Redi
sEx
tern
al d
ata
serv
ices
MyS
QLM
emca
ched
Load
Bal
ance
r
Configuration, orchestration and deployment
host
service
service
service
service
host
service
service
service
service
host
service
service
service
service
host
service
service
service
service
Configure topology, maintain state.
Build images with code, deploy to or scale target environments.
Run containers in staging, load test, and production. Capture
metrics.
• Centralised Logging
• Monitoring - processes inside containers
Future Work
— Scott Porter
“Using IBM and Docker container solutions, we have been able to build a gaming platform that is massively scalable, highly performant and meets our deployment velocity needs.”
33
Thank you!@cloudtrollhately@us.ibm.com
Andrew HatelyScott Porter@sl_portersporter@ea.com