+ All Categories
Home > Technology > Scaling an API: From Reboots to Redis, how (not) to do it

Scaling an API: From Reboots to Redis, how (not) to do it

Date post: 09-Jun-2015
Category:
Upload: ciaran-rooney
View: 656 times
Download: 3 times
Share this document with a friend
Description:
How do you massively scale an API as your business grows? Ciaran worked for a company that has grown from a small B2C app written in PHP/MySQL to a B2B advertising company doing 15 Billion API calls per month. Along the way there have been some big technology changes and decisions that have both helped and hindered us. Along the way we have had 3 day outages, several AWS outages, massive traffic spikes especially early days when we signed customers like Wordpress.com, and many many improvements, re-factors and even successes. Today and for the last year we are highly modern engineering team, using many of the latest technologies and methodologies. We code in PHP and Python with C, Go, CoffeeScript and more, store data in MongoDB, Redis, memcache, Solr and MySQL, we have a full CI environment, do deploys using IRC (and Hubot) and Puppet everything. We have 100% (not 99.999%) up time for over 12 months now.
Popular Tags:
55
Scaling an API From Reboots to Redis, how (not) to do it
Transcript
Page 1: Scaling an API: From Reboots to Redis, how (not) to do it

Scaling an APIFrom Reboots to Redis, how (not) to do it

Page 2: Scaling an API: From Reboots to Redis, how (not) to do it

{ "fullName": "Ciarán Rooney", "work": { "position": ["CTO", "Speaker", "Mentor"] }, "home": { "email": "[email protected]", "twitter": "@CiaranR", "website": "http://ciaran.ie", "hobby": "http://phplondon.org" }}

Page 3: Scaling an API: From Reboots to Redis, how (not) to do it
Page 4: Scaling an API: From Reboots to Redis, how (not) to do it

{ "dataYear": "2013", "dataMonth": "april", "dataPeriod": "30 days", "dataRequests": { "apiRequests": "18,256,654,541", "impressions": “5,773,142,857", "uniqueUsers": "145,137,679" }, "customers": { "websitesPowered": "1,087,348", "customers": "26,379" }}

Page 5: Scaling an API: From Reboots to Redis, how (not) to do it
Page 6: Scaling an API: From Reboots to Redis, how (not) to do it
Page 7: Scaling an API: From Reboots to Redis, how (not) to do it
Page 8: Scaling an API: From Reboots to Redis, how (not) to do it
Page 9: Scaling an API: From Reboots to Redis, how (not) to do it
Page 10: Scaling an API: From Reboots to Redis, how (not) to do it
Page 11: Scaling an API: From Reboots to Redis, how (not) to do it
Page 12: Scaling an API: From Reboots to Redis, how (not) to do it
Page 13: Scaling an API: From Reboots to Redis, how (not) to do it
Page 14: Scaling an API: From Reboots to Redis, how (not) to do it
Page 15: Scaling an API: From Reboots to Redis, how (not) to do it

BIND / DNS

imageStorage

eMail

Page 16: Scaling an API: From Reboots to Redis, how (not) to do it

imageStorage

eMail

Page 17: Scaling an API: From Reboots to Redis, how (not) to do it

imageStorage

Page 18: Scaling an API: From Reboots to Redis, how (not) to do it
Page 19: Scaling an API: From Reboots to Redis, how (not) to do it
Page 20: Scaling an API: From Reboots to Redis, how (not) to do it
Page 21: Scaling an API: From Reboots to Redis, how (not) to do it
Page 22: Scaling an API: From Reboots to Redis, how (not) to do it
Page 23: Scaling an API: From Reboots to Redis, how (not) to do it

APIs

Page 24: Scaling an API: From Reboots to Redis, how (not) to do it

Database Master/Slave

Page 25: Scaling an API: From Reboots to Redis, how (not) to do it

Client App

Page 26: Scaling an API: From Reboots to Redis, how (not) to do it

Client App

Database Master/Slave

APIs

Page 27: Scaling an API: From Reboots to Redis, how (not) to do it
Page 28: Scaling an API: From Reboots to Redis, how (not) to do it

WordPress

Page 29: Scaling an API: From Reboots to Redis, how (not) to do it

WordPress Pinterest

Page 30: Scaling an API: From Reboots to Redis, how (not) to do it

Client App

Database Master/Slave

APIs

Page 31: Scaling an API: From Reboots to Redis, how (not) to do it

WordPress

Page 32: Scaling an API: From Reboots to Redis, how (not) to do it
Page 33: Scaling an API: From Reboots to Redis, how (not) to do it
Page 34: Scaling an API: From Reboots to Redis, how (not) to do it

SkimWords™

Page 35: Scaling an API: From Reboots to Redis, how (not) to do it
Page 36: Scaling an API: From Reboots to Redis, how (not) to do it
Page 37: Scaling an API: From Reboots to Redis, how (not) to do it

APIxx

Page 38: Scaling an API: From Reboots to Redis, how (not) to do it

APIxx APIxx APIxx APIxx APIxx APIxx APIxx APIxx

CLUSTERxx

ELBELB

Page 39: Scaling an API: From Reboots to Redis, how (not) to do it

CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxxCLUSTERxx

US-WEST US-EAST EU-WEST

Page 40: Scaling an API: From Reboots to Redis, how (not) to do it
Page 41: Scaling an API: From Reboots to Redis, how (not) to do it
Page 42: Scaling an API: From Reboots to Redis, how (not) to do it

Pinterest

Page 43: Scaling an API: From Reboots to Redis, how (not) to do it
Page 44: Scaling an API: From Reboots to Redis, how (not) to do it
Page 45: Scaling an API: From Reboots to Redis, how (not) to do it
Page 46: Scaling an API: From Reboots to Redis, how (not) to do it
Page 47: Scaling an API: From Reboots to Redis, how (not) to do it

JAYxx

Apache Flume

Page 48: Scaling an API: From Reboots to Redis, how (not) to do it

JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx

ELBELB

CLUSTERxx

Apache FlumeCollector

C Prefix Tree C NLP Filterin

g

Page 49: Scaling an API: From Reboots to Redis, how (not) to do it

CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxxCLUSTERxx

US-WEST

US-EAST EU-WEST

Apache FlumeMaster

Page 50: Scaling an API: From Reboots to Redis, how (not) to do it
Page 51: Scaling an API: From Reboots to Redis, how (not) to do it

Cacti

Page 52: Scaling an API: From Reboots to Redis, how (not) to do it
Page 53: Scaling an API: From Reboots to Redis, how (not) to do it

Conclusions

Small Scalable units

Horizontal is always better

Eliminate Single points of failure

Scale writes and reads separately

Automate everything

Monitor everything

Page 54: Scaling an API: From Reboots to Redis, how (not) to do it

Scaling an APIFrom Reboots to Redis, how (not) to do it

Questions?

Page 55: Scaling an API: From Reboots to Redis, how (not) to do it

{ "fullName": "Ciarán Rooney", "workLife": { "position": "CTO" }, "homeLife": { "email": "[email protected]", "twitter": "@CiaranR", "website": "http://ciaran.ie", "hobby": "http://phplondon.org" }}


Recommended