Go90
Vidhya Narayanan, Director, Verizon Christopher Webster, Associate Fellow, Verizon
October 2015
A Look inside go90 services architecture
Micro service is a…
a. new taxi hailing app developed by a middle schooler in India
b. catering service run by bacteria
c. microwave-‐as-‐a-‐service from Verizon
d. Complex applica@on broken down into separate services
©2015 Apigee Corp. All Rights Reserved.
The views expressed in this presenta@on are those of the presenter, and not
necessarily those of Apigee Corpora@on or the presenter’s employer.
3
Go90 — from 30,000 P
• 100+ micro services/systems
• 10+ deployments per week
• Mul@ple clients, large target user base
• High reliability/availability, low latency, superior user experience
Goals
• Must scale and be highly available • Support mul@ple video products • Meet security requirements • Need to use exis@ng services
Solu@on
• Services with contained domain logic • Resource centric APIs that expose data model • Common libraries • Graceful degrada@on • Security readiness
Accounts
Log Upload
Search
Profile Image
Discovery
Event Q
DRM
Analy@cs
Favorites
Cert Prov
AWS HSM
Ads
Offers and Packages
DAI backend
STB
Content Ingest
CDN
Reco
Discovery
Accounts
Favorites
Search
DRM
Profile
Rights
SU
and…a 100 others
Profiles
Profiles Q
Added Goals
• Reduce development @me for new client placorms
• Improve release velocity • SLA for call latency • Keep security simple
Governor
Accounts
Log Upload
Search
Profile Image
Discovery
Event Q
DRM
Analy@cs
Favorites
Cert Prov
AWS HSM
Ads
Offers and Packages
DAI backend
STB Maestro
Content Ingest
CDN
Reco
Discovery
Accounts
Favorites
Search
DRM
Profile
Rights
SU
and…a 100 others
Profiles
Profiles Q
Our Approach
Model Predicted Load
Develop and
Execute Load Test
Op@mize / Scale for Load Test
Validate Model against
known data
Our Approach
• Benchmark individual APIs • Develop model • Execute cycle on system • Verify model • Launch • Use produc@on traffic logs • Inject failure during load tests
Scenario Runs
• First run • Op@mized app call pagern • Model itera@on using earlier results and expected users projec@ons
• Execute app scenario (on en@re system) with larger number of users, but s@ll 10x less than target
Validate Scenarios
• Increased scenario to target concurrent users • Added think @me to scenario • Ran scenario with target users • Scenario running with load < .1 • Chaos Monkey