Download - Re:Inventing your Innovation Cycle by Scaling Out with Spot Instances (CPN207) | AWS re:Invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Weston Jossey

November 13th, 2013

Choosing Our SpotsHow Tapjoy Saves with Amazon Spot Pricing®

Friday, November 15, 13

A Brief Introduction


Who Am I?• Engineer turned DevOps junkie• Multiple Areas of Focus

• Performance• Cost Management & Reduction• Efficiency• As Needed Firefighter


Tapjoy’s Challenge• Premier Mobile Ad Network Across iOS & Android• Global Network (435 Million Monthly Reach)• Diverse user base (54% Male, 46% Female)• Billions of requests per day• Growth requires iteration, experimentation, and massive

scale• Small mistakes are magnified to millions of consumers

instantly


Scale HorizontallyEverywhere


What Everyone Sees1. Develop a killer product, hire more engineers2. Expand product as fast as possible, gain market

share3. Iterate, grow, refine, scale...4. #Profit


What Everyone Forgets to Mention• Engineering is a game of tradeoffs. Fast growth

comes at a price.• Testing and QA are often the first to lag behind.• Building quality test beds for engineers can be

complicated.


Food for Thought• Where are your bottlenecks in your development

cycle?• How do you simulate production?


Spot Instances: The Dremel of Engineering®


The Mindset Required• Engineer application level redundancy and fault

tolerance.• Spread yourself amongst many zones, potentially even

many regions.• Identify areas of required, not preferred, persistence.• Understand how your “neighbors” are utilizing your

preferred instance type.


Always Be Testing• Lots of engineering means...

• Lots of code pushes• Lots of pull requests• Lots of automated tests to be run frequently to prevent regressions

• Full test suites can take > 1 hour when run serially.• Can slow down the review process when changes

require at minimum an hour lag for automated sign-off.


Scale Horizontally With Spot Instances • Jenkins + Spot Instances

• https://github.com/bwall/ec2-plugin

• Go wide during business hours, scale back in the evenings. Automatically kicks online at 06:00ET

• Workers scale horizontally to support dozens of simultaneous regression tests spread out over dozens of workers

• Jenkins automatically guards against spot termination


https://github.com/bwall/ec2-plugin

https://github.com/bwall/ec2-plugin

Tapinabox• Born from Tapjoy Hackathon

• Quarterly hackathons to promote engineering growth and creativity.• Question Posed: “What if spinning up a fresh QA was as easy as launching

a app server on Heroku?”

• Goals• Behaves, operates, and performs like production.• Should be dead-simple to use. “Push a button, get a server.”• Save money through improved quality and low cost to operate.• End-To-End Product. No mocks, no stubs.


Tapinabox Cont.• Fully automated build & deployment process tied to a

developers Github account. • Simple web interface for creating, managing, and

editing instances.• Quick collaboration with Product Managers, remote

QA, and partner testing.• Fully run on Spot Instances.




How we Built Tapinabox• Ruby on Rails• AWS CLI• Chef• Amazon EC2 Spot Instances (zone & region agnostic)• Tapjoy Slugs: Custom slug-deployments based on

FPM https://github.com/jordansissel/fpm


https://github.com/jordansissel/fpm

https://github.com/jordansissel/fpm

Pointers & Lessons Learned• Spot Instances can take a while to provision with a different

workflow than a traditional on-demand model.• Very important to find quiet zones / regions based on your

workload. • Pricing sometimes takes a back seat to reliability. What

instance types have contention?• Guard against termination with adequate pricing, but don’t try

and prevent it. Automation is key. The price will eventually fall.• Pick the right tool for the job. Don’t get greedy!


Kudos• Tapjoy Automation Team

• John Russell• Chris Gerber• JLo• Adam Bell• Hugh Barrigan


Thank You

[email protected]@dustywes


mailto:[email protected]

mailto:[email protected]

Please give us your feedback on this presentation

As a thank you, we will select prize winners daily for completed surveys!

CPN207 Thank You
