Building Agile Applications using

Continuous Integration & Deployment

on AWS

Leo Zhadanovsky

Senior Solutions Architect

CONTINUOUS

INTEGRATION

What is Continuous Integration?

Changes to code automatically deployed to mainline branch• After passing unit and mock tests

Makes changes to code, and deployments iterative, not monolithic

Bugs are detected quickly

Allows rapid development

Helps automate deployments

DEVELOPER

SOURCE CODE

REPOSITORY

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

PICK

TASKS

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

SUBMIT

CODE

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

SCHEDULE

BUILD

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

RECURRENT

BUILDS

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

CODE

FETCH

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

CODE QUALITY

TESTS

TEST

RESULTS

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

BUILD OUTPUT

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

DOCS

BINARIES

& PACKAGES

DEV FACING

NOTIFICATIONS

SOURCE CODE

REPOSITORY

DNS

CONTINUOUS

INTEGRATION SERVER

PROJECT

MANAGEMENT SERVER

BUILDS

New AWS Managed Options

CodeDeploy• Deploy your code to hundreds or thousands of instances

CodeCommit• AWS Managed Git Hosting

CodePipelines• AWS Managed Continuous Integration System

Cloud Software Development Lifecycle

10/13/14 19

MonitorProvisionDeployTestBuildCode

Elastic Beanstalk

OpsWorks

Cloud

Watch

Cloud

Formation

?

Why?

Customers are asking

Share Amazon tools and processes with customers

Showcase and integrate AWS partner tools

9/25/14 20

Source ControlPrivate Git repositories hosted on Amazon S3

Full Git compatibility (use with existing tools)

All the benefits of the cloud(scalable, durable, reliable, low pay as you go pricing)

No size limits on repositories (store binary files)

Online code tools with browse, edit, diff

10/13/14 21

Continuous DeliveryCustomizable release automation, with integrated build and test

Model and visualize custom release workflow(source build beta gamma prod)

Automate builds, tests, and deployments

Enforce custom rules, approvals, and gates

Integrate with third-party and custom tools

10/13/14 23

DeploymentCoordinate software updates to fleets of EC2 instances

Rolling updates for no downtime

Deployment health checks and easy rollback

Auto Scaling integration

Works with any application

Reuse existing setup tools(Bash, Powershell, Chef, Puppet…)

10/13/14 25

9/25/14 Slides not intended for

redistribution.

26

Cloud Software Development Lifecycle

10/13/14 27

MonitorProvisionDeployTestBuildCode

Elastic Beanstalk

OpsWorks

Cloud

Watch

Cloud

Formation

Code

Deploy

Code

Commit

Code

Pipeline

PAIN POINTS• UNIT TESTS INCOMPLETE

• MOCK TESTS MAINTENANCE

• EXPENSIVE TEST ENVIRONMENT

• TEST ENVIRONMENT ≠ PRODUCTION

• DEPLOYMENT CYCLES

ON-DEMAND

PAY AS YOU GO

ELASTIC

=

PROGRAMMABLE

PLATFORM

IF YOU CAN PROGRAM IT

YOU CAN AUTOMATE IT

A lot of options…

Configuration Management Systems• Puppet• Chef• Saltstack

Deployment Frameworks• Elastic Beanstalk• OpsWorks• Ansible• Fabric• Capistrano

Infrastructure Management• CloudFormation

Bake an AMI Configure dynamically

Time consuming configuration (startup time)

Static configurations (less change management)

Bootstrapping

Bake an AMI Configure dynamically

Continuous deployment (latest code)

Environment specific (dev-test-prod)

Bootstrapping

Obama for America

awsofa.info

Web-Scale Applications

500k+ IOPS DB Systems

Services API

Typical Charts

How?

IngredientsUbuntu nginx boundary Unity jQuery SQLServer hbase NewRelic

EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3

ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire

nagios Paypal CentOS CloudSearch levelDB mongoDB python

securitygroups Usahidhi PostgresSQL Github apache bootstrap

SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS

Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch

ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS

Objective-C DynamoDB

Infrastructure, Configuration Management & Monitoring

Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic

EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3

ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire

nagios Paypal CentOS CloudSearch levelDB mongoDB python

securitygroups Usahidhi PostgresSQL Github apache bootstrap

SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS

Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch

ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS

Objective-C DynamoDB

One thing that is difficult to prepare for…

They had this built for the previous 3 months, all on the East Coast.

They had this built for the previous 3 months, all on the East Coast.

We built this part in 9 hours to be safe.

AWS +Puppet +

Netflix Asgard + WAN Optimization Software +

DevOps =

Cross-Continent Fault-Tolerance On-Demand

Mozilla Foundation

Webmaker.org circa 2012

• Included Apps, non-SOA: Thimble, Popcorn, Goggles

• ~20 pushes of new software in 2012

• Operations and Development interacted mostly through bugzilla

tickets for deploys.

• Hosting in physical datacenter at Mozilla

Webmaker.org circa early 2013

• Deciding to go 12-factor, SOA in app layer

• Weekly pushes of Popcorn on train model

• Operations and Development interacted mostly through bugzilla

tickets for deploys.

April 2013

Webmaker begins rebuilding entire platform

SOA, 12-factor in node.js exclusively

Moving apps into AWS and DevOps / CI

Since April 2013….Openbadges, Webmaker combine for: 1339 Pushes

Pushes Per Day to Staging / Prod

Pushes Per Day (Staging and Prod)

Who?

• ~30 Paid Developers

• Hundreds of Students

• Thousands of Contributors

• One DevOps / Internet Jedi

• Multiple Teams

How?

• Puppet, Jenkins, Fabric

• Tight feedback loops:

Newrelic, Opsview

• Culture Shift

• Staging Envs

• Brave devs iterate,

keeping work in-context

• Visible Ops

• Cross-train developers in

operations

What changed?

1) Know How You Were Doing Before

2) Know What Changed When

3) Know How You Are Now Doing

=

The confidence to try more things

and try them faster, with minimum

viable planning.

Deployment

Pipeline

AWS

CLOUDFORMATIONSTACK-BASED DEPLOYMENT

SERVICE

CLOUDFORMATION

TEMPLATE

{"Description" : "Create RDS with username and password","Resources" : {

"MyDB" : {"Type" : "AWS::RDS::DBInstance","Properties" : {

"AllocatedStorage" : "500","DBInstanceClass" : "db.m1.small","Engine" : "MySQL","EngineVersion" : "5.5","MasterUsername" : "MyName","MasterUserPassword" : "MyPassword"

}}

}}

"AWS::CloudFormation::Init" : { "config" : {

"packages" : {"yum" : {

"mysql" : [],"mysql-server" : [],"httpd" : [],"php" : [],"php-mysql" : []

}},"sources" : {

"/var/www/html" :"https://s3.amazonaws.com/my-builds/build-v4.zip"

}}

{"Parameters" : {"KeyName" : {

"Description" : "Name of an existing EC2KeyPair to enable SSHaccess to the instance",

"Type" : "String"}

},}

CLOUDFORMATION

TEMPLATE

PROCEDURAL

DEFINITIONCreate it programmatically

KNOWN

CONFIGURATIONStore stack configuration in

source control

PARAMETER

DRIVENDynamic and user-driven

templates

COLLABORATIONShare templates with ease

as just files

APPLICATION

VERSIONS

+INFRASTRUCTURE

VERSIONS

CLOUDFORMATION

TEMPLATE

LOAD TESTING

Benchmarking

US-East Cache Node Performance 25.3 Gbps

Only ~42MbpsImpact on US-East FMS Origin Servers

CONTINUOUS

DEPLOYMENTSMALL, FREQUENT CHANGES

CONSTANTLY INTEGRATING INTO

PRODUCTION.

KEY = ITERATION

ITERATION

=MODIFY THE SYSTEM TO BETTER

MEET THE EXPECTATIONS OF

YOUR USERS

11.6s

Mean time

between

deployments

(weekday)

1,079

Max number of

deployments in a

single hour

10,000

Mean number of

hosts

simultaneously

receiving a

deployment

30,000

Max number of

hosts

simultaneously

receiving a

deployment

DEPLOYMENTS AT

AMAZON.COM

SOFTWARE DEPLOY

≠PRODUCT LAUNCH

1.5 BILLION PAGE VIEWS

OCTOBER 2012

$83 MILLION IN TRANSACTIONS

4.2 MILLION ITEMS SOLD

30 DEPLOYS PER DAY1 DEPLOY EVERY 20 MINUTES

"Production is truly the only place you

can validate your code."

DATA-DRIVEN

ARCHITECTURES

METRICS @ETSY

METRICS @OBAMA FOR AMERICA

COST-ORIENTED

ARCHITECTURES

PHP+APACHE+VARNISH

NGINX+NODEJS

CONTINUOUS

INTEGRATION

CONTINUOUS

DEPLOYMENT

CONTINUOUS

DEPLOYMENT

=

CONTINUOUS

EXPERIMENTATION

CONTINUOUS

DEPLOYMENT

=

CONTINUOUS

IMPROVEMENT

INNOVATE

« Want to increase innovation?

Lower the cost of failure »

Joi Ito

SPEED AND AGILITY

Experiment

Often

Fail quickly at

a low cost

More

Innovation

Experiment

Infrequently

Failure is

expensive

Less

Innovation

“ON-PREMISE”