© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Lucy Chang

Sr. Software Engineer in Quality, Intuit

CI/CD with Mocking &

Resiliency Testing Using AWS

Alfred Tan

Sr. DevOps Manager, Intuit

Session Overview

• CI/CD and AWS onboarding

• Cross team AWS strategy alignment

• Automating AWS deployment

• Development work is blocked by dependency

• Integration tests fails due to unreliable dependency

• Need to do resiliency testing

CI/CD in AWS

Our Challenges

• Security requirements

• Onboarding into CI/CD

• Highly Available

Our Solution - Slingshot

• Build security in

• Automate onboarding to CI/CD

• Build HA/DR in

CI/CD Pipeline

Continuous Integration

Continuous Delivery/Deployment Pipeline

Promotion Criteria:

• Build pass: 100%

• Unit Test pass: 100%

• Code Coverage: >80%

Build

Promotion Criteria:

• BAT pass: 100%

CI

Promotion Criteria:

• Regression Test

pass: 100%

QA

Promotion Criteria:

• E2E Test pass:

100%

Test Run:

• E2E Test

• Performance Test

E2E/Perf

Test Run:

• Smoke Test pass: 100%

Prod

Slingshot Setup

Initial Setup

GitHub Repo

CI/CD Pipeline

KMS/SSH keys

S3 Buckets

Egress Proxy and Bastion Host

Splunk Forwarder

AWS Account SetupAWS Account

VPC, Subnets, Routing tables, Route 53 Zone Delegation

One time events

Slingshot Initial Setup

Region US-WEST-2

Bucket for Artifacts KMS Key for Secrets KMS Key for EBSBucket for Secrets

Internet

GatewayBastion

ASG

Splunk

Forwarder

Egress

ASG

Public Bastion Subnets Public Egress Subnets

Private DB Subnets

Private APP Subnets

Public ELB Subnets

Private WEB Subnets

Delegated DNS Zones

Slingshot Setup

Initial Setup

GitHub Repo

CI/CD Pipeline

KMS/SSH keys

S3 Buckets

Egress Proxy and Bastion Host

Splunk Forwarder

CI/CD

ELB

Web Tier

App Tier

CNAME

Recurring events

AWS Account SetupAWS Account

VPC, Subnets, Routing tables, Route 53 Zone Delegation

One time events

CD with Blue-Green Deployment

ci-svc.intuit.com qa-svc.intuit.com svc.intuit.com

PreProd

Account

Prod

Account

Public ELB

Subnets

Private Web

Subnets

Private App

Subnets

100% 5%95% 100%0%

CI Web

Build 10

CI App

Build 10

CI Web

Build 12

CI App

Build 12

QA App

Build 10

QA Web

Build 10

QA App

Build 10

QA Web

Build 12

QA App

Build 12

Prod App

Build 10

Prod Web

Build 10

Prod App

Build 10

Prod Web

Build 12

Prod App

Build 12

Benefits

• Early feedback on changes flowing through the system

• Increase in quality

• Frequent releases to production

• Development productivity from day 1

Slingshot Demo

Recap

Challenges

• Security requirements

• Onboarding into CI/CD

• Highly Available

The Solution: Slingshot

• Build security in

• Automate onboarding to CI/CD

• Build HA/DR in

The Next Problem

• Automation tests failed due to unreliable dependency server

• Builds are not promoted

Our Solution

Overview of Wiremock

Wiremock is a library for stubbing and proxying web services

• Stubbing

• Fault Injection

• Easy Set up

• Easy onboarding

How does Wiremock work?

System

Under Test

Wiremock

Server

Dependency

ServerAutomation

Test

• Configure the Wiremock server to be man-in-the-middle

• Increased integration test pass rate

• Increased code coverage

• Does not interrupt other team’s calling the SUT

= Stubs

Before Wiremock

SubnetELB for SUT

SUT 1

SUT 2

Dependency

Server

Subnet

Automation

Test

We tried this

SubnetELB for SUT

SUT 1

SUT 2Subnet

Automation

Test

• Deploy Wiremock on SUT EC2 instance

• No consistent stub response!

= Stubs

Our Solution

ELB for WM

SUT 1

SUT 2Automation

Test

Wiremock(Stub

Dependency)

ELB for SUT

• Deploy Wiremock on dedicated EC2 and ELB

• Consistent Stub responses!

= Stubs

If no stubs…

ELB for WM

SUT 1

SUT 2Automation

Test

Wiremock

ELB for SUT

Dependency

Server

WM will proxy the

request to

dependency

server

Wiremock Code Snipets

Starting Up Wiremock Server

java -jar wiremock-1.53-standalone.jar --verbose --port 8080 --proxy-all=[Dependency Server DNS Name]

Stubbing the response

//This calls Wiremock API to stub the response

stubFor(get(urlEqualTo(“/from/where”))

.willReturn(aResponse().withStatus(200)

.withHeader("Cache-Control", "no-cache")

.withHeader("Content-Type", ”text/plain")

.withBody(“Taiwan” )));

Simulating Fault

//This calls Wiremock API for fault injection

stubFor(get(urlEqualTo(“/some/thing”))

.willReturn(aResponse()

.withFault(Fault.EMPTY_RESPONSE)));

Benefits

• We fixed the CI/CD pipeline

• No more unnecessary test failures debugging

• Less production escapes and firefighting

Recap

The Second Challenge

• Integration tests failures broke CICD pipeline

• Hard to do resiliency testing

The Solution : Wiremock

Next Step

Why don’t we combine them?

Slingshot With Wiremock

Slingshot with Wiremock

W

ASG

Web

ASG

App

ASG

System Under Test

Region US-WEST-2

Availability Zone #1

Wiremock

ASG

Wiremock

Internet Gateway

Dependency

Server

AWS Region X / Datacenter X

Automate WM Deployment

Automate Wiremock Deployment

Chef is an infrastructure automation code tool we use

• Code how you deploy and manage your infrastructure

• Allows version control

• Code can be reused

Automate Wiremock Deployment

We wrote a Wiremock Recipe

• Download the Wiremock jar

• Start up the Wiremock server

Automate Wiremock Deployment

Berkshelf is a dependency manager for chef

• Get the Java recipe to download Java

• Get the Wiremock recipe to deploy Wiremock server

Chef Snipet

#This will start the wiremock server with the parameters passed in

function start { cd "${USER_DIRECTORY}" ;java -jar wiremock-${WIREMOCK_VERSION}-standalone.jar --port ${PORT} --proxy-via ${PROXY_VIA} -–proxy-all= ${PROXY_ALL} --verbose > /var/log/wiremock.log 2>&1 & }

Automate Wiremock Stack Creation

Use AWS CloudFormation API

• Provision EC2 instances and ELB

• Create Auto Scaling Group

• Set up other AWS resources

Use WireMock in Slingshot

Call Chef from CloudFormation

How to Call Cookbook From CloudFormation

Write shell scripts In the InstanceLaunchConfig section

1. Download and install chef

2. Run Chef. In this case we created a Wiremock role to

execute the java and Wiremock cookbooks.

Call Chef from CloudFormation

"5_run_chef": {

"command": { "Fn::Join": [ "", [ "/usr/bin/chef-solo -c /var/chef/config/solo.rb -o 'role[", { "Ref": "Role" }, "]' -E '", { "Ref": "Environment" },"'" ] ]

}

Benefits

• A simplified CI/CD pipeline onboarding

• A successful CI/CD pipeline with increased test pass rate

• Resiliency testing capability built in

• Security features built in

Deep Dives

Demo

SUT Wiremock

ServerYelpTest

Automation

Recap

Recap

Combined Solution: Slingshot with Wiremock

• CI/CD pipeline easy onboarding

• Builds are auto-promoted

• Less Engineers’ time spent on debugging

• Resiliency issue found before production

• Happy Engineers

What we learned

What We Learned

• The initial investment is worth it

• Try to be flexible

• Set up DNS

References

• http://www.pnsqc.org/the-journey-of-mocking-in-aws/

• http://wiremock.org/

• https://www.chef.io/

• http://docs.aws.amazon.com/AWSCloudFormation/latest

/APIReference/Welcome.html

Related Sessions

Breakout Session:

ARC344

How Intuit Improves Security and Productivity with

AWS Virtual Networking, identity, and Account

Services

Track: Architecture

Session Level: Advanced (300 level)

Session Time: Thursday, Oct 8, 2:45 PM – 3:45 PM–

Palazzo

Contact

Lucy Chang

Sr. Software Engineer in Quality , Intuit

https://www.linkedin.com/pub/lucy-chang/11/312/a83

Alfred Tan

Sr. DevOps Manager, Intuit

https://www.linkedin.com/pub/alfred-tan/1/938/9b

Thank you!

Remember to complete

your evaluations!