Do It In ProductionTesting Where It Counts

Seth EliotSenior Knowledge Engineer, Test Excellence

Services and Cloud

A/B Testing of Services

Digital Media Services

Petabytes Processed

About Seth

Part 3July 2012The Cloud

Part 2March 2012TestOps

Part 1Nov 2011Testing in Production

The Future of Software Testing

Measurement

A quantitatively expressed reduction of uncertainty

…based on one or more observations

Testing

…about quality of a system under test…

Data Driven Decision Making

Data Driven Validation

Testing in Production

TiPReal Users

Production Environments

Real World Performance Monitoring

500 Million measurements per month

JSIJavaScript Instrumentation

What is Big Data?

Value

VelocityVolume

MB GB TBPB EB ZB

Variety

TwitterXbox Kinect

What Does Data Have To Do With TiP?

Biggest and Best Data is in Production

Testing in Production (TiP)

Use Data for Quality Assessment

Production is Truth

Why Testing in Production (TiP)?

Find this with a unit test….

“Walking Directions”

This route may be missing sidewalks

5 Million Metrics

Grid Report:CPU, Network…

Operational Data Store (ODS)

System…Application…Business……Metrics

Historical trending and analysisHadoop

Metrics find…Feature broken for significant percent of users

Metrics find…Problems at Scale

Constant dogfooding with reporting tools

Engineers stay with code every step of the way

“This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software” Really?

Just Throw it in Production?

Integration Testing

Functional TestingUnit Testing

Metrics and Optics

…maybe less of thisTesting in Production

Functional Testing

Integration Testing

Performance and ScalabilityTDD

InstrumentEverything

Real … Users… Applications… Infrastructure

Synthetic Transactions

Active ValidationPassive Validation

AvailabilityReliabilityPerformance

Operational Intelligence

Business Intelligence

Two Types of Data-Driven Validation

AvailabilityPerformance

From Azure

Fault Injection

Chaos Monkey

Latency Monkey

Game Day

Chaos Gorilla

Synthetics Have Risks

Who's Got the Monkey?

Who's Got the Monkey Now?

Monkey

Synthetics Have Risks

Provides insight into real usage

Reproducible and well understood scenarios

Covers a vast variety of environments

Requires proper handling of Personally Identifiable Information (PII)

May adversely alter production and production data

P

P

P

A

A

You’ve learned a lot already…

Experimentation

Exposure Control

Dogfood and Beta

A/B Testing

“To have a great idea, have a lot of them”

-Thomas Edison

Experimentation at Google

“…dice and slice in any way you can possibly fathom”

1% launches

Shadow launches

1/3

2/3

Experimentation at Netflix

1B API requests per day

Canary Deployment

A brief look at data science

Beware of averages

Even small data is useful data

Science!

Watch out for sample bias

How many years have you worked in software?

Averages are Your Enemy

Tools of Big Data: Hadoop

Hadoop HDFS

ADDFCAECC

ADD FCA

ECC

ADDADD

ADD

FCAFCA

FCA

ECCECC

ECC

Map-Reduce

1xA1xC1xF

1xA2xD

1xE2xC

2xA0xB3xC2xD1xE1xF

Tools of Big Data: Cosmos

Processing 2PB per day

On tens of thousands of computers

Stores hundreds of petabyte

“We know we can't anticipate the 101 things that will go wrong, The only thing we can control is ensuring our team responds appropriately to those situations.“ – Jerry Hook, Executive Producer Halo

…Hundreds of thousands of requests per second

Hack using modified Xbox

Halo ServersXbox Console SharepointPower Pivot Action

Target

“But even if you’re following the law, you can do things where people get queasy.”

HDInsight

Hadoop on Azure

HDInsight

Availability (y) over time (x)

Predict 75% of dips 24 hours ahead of timeData

c

Machine Learning

c

Cosmos

c

Passive

Active

The best observations are often in production

A set of observations to reduce uncertainty about quality of a system under testTesting

blog: http://bit.ly/seth_qa

Do It In Production

Testing Where it Counts

seth.eliot@microsoft.com

me….

@setheliot

?

Thank you!