Post on 15-Jan-2016
transcript
Do It In ProductionTesting Where It Counts
Seth EliotSenior Knowledge Engineer, Test Excellence
Services and Cloud
A/B Testing of Services
Digital Media Services
Petabytes Processed
About Seth
Part 3July 2012The Cloud
Part 2March 2012TestOps
Part 1Nov 2011Testing in Production
The Future of Software Testing
Measurement
A quantitatively expressed reduction of uncertainty
…based on one or more observations
Testing
…about quality of a system under test…
Data Driven Decision Making
Data Driven Validation
Testing in Production
TiPReal Users
Production Environments
Real World Performance Monitoring
500 Million measurements per month
JSIJavaScript Instrumentation
What is Big Data?
Value
VelocityVolume
MB GB TBPB EB ZB
Variety
TwitterXbox Kinect
What Does Data Have To Do With TiP?
Biggest and Best Data is in Production
Testing in Production (TiP)
Use Data for Quality Assessment
Production is Truth
Why Testing in Production (TiP)?
Find this with a unit test….
“Walking Directions”
This route may be missing sidewalks
5 Million Metrics
Grid Report:CPU, Network…
Operational Data Store (ODS)
System…Application…Business……Metrics
Historical trending and analysisHadoop
Metrics find…Feature broken for significant percent of users
Metrics find…Problems at Scale
Constant dogfooding with reporting tools
Engineers stay with code every step of the way
“This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software” Really?
Just Throw it in Production?
Integration Testing
Functional TestingUnit Testing
Metrics and Optics
…maybe less of thisTesting in Production
Functional Testing
Integration Testing
Performance and ScalabilityTDD
InstrumentEverything
Real … Users… Applications… Infrastructure
Synthetic Transactions
Active ValidationPassive Validation
AvailabilityReliabilityPerformance
Operational Intelligence
Business Intelligence
Two Types of Data-Driven Validation
AvailabilityPerformance
From Azure
Fault Injection
Chaos Monkey
Latency Monkey
Game Day
Chaos Gorilla
Synthetics Have Risks
Who's Got the Monkey?
Who's Got the Monkey Now?
Monkey
Synthetics Have Risks
Provides insight into real usage
Reproducible and well understood scenarios
Covers a vast variety of environments
Requires proper handling of Personally Identifiable Information (PII)
May adversely alter production and production data
P
P
P
A
A
You’ve learned a lot already…
Experimentation
Exposure Control
Dogfood and Beta
A/B Testing
“To have a great idea, have a lot of them”
-Thomas Edison
Experimentation at Google
“…dice and slice in any way you can possibly fathom”
1% launches
Shadow launches
1/3
2/3
Experimentation at Netflix
1B API requests per day
Canary Deployment
A brief look at data science
Beware of averages
Even small data is useful data
Science!
Watch out for sample bias
How many years have you worked in software?
Averages are Your Enemy
Tools of Big Data: Hadoop
Hadoop HDFS
ADDFCAECC
ADD FCA
ECC
ADDADD
ADD
FCAFCA
FCA
ECCECC
ECC
Map-Reduce
1xA1xC1xF
1xA2xD
1xE2xC
2xA0xB3xC2xD1xE1xF
Tools of Big Data: Cosmos
Processing 2PB per day
On tens of thousands of computers
Stores hundreds of petabyte
“We know we can't anticipate the 101 things that will go wrong, The only thing we can control is ensuring our team responds appropriately to those situations.“ – Jerry Hook, Executive Producer Halo
…Hundreds of thousands of requests per second
Hack using modified Xbox
Halo ServersXbox Console SharepointPower Pivot Action
Target
“But even if you’re following the law, you can do things where people get queasy.”
HDInsight
Hadoop on Azure
HDInsight
Availability (y) over time (x)
Predict 75% of dips 24 hours ahead of timeData
c
Machine Learning
c
Cosmos
c
Passive
Active
The best observations are often in production
A set of observations to reduce uncertainty about quality of a system under testTesting
blog: http://bit.ly/seth_qa
Do It In Production
Testing Where it Counts
seth.eliot@microsoft.com
me….
@setheliot
?
Thank you!