Build The Right
Stuff.Workshop by Adam Scott & Arti Mathanda
ADAM SCOTT
@adamacs
ARTI MATHANDA
@artismarti
SIMPLY BUSINESS
❖ Britain’s biggest business insurance provider
❖ Selling insurance online for 10 years
❖ 300,000+ customers and counting
❖ Iterating on Agile & Lean methodologies for 4.5 years
I HAVE AN IDEA!
BUT…
WHERE DO YOU
WANT TO GO?
BUILD
SHOULD YOU BUILD IT?
Why? ---> Why? ---> Why? ---> Why? ---> Why?
BUILD
CAN YOU BUILD IT?What? & Who? & How?
BUILD
WILL YOU BUILD IT?When?
LEAN START UP
- Build
- Measure
- Learn
BUILD
DEFINE EXPERIMENT
BUILD
HYPOTHESIS:We believe that by doing x we will get benefit y.
ex. We believe that by making the Call To Action button
more prominent, we will get a better conversion rate
from Page 1 to Page 2
BUILD
BAD HYPOTHESIS (a fact or a given)
BUILD
GOOD HYPOTHESIS (it can be true or false)
BUILD
BASELINE:
Currently 25% conversion from Page 1 to Page 2..
BUILD
EXPECTED RESULTS:
Page 1 to Page 2 conversion increases by 10% points.
BUILD
MEASURE FOR:
2 weeks.
(Don’t forget Statistical Significance).
BUILD
GOOD MEASURES
BUILD
BAD MEASURES
BUILD
ALSO KEEP AN EYE ON:
Site Performance
Overall Conversions
Seasonal effects
Bounce Rates
etc etc etc
MEASURE
ACTUAL RESULTS:
Page 1 to Page 2 conversion increased by 3% points
MEASURE
IS IT STATISTICALLY SIGNIFICANT?
Yes!
LEARN
WHY DID IT FAIL?
LEARN
WHAT OTHER METRICS ARE AVAILABLE
TO ANALYSE THIS?
LEARN
TRY SOMETHING DIFFERENT?
LEARN
REVERT CHANGE?
LEARN
LEAVE IT RUNNING?