Testing made Socialcam successful!•16 Million Downloads
•#1 Free App•Acquired by Autodesk
What is A/B testing?The scientific method applied to product development
Intuition is a bad way to solve complex problems!
How to run A/B test?•Ask a question
•Construct a hypothesis
•Test your hypothesis with an experiment
•Analyze the results and draw a conclusion•Repeat
1) Ask a question =Decide what you're trying to improve
•Pick a single metric
•Low hanging fruit first
•Get the team on board
•Have good stats before you start
At Socialcam we started with the account creation flow
2) Construct a hypothesis =Build things that might improve your metric
•Simplify, remove steps
•Small optimizations or large shifts
•Number of variants depends on usage
At Socialcam, we:•Added email / password login
•Added twitter login•Tried many designes of the initial view in the app
3) Test your hypothesis with an experimentChoose analytics and testing systems, deploy!
3rd party products•https://www.optimizely.com
•http://apptimize.com
Write-up of our system athttp://blog.socialcam.com/mobile-ab-testing-made-easy
4) Analyze the data and draw a conclusionPick a winner!
Humans are bad at statistics!You have to check statistical significance
Beware repeated significance testing errors!
5) Repeat!
At Socialcam, we got the account creation rate to 75%
When You Can't A/B Test•When you don't have any users
•When external services are involved•A totally new product
Takeaways•Testing made Socialcam blow up
•It’s the scientific method