Testing made Socialcam successful!•16 Million Downloads

•#1 Free App•Acquired by Autodesk

What is A/B testing?The scientific method applied to product development

Intuition is a bad way to solve complex problems!

How to run A/B test?•Ask a question

•Construct a hypothesis

•Test your hypothesis with an experiment

•Analyze the results and draw a conclusion•Repeat

1) Ask a question =Decide what you're trying to improve

•Pick a single metric

•Low hanging fruit first

•Get the team on board

•Have good stats before you start

At Socialcam we started with the account creation flow

2) Construct a hypothesis =Build things that might improve your metric

•Simplify, remove steps

•Small optimizations or large shifts

•Number of variants depends on usage

At Socialcam, we:•Added email / password login

•Added twitter login•Tried many designes of the initial view in the app

3) Test your hypothesis with an experimentChoose analytics and testing systems, deploy!

3rd party products•https://www.optimizely.com

•http://apptimize.com

Write-up of our system athttp://blog.socialcam.com/mobile-ab-testing-made-easy

4) Analyze the data and draw a conclusionPick a winner!

Humans are bad at statistics!You have to check statistical significance

Beware repeated significance testing errors!

5) Repeat!

At Socialcam, we got the account creation rate to 75%

When You Can't A/B Test•When you don't have any users

•When external services are involved•A totally new product

Takeaways•Testing made Socialcam blow up

•It’s the scientific method

Happy Testing!

ammon@socialcam.com