1
@PaulCutsinger
Russ Carroll
A|B Testing
2
3
4
Surface the Value
Image from Diner Dash by PlayFirst
Source: Amazon Appstore, January 2013
+75% ARPPU
5
Show Users How to “Consume”
Image from Cut the Rope by ZeptoLab
Source: Amazon Appstore, January 2013
+65%
repeat orders
6
Experiment and react quickly
7 day retention
Time
Active Players
7
A|B Testing
8
This is Air Patriots
Case Study: Store Optimization
10
• Goal – Easier to access store content
• Platform – Android & iOS
• Test – Make it possible to unlock content on Main Menu
STORE OPTIMIZATION
11
LOCKED ON MAIN MENU
UNLOCKABLE >
<
12
STORE OPTIMIZATION
Test Case Conversion to
Customer 7-Day Retention
Locked on Main Menu
(Control) – –
Unlockable on Main
Menu 20.93% 4.17%
13
• Goal – Drive players to our other games
• Platform – Android
• Test – Which image is clicked on more
CROSS PROMOTION
CROSS PROMOTION
15
CROSS PROMOTION
16
Air Patriots: Cross Promotion
B
A 1.77%
0.93%
Clic
k T
hro
ug
h
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.89% +3.07%
43,120
The Producer’s Perspective Ads on menu or in game
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads
The Producer’s Perspective
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads Goals
The Producer’s Perspective
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads Goals
The Producer’s Perspective
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads Goals
The Producer’s Perspective
22
Retrieve a variation
Objective C
23
Record a view and conversion Objective C
bit.ly/ABTestXPlat
bit.ly/ABTestiOS
bit.ly/ABTestJava
More details at
Case Study: Junk Race by GameNug
Junk Race Overview
27
Tips: Producers in control: “With two-way services you can request a value from the
server that you control. Two-way services let you make changes to your live game
instantly.
Free: Amazon is the only free two-way service that we have found.”
Multi-variable: “We don't test variables, we test user experiences.”
Watch sample size: “If a couple users can change your conclusion, you need a
larger sample”
Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of
skipping the week long review period ”
-John Goforth
28
Case Study: Difficulty Level
29 Amazon Services Confidential
GAME DIFFICULTY TEST
7 day retention
Time
Active Players
30 Amazon Services Confidential
GAME DIFFICULTY TEST
• Goal – A Dramatic Increase!
• Platform – Android & iOS
• Test – A/B/n Test – 5 Difficulty levels!
OR?
31 Amazon Services Confidential
GAME DIFFICULTY TEST
32 Amazon Services Confidential
Test 1:
A
Control
B
2% easier
C
4% easier
D
6% easier
E
8% easier
7-Day Retention – tbd tbd tbd tbd
Conversion to customer – -1.71% 16.11% 19.23% 4.84%
GAME DIFFICULTY TEST
33 Amazon Services Confidential
GAME DIFFICULTY TEST
• How much would you pay to increase retention by 20%?
• How much time would you invest?
Our Take-Aways!
• Never launch a game in future w/o AB difficulty test!
• Low-hanging fruit!
• Opportunities abound!
34 Amazon Services Confidential
A|B Testing
• Monetization – minimize luck and maximize
skill
• Delight your customers
• test customer experiences
• AB testing tools are easy to use
• Cross Platform
• Deploy changes instantly
• Free
• Low-hanging fruit is waiting to be harvested
Paul Cutsinger Russell Carroll Find me at: • @PaulCutsinger • @AmazonAppDev • developer.amazon.com/blog • bit.ly/ABTestXPlat
