May 27-30, 2014 Congress Plaza Hotel & Convention Center #MRMW

Organized by

Title Sponsor Gold Sponsors

Silver Sponsors

MRMW Chicago May 27-30, 2014 #MRMW

Workshop Sponsors

Association & Education Partners

Networking Reception

Sponsor

Event App

Partner

Media Partners

MRMW Chicago May 27-30, 2014 #MRMW

OutlineIntroduction

ResultsSummary

A US View of Smartphone Researchwith Global Benchmarks

Brian Fine 1 Con Menictas 2

1ChairmanQOR, Quality Online Research

2Senior StatisticianQantas Loyalty, Qantas Airways Ltd.

May 28, 2014

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

1 IntroductionInitial motivationWhy the case study?What we testedVisualsSample

2 ResultsEnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

3 SummaryMythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

What inspired us

We looked at myths about mobile research: too many not tocare about.

This compelled us to carry out a case study.

Previously, we examined Asia i.e., Japan, Korea, China.

Today we examine the US, Australia, UK.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

We put a number of myths to the test

Myth 1

Long surveys can’t bedone on mobiles!

Myth 2

PC results differ toMobile.

Myth 3

Mobile data notrepresentative or valid.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

Because of the importance of mobile devices to marketresearch and today’s data collection

Smartphone penetration growing, particularly in developedmarkets.

40-70% penetration, with USA sitting about average.

Penetration levels as high as 80% among youngerdemographic segments.

2013 Penetration USA Australia UK Japan Korea China

Smartphones 56% 65% 49% 42% 68% 55%

Table 1 : USA: Pew Research, Jun 2013; Australia: Google, June 2013; UK:eMarketer, April 2013; eMarketer, Japan: May 2013; Korea: StrategyAnalytics, June 2013; China: China Internet Watch, 2013; and Nielsen Sep2013

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

The Platforms

Smartphones versus PC

1 SMoSM: Smartphone users that were invited to conduct thestudy on their Smartphone.

2 SMoPC: Smartphone users that were invited to conduct thestudy on their PC.

3 PC: PC users.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

Mobile Phone Screens

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

PC Screen

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

Question types and visual response options

Mix of Question Types Mix of Response Options

Penetration Single selection

Satisfaction Multiple selection

Recommendation Drop down box

Attitudes Grid selection

Demographics Open ended

Rating scales

Table 2 : Case Study Design

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

Initial motivationWhy the case study?What we testedVisualsSample

Sample sizes across modes and countries

Category USA Australia UK Japan Korea China Total

SMoSM 546 493 407 543 642 397 3,028

SMoPC 941 498 1,213 155 510 689 4,006

PC 607 503 734 413 356 171 2,784

TOTAL 2,094 1,494 2,354 1,111 1,508 1,257 9,818

Table 3 : Case Study Sample Sizes (SoS: Smartphone Users on Smartphone;SoPC: Smartphone Users on PC; PC: PC Users; TPC: Total on PC which wasGeneral population that includes smartphone owners responding on PC)

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Enjoyment as an indicator of validity

Figure 1 : Assessing Validity via Enjoyment

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Repeatability as an indicator of validity

Figure 2 : Assessing Validity via Likelihood of Repeatability

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Drop-outs as an indicator of validity

Survey Section Smartpone (%) PC (%)

Start of survey 0.08 0.19

Demographics 0.02 0.01

1st category 0.00 0.00

2nd category 0.04 0.01

3rd category 0.02 0.01

Final section 0.05 0.01

Total drop outs 0.21 0.22Total sample 157 664

Table 4 : Assessing Validity via Drop-outs as % of Total Completes

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Assessment for difference in results by mode

We carried out extensive statistical analysis to assessdifferences in the quality of data obtained via SMoSM,SMoPC or PC.

Very few differences if any!

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Comparability across NPS type questions and Mode as anindicator of validity

Satisfaction USA Australia UK

Mobile 3 3 3

Banks 3 3 3

Insurance 7 3 3

Table 5 : Satisfaction across Country, Category and Mode.

Recommendation USA Australia UK

Mobile 3 3 3

Banks 3 3 3

Insurance 3 3 3

Table 6 : Recommendation across Country, Category and Mode.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Mode MobileProv FinInst’n MVInsSample Block 1 Block 4 Block 7

SMoSM 1.95 4.24 4.51

SMoPC 1.94 3.91 4.33

PC 1.97 3.89 4.19

Sample Block 2 Block 5 Block 8

SMoSM 1.96 4.03 3.83

SMoPC 1.96 3.81 4.08

PC 1.97 3.94 4.19

Sample Block 3 Block 6 Block 9

SMoSM 1.98 4.15 4.06

SMoPC 2.00 4.02 3.99

PC 2.00 3.87 4.16

Table 7 : Satisfaction Means by Block Rotation

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Attitudes Towards Technology

Table 8 : Estimation Results : Logit

Variable zAUS zUK zUSDiffKpUpEmail -0.20 1.68 -0.12DontLkeTwitter -4.36 -5.26 1.97OnlineIsFuture 3.28 3.81 4.55WebLimited -0.84 1.23 1.16PersRelBetThanConn 0.13 2.67 1.20PrivConc -5.40 -0.76 -1.25PersIntIT 2.53 1.99 2.30TechSavv 5.30 4.65 -0.24LikeSMSabbrev 3.07 1.47 -0.40ReadOnTechDev 3.91 4.48 2.68SeekInfOnBlogs 1.02 -0.12 0.97Intercept -2.66 -10.02 -6.34

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

EnjoymentLikelihood to Do AgainWear OutMode VarianceLogit Models

Logits Summary

Figure 3 : Z-values to Demonstrate Country Ratios

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

MythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

Myths we aimed to disprove

Myths Long held views Our results suggest1 Long surveys can’t be done on mobiles 7

2 PC results differ to mobiles 7

3 Mobile data are not representative or valid 7

Table 9 : Negativing 3 Myths of Mobile Research

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

MythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

1 Can a longer survey (eg. 15 minutes) be done on a mobile?Yes! Enjoyment is high.Response rates and drop out rates do not differ.

2 What kind of surveys can be done on mobile?All question types proved viable use of software which adjuststo screen size allows for consistency.

3 Are the data valid?YES!Similar results were achieved.

4 Are the data representative of population?YES!As long as weighting by age and gender is applied.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

MythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

Device Independence.

Convenience for respondents.

More immediate response.

Speed of survey results.

Accessing younger and harder-to-reach demographics.

Increased reach as Smartphone penetration increases.

Centralised communication social medium of the future.

Cost effectiveness - fast and non-invasive medium.

Smartphone features lend to more capabilities.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

MythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

Limitations of smaller screen. (Although 146m Phablets to besold by 2016 (IHS, 2013))

Some evidence that higher drop out with more complexquestions (e.g. grids, multi-selection).

Need to adapt questions and software to smaller screen.

Smaller pool of potential respondents.

May need to calibrate results at mobile brand level.

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

MythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

No need to limit to short 5-minute or SMS surveys!

Myths of limitations may be based on historical softwarelimitations.

Can reach anyone at any time.

Can be fun and likely to retain panellists.

Provides valid data, across all sections of a 15-minutequestionnaire!

Overall Conclusion

Smartphone research is a viable alternative to PC research!

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

OutlineIntroduction

ResultsSummary

MythsBenefits of Smartphone ResearchCaveats of Smartphone ResearchFinal Comments

Contact Details

Brian Finebrian.fine@australiaonlineresearch.com61 408 672 754

Con Menictasconmenictas@qantas.com.aucmenictas@gmail.com61 425 339 790

Brian Fine and Con Menictas A US View of Smartphone Research with Global Benchmarks

Title Sponsor Gold Sponsors

Silver Sponsors

MRMW Chicago May 27-30, 2014 #MRMW

Workshop Sponsors

Association & Education Partners

Networking Reception

Sponsor

Event App

Partner

Media Partners

MRMW Chicago May 27-30, 2014 #MRMW

May 27-30, 2014 Congress Plaza Hotel & Convention Center #MRMW

Organized by