Deliberating the efficiency of desktop vs mobile research qor,aor and qantas

March 4-7, 2014 Ramada Hotel, Singapore

www.mrmw.net

Title Sponsor Gold Sponsor Silver Sponsors

Bag Sponsor Workshop Host

Sponsor

Event App

Partner

Networking Reception

Partner


www.mrmw.net

Association and Media Partners


www.mrmw.net

Brian Fine, Chairman, QOR (Quality Online Research) / CEO, (AOR) Australia Online Research

Dr. Con Menictas, Senior Statistician Loyalty Programs, Qantas

Deliberating the efficiency of desktop versus mobile research: a pan-Asian perspective

www.qor.com.au www.australiaonlineresearch.com 2

INTRODUCTION

What did we do?

• Looked at theory and myths about mobile research and decided to carry out a case study

• Covered Asia (Japan, Korea, China) and the West (Australia, UK, USA)

• Tested a number of myths: 1. Cannot do a long survey on mobile

2. Results will differ from PC to Mobile

3. The data collected will not be

representative and valid


WHY DID WE DO THIS? Importance of mobile phones to future research

• Smartphone penetration growing across the region, particularly in developed markets

• 40-70% penetration, with Korea at the highest in the world and Japan one of the lowest

• 78% growth across Asia as of September 2012 (GFK Asia report)

• Penetration levels as high as 80% among younger demographic segments

2013 Penetration Japan Korea China Australia UK USA

Smartphones 41.7% 67.6% 55% 65% 48.4% 56%

• Sources: Japan: eMarketer, May 2013; Korea: Strategy Analytics, June 2013; China: China Internet Watch, 2013; Australia: Google, June 2013; US: Pew Research, Jun 2013 and Nielsen, Sep 213; UK: eMarketer, April 2013


WHAT DID WE NEED TO TEST?

• Platforms: • Smartphones (new platform) vs. PC (standard platform)

• Smartphone user: • on Smartphone vs. on PC (same people, different platform)


Questionnaire Design

• Mix of question types: • KPIs (Penetration, Satisfaction,

Recommendation)

• Attitudes

• Demographics

• Mix of question styles: • Single selection, Multi selection, Drop down box,

Grid, Open ended, Rating scales


METHOD • Sample sizes achieved by country:

Japan Korea China Australia UK USA TOTAL

Smartphone on Smartphone 543 642 397 493 407 55 2537

Smartphone on PC 155 510 689 498 1213 941 4006

Other on PC 413 356 171 5605 734 607 2786

TOTAL on PC * 568 866 860 1003 1947 1548 6792

TOTAL RESPONDENTS 1118 1508 1259 1498 2355 1604 9342

* General population - Includes smartphone owners responding on PC

Screen shots of survey on Smartphone vs. PC

Smartphone PC


32 31 22

33 42 38

65 77 74

54 56 64

60 53 57

30 21 24

0

10

20

30

40

50

60

70

80

90

100

SMoSM SMoPC PC SMoSM SMoPC PC SMoSM SMoPC PC

JAPAN KOREA CHINA

Enjoyment (% Top 2 Boxes)

Enjoyed filling it out It was ok - like other surveys

FINDINGS: VIABILITY? Enjoyment • Positive responses (86-98% Top 2 Box ) – particularly China

• Comparable across all three cells in each country

• Highest scores were among Smartphone users


Likelihood to do again • Very positive responses (90-96% Top 2 Box )

• Comparable across all three cells in each country

• Highest scores were among Smartphone users

63 58 56 69 65 66 71

58 57

33 35 36 25 29 29 23

33 33

0

10

20

30

40

50

60

70

80

90

100

SMoSM SMoPC PC SMoSM SMoPC PC SMoSM SMoPC PC

JAPAN KOREA CHINA

Likelihood to do again (% Top 2 Boxes)

Very likely Somewhat likely


Wear out at 5 minute intervals – Australia only

% Smartphone PC

Start of survey 8% 19%

Demographics 2% 1%

First category 0% 0%

Second category 4% 1%

Third category 2% 1%

Final section 5% 1%

TOTAL DROP-OUTS 21% 22%

Total sample 157 664

• Overall drop out levels the same across platforms

• Slightly higher proportion of drop outs on PC at start of survey and on Smartphone in last 5 minutes (possibly effect of grid question)

Drop-outs as % of Total Completes + Incompletes


ASSESSMENT FOR DIFFERENCE IN RESULTS BY MODE

• We carried out extensive statistical analysis to assess differences in the quality of data obtained via SMoSM, SMoPC or PC: • ANOVA - to test for the differences in means

• ANOVA PLOTS - to visually demonstrate differences in means

• CORRESPONDENCE ANALYSIS - to visually represent spatial associations of differences

• MULTINOMIAL LOGIT MODELS - to provide for statistically different predictors (however none were found)

• TEXT MINING and TEXT ANALYTICS - to visually demonstrate the way respondents view their experience of each mode


FINDINGS: SIMILAR DATA? KPI's: Satisfaction and Recommendation

• Results are comparable across modes • Most KPIs show no significant differences

• The most significant differences are in Korea in ratings of mobile service providers. A couple differences in China on small brands.

Satisfaction Japan Korea China

Mobile x

Banks x

Insurance

Recommendation Japan Korea China

Mobile x x

Banks

Insurance

= no significant difference between SM and PC X = significant difference for one or more brands


SIMILAR RESPONDENTS? Attitudes to Technology

• Smartphone respondents are generally more engaged with technology

• In China there is high agreement across all cells

67

75 75

46

54

71

44

56

69

Japan Korea China

“I have a personal interest in IT” (% Agree)

SMoMS SMoPC Total PC


THE ORIGINAL QUESTIONS ANSWERED!

1. Can a longer survey (eg. 15 minutes) be done on a mobile? YES! Enjoyment is high. Response rates and drop out rates do not differ.

2. What kind of surveys can be done on mobile? All question types proved viable; use of software which adjusts to screen size allows for consistency.

3. Is the data valid? YES! Similar results were achieved.

4. Is the data representative of population? YES, as long as weighting by age and gender is applied.


Benefits of Smartphone Research

• Device Independence

• Convenience for respondents

• More immediate response; Speed of survey results

• Accessing younger and hard-to-reach demographics

• Increased reach as Smartphone penetration increases

• Centralised communication & social medium of the future

• Cost effectiveness - fast and non-invasive medium

• Smartphone features lend to more capabilities


Disadvantages

• Limitations of smaller screen

• Some evidence that higher drop out with more complex questions (e.g. grids, multi-selection)

• Need to adapt questions and software to smaller screen

• Smaller pool of potential respondents

• May need to calibrate results at brand level


SUMMARY

Smartphone research is a viable option:

• No need to limit to short 5-minute or SMS surveys!

• Need to use comparable software to PC platform

• Myths of limitations may be based on historical software limitations

• Can reach broader age groups

• Can be fun and likely to retain panellists

• Provides valid data, across all sections of a 15-minute questionnaire!

Brian Fine, Chairman, QOR (Quality Online Research) / CEO, (AOR) Australia Online Research Dr. Con Menictas, Senior Statistician Loyalty Programs, Qantas

Deliberating the efficiency of desktop versus mobile research: a pan-Asian perspective

Title Sponsor Gold Sponsor Silver Sponsors

Bag Sponsor Workshop Host

Sponsor

Event App

Partner

Networking Reception

Partner


www.mrmw.net

Association and Media Partners


www.mrmw.net


www.mrmw.net

Date post:	21-May-2015
Category:	Technology
Upload:	merlien-institute
View:	129 times
Download:	1 times