Association for Institutional Research Annual Forum May 2014
Amber D. Lambert, Ph.D. Angie L. Miller, Ph.D. Center for
Postsecondary Research, School of Education, Indiana University
Living with Smartphones: Does Completion Device Affect Survey
Responses?
Slide 2
Literature Review In higher education, surveys are used
frequently for collecting information to demonstrate effectiveness
(Kuh & Ikenberry, 2009) Example purposes: curriculum
improvement, internal evaluation, accreditation, outcomes
assessment, strategic planning Student surveys are most prominent,
but surveys are also used to gather information from other
stakeholders, including faculty, staff, and alumni (Cabrera et al.,
2005; Kuh & Ewell, 2010)
Slide 3
Literature Review Although online surveying is more efficient
and convenient, survey response rates have actually been falling
(Atrostic et al., 2001; Baruch, 1999; Porter, 2004) Transition to
web-based surveys over the past decade has generated much research
on the new mode of delivery (Dillman, 2007) Initial concerns over
sampling bias and coverage (Couper, 2000)
Slide 4
Literature Review As internet access grows, now research
focuses on how mode impacts responses themselves: Content of ones
response (Descombe, 2006) Humanizing aspects of interface
(Tourangeau et al., 2003) Technology to make survey dynamic (rather
than static) (Norman et al., 2001)
Slide 5
Literature Review Also research on web-based surveys and: Page
breaks and scrolling (Peytchev, 2009) Effectiveness of progress
bars on completion (Villar et al., 2013) Browser compatibility and
response placement (Kaye & Johnson, 1999) Color contrast and
placement of emphasis (Tourangeau, 2004)
Slide 6
Literature Review Research on web-based surveys now must shift
away from laptops and desktops to smartphones and tablets Mobile
devices offer internet access virtually anywhere, but touch screen
functioning, truncated viewing area, and smaller keyboards can
place additional burdens on survey respondents (Buskirk &
Andrus, 2012; Peytchev & Hill, 2010)
Slide 7
Literature Review Recent research comparing survey patterns
between PC, tablet, and smartphone users found: Young people more
likely to use smartphones, young and employed people more likely to
use tablets (de Bruijne & Wijnant, 2014) Mobile phone
respondents have lower completion rates, shorter open-ended
answers, and are younger; no gender or education differences
(Mavletova, 2013)
Slide 8
Research Questions Goals of this study: A) explore patterns in
responses to a multi-institution alumni survey, looking at how type
of completion device is impacted by various demographic variables,
including age, income, gender, and employment status B) examine
relationships between type of device and other survey-taking
characteristics, including breaking off, backing up, time duration,
item nonresponse, and open-ended text box completion
Slide 9
Method: Participants Data from the 2012 and 2013
administrations of the Strategic National Arts Alumni Project
(SNAAP) Participants were 58,768 alumni from 109 different arts
high schools, arts colleges, or arts programs within larger
universities Sample consisted of 2% high school level, 76%
undergraduate level, and 22% graduate level alumni 41% male, 59%
female,.2% transgender Majority (85%) reported ethnicity as
Caucasian Average institutional response rate: 18%
Slide 10
What is SNAAP? On-line annual survey designed to assess and
improve various aspects of arts-school education Investigates the
educational experiences and career paths of arts graduates
nationally Questionnaire topics include: Formal education and
degrees Institutional experience and satisfaction Postgraduate
resources for artists Career Arts engagement Income and debt
Demographics
Slide 11
Method: Metadata Measures Completion device: tracked through
data collection platform- PC (42%), Mac (43%), Smartphone (9%), and
Tablet (5%) (with an other.4% not traceable) Breakoff: did
respondents reach the end of the survey and hit the submit button?
Backup status: did the respondent go back (using the browser) to
previously completed pages?
Slide 12
Method: Metadata Measures Time duration: how long (in minutes)
did respondents spend with the survey open in their browser? Item
nonresponse: did respondents answer complex matrix layout items
without leaving any missing? Open-ended text boxes (10 total
throughout survey): did respondents write ANY response? If so, how
long was their response?
Slide 13
Method: Metadata Measures Example of matrix layout question
sets:
Slide 14
Method: Demographic Measures Demographic information collected
for: Continuous variables of age (write-in number box) and income
(used midpoints of response ranges) Categorical variable of gender
Male, female, transgender Categorical variable of current
employment status Full-time (35 hours or more per week) Part-time
only Unemployed and looking for work In school full time Caring for
family full time Retired Other
Slide 15
Analyses Series of 16 chi-squared analyses was done for each of
the categorical metadata and demographic variables For gender,
current employment status, completion status, backup status, item
nonresponse status (for two sets of matrix layouts), and open-ended
response status (for 10 open-ended questions) Series of ANOVAs and
Mann-Whitney tests for continuous metadata and demographic
variables For age, income, duration, and length of open-ended
responses
Slide 16
Results: Demographic Variables Smartphone users are
significantly younger, while tablet users have significantly higher
income AgeIncome PCMean46.5955172.45 N2100517976 Std.
Deviation15.8344474.32 MacMean43.0752632.77 N2137118397 Std.
Deviation15.2044100.82 Smart PhoneMean37.3247452.60 N30632848 Std.
Deviation12.4240824.37 TabletMean46.4560997.26 N25492186 Std.
Deviation14.4348138.70 OtherMean41.3546791.91 N200173 Std.
Deviation14.81043216.90 TotalMean44.4153791.37 N4818841580 Std.
Deviation15.4944343.01 F 326.00537.690 Sign 0.000
Slide 17
Results: Demographic Variables Women are more likely to use
tablets and smartphones; retired people less likely to use
smartphones but more likely to use tablets PCMacSmart
PhoneTabletTotal Count% % % % % Gender Male
898342.2%884540.9%102532.4%95836.8%1981140.7% Female
1226357.7%1276559.0%212267.2%163962.9%2878959.1% Transgender
24.1%35.2%13.4%7.3%790.2% Employment Status Full-time (35 hours or
more per week) 1396065.5%1332961.5%203164.9%157761.0%3089763.5%
Part-time only 268012.6%344115.9%48015.3%36314.0%696414.3%
Unemployed and looking for work 6212.9%7383.4%1284.1%762.9%15633.2%
In school full time 4792.2%7283.4%1344.3%642.5%14052.9% Caring for
family full time 2601.2%2771.3%832.7%783.0%6981.4% Retired
19849.3%12966.0%902.9%26010.1%36307.5% Other
13216.2%18588.6%1825.8%1696.5%35307.3%
Slide 18
Results: Metadata Variables Smartphone users took a
significantly longer amount time (selecting only for those who
completed the survey) MedianN Std. Deviation PC 27.6221372244.61
Mac 27.6021706297.08 Smart Phone 31.82314199.38 Tablet
28.482593119.46 Other 29.2720476.80 Total 27.9849016258.10 F 5.907
Sign 0.000
Slide 19
Results: Metadata Variables Smartphone users were far more
likely to break off, but all device users were equally likely to
back up to previous pages PCMacSmart PhoneTabletTotal Count% % % %
% Completion Status Complete
2137287.0%2170785.4%314157.6%259383.8%4881383.4% Partial complete
318312.9%371914.6%231042.4%50216.2%971416.6% Backup Status
Respondent did not back up in survey
2344095.5%2428195.5%523396.0%294395.1%5589795.5% Respondent backed
up in survey 11154.5%11454.5%2184.0%1524.9%26304.5%
Slide 20
Results: Metadata Variables Smartphone users were more likely
to provide complete responses to complicated layout questions (i.e.
lower item nonresponse) PCMacSmart PhoneTabletTotal Count% % % % %
Response to Complicated Question 1 Did not respond to all items
218010.2%21529.9%1665.3%2409.3%47389.7% Did respond to all items
1919289.8%1955490.1%297594.7%235390.7%4407490.3% Response to
Complicated Question 2 Did not respond to all items
214910.1%19839.2%1655.3%2198.5%45169.3% Did respond to all items
1918989.9%1968490.8%296794.7%237091.5%4421090.7%
Slide 21
Results: Metadata Variables Overall, smartphone and tablet
users were less likely to write responses to open-ended questions
PCMacSmart PhoneTabletTotal Count% % % % % Question 1 Respondent
did NOT write something
476534.5%580742.4%74437.7%63236.5%1194838.3% Respondent did write
something 903965.5%789857.6%123062.3%109863.5%1926561.7% Question 2
Respondent did NOT write something
722833.8%692831.9%120138.2%101539.1%1637233.5% Respondent did write
something 1414466.2%1477868.1%194061.8%157860.9%3244066.5% Question
3 Respondent did NOT write something
1672478.3%1689977.9%257481.9%209780.9%3829478.5% Respondent did
write something 464821.7%480722.1%56718.1%49619.1%1051821.5%
Question 4 Respondent did NOT write something
578862.8%398360.3%94370.5%71165.6%1142562.6% Respondent did write
something 342837.2%261939.7%39429.5%37334.4%681437.4% Question 5
Respondent did NOT write something
1944691.0%1964690.5%290692.5%240092.6%4439891.0% Respondent did
write something 19269.0%20609.5%2357.5%1937.4%44149.0%
Slide 22
Results: Metadata Variables PCMacSmart PhoneTabletTotal Count%
% % % % Question 6 Respondent did NOT write something
628632.0%677733.4%138347.7%90938.8%1535534.0% Respondent did write
something 1338268.0%1350766.6%151652.3%143361.2%2983866.0% Question
7 Respondent did NOT write something
256013.0%278313.7%59120.4%37015.8%630413.9% Respondent did write
something 1710887.0%1750186.3%230879.6%197284.2%3888986.1% Question
8 Respondent did NOT write something
1330362.2%1398364.4%234874.8%187172.2%3150564.5% Respondent did
write something 806937.8%772335.6%79325.2%72227.8%1730735.5%
Question 9 Respondent did NOT write something
1698979.5%1749680.6%275287.6%214682.8%3938380.7% Respondent did
write something 438320.5%421019.4%38912.4%44717.2%942919.3%
Question 10 Respondent did NOT write something
1936590.6%1964390.5%295994.2%241393.1%4438090.9% Respondent did
write something 20079.4%20639.5%1825.8%1806.9%44329.1%
Slide 23
Results: Metadata Variables And of those who did write
open-ended text responses, smartphone and tablet users overall
wrote significantly shorter responses Question 1Question 2Question
3Question 4Question 5 PCMedian 352459210980 N 903914144464834281926
Std. Dev. 50.08426.84212.87213.18150.07 MacMedian 3326410910284 N
789814778480726192060 Std. Dev. 48.45458.49210.85188.51180.06 Smart
PhoneMedian 2917973 46 N 12301940567394235 Std. Dev.
28.70305.00135.84111.1583.63 TabletMedian 30184758352 N
10981578496373193 Std. Dev. 51.40302.53148.12189.27138.32
OtherMedian 332029317274 N 79142463027 Std. Dev.
27.76367.66154.57313.9778.71 TotalMedian 332459710378 N
19344325821056468444441 Std. Dev. 48.41431.56206.28199.40162.13 F
16.51555.79316.90214.31810.893 Sign 0.000
Slide 24
Results: Metadata Variables Question 6Question 7Question
8Question 9Question 10 PCMedian 150119126278119 N
1338217108806943832007 Std. Dev. 194.52202.78164.34363.87286.35
MacMedian 147118127283125 N 1350717501772342102063 Std. Dev.
201.17206.43165.08399.88266.77 Smart PhoneMedian 103789218567 N
15162308793389182 Std. Dev. 124.94117.37120.12337.23133.72
TabletMedian 12291113228102 N 14331972722447180 Std. Dev.
161.72133.83145.75296.27188.84 OtherMedian 17610613132094 N
126157743928 Std. Dev. 206.36179.03176.38342.34111.47 TotalMedian
144114124274119 N 29964390461738194684460 Std. Dev.
193.99198.16162.61377.40269.10 F 54.96876.38720.97013.1515.917 Sign
0.000
Slide 25
Discussion Many patterns of results are consistent with
previous literature Differences on type of device used based on age
and employment status Potential generational effects? Unlike
previous studies, also found differences for gender and income
Tablets as luxury items? Are women more compliant to survey
requests, regardless of device?
Slide 26
Discussion Types of devices do seem to affect (in some ways)
respondents survey-taking behaviors Also mirrors previous
literature that taking surveys on smartphones and tablets can
increase respondent burden Smartphone users are far more likely to
abandon the survey, and those who do finish require more time to
complete it Smartphone and tablet users were less likely to answer
open-ended questions, and when they did answer them their responses
were much shorter
Slide 27
Discussion Interestingly, smartphone users were more likely to
fully complete complex layout item sets Counterintuitive at first
glance because these questions may require more scrolling
(vertically and horizontally) on a truncated screen, so one would
expect fewer complete responses Could be that those who persevered
to these points in the survey on a smartphone (about 1/3 and of the
way through) are the more dedicated and conscientious survey
takers
Slide 28
Conclusions Limitations of study: sample may not be completely
representative of all survey takers (only arts alumni, lower
response rates, and selective participation) When designing
web-based surveys, need to take into account that respondents may
use smartphones and tablets May need to rely less heavily on long
layouts and open- ended questions Future research continuing to
look at device type is necessary as technology rapidly becomes
available to larger populations Convenience of anytime, anywhere
internet access may have negative impact on data quality
Slide 29
Questions or Comments? Contact Information: Amber D. Lambert
[email protected]@indiana.edu Angie L. Miller
[email protected]@indiana.edu Strategic National Arts
Alumni Project (SNAAP) www.snaap.indiana.edu (812) 856-5824
[email protected] *Reference list available upon request or in full
paper