How and when doesthe mode of data collection

affect survey measurement?

Annette Jäckle (University of Essex)Pamela Campanelli (Independent Survey Methods Consultant)

Peter Lynn (University of Essex)Gerry Nicolaas (National Center for Social Research)

Steven Hope (University College London)Alita Nandi (University of Essex)

RSS 11 Oct 2011

Background

Existing guidelines for designing questionnaires for use in different modes

US Census Bureau 2007Dillman 2000

E.g. Census Bureau guidelines (2007)

Aimmeaning and intent of the question and response options must be consistent across modes.

Some differences between modes in question wording may be necessary to collect equivalent information

But differences should be minimized and tested

Census Bureau guidelines (2007) ctd.

30 specific guidelines regarding Q wording and instructionsExamplesResponse categoriesFormatting of answer spacesVisual design elementsQuestion order and groupingShowcardsPrompts and help

Census Bureau guidelines (2007) ctd.

Examples:1. Maintain the same wording of questions

across modes.6. The underlying response task posed by a

question should be consistent across modes12. Use identical response categories...

Census Bureau guidelines (2007) ctd.

But: Even if Q wording, response options, task etc are the same across modesMode effects sometimes occur

Why?Which questions at risk?

Project aims

Effect of mode on measurementWhich survey Qs are at risk?

generalisable features of questions / modesvariable specific mechanisms

Practical advice for improving question portability across modesFunded by ESRC-SDMI

TodayBackground

1. Framework, design of experiments2. Cognitive interviewing follow-up study

Empirical results: the role of....3. ...visual and aural stimuli4. ...the interviewer5. ...mode specific question formats

Implications6. Questionnaire design for mixed mode surveys

Framework: causes of mode effects

Comprehension Retrieval Judgment Response

Extent of Interviewer involvement

x xRespondent motivationImportance of surveyNon-verbal communicationInterviewer probing

Respondent ability

Task difficultyReporting situation•Time pressure•R distraction

Cognitive demands of Q

Visual/aural stimulus Q format Respondent control over Qaire

Extent of Interviewer involvement

Reporting situationPrivacy of reportingLegitimacy of surveyI-R interaction

Context informationInterviewer characteristicsNon-verbal communicationVisual layoutSequential/simultaneous Qs

Information available to R: How is Q processed?►Response effects

Perception of risk:Willingness to disclose?► Social desirability bias

Depth of cognitive processing: Sufficient effort?

►Satisficing

Hypotheses: survey-level

Interviewers motivate respondents to make required effort ⇒Less satisficing

Interviewers can help with difficult tasks⇒Less confusion / fewer errors

Visual stimulus helps cognitive processing⇒Less satisficing / less confusion with visual

Format effects versus mode effects⇒No mode differences if same format

Hypotheses: item-levelExtent and nature of item-specific mode effect depends on Q characteristics:

Question formatInherent difficulty (content, wording)SensitivityType (attitude, behaviour, other factual, ...)Level of measurement (ordinal, nominal, ...)Response format (yes/no, frequency, ...)

Specific hypotheses in papers 3, 4, 5

Experimental design I

Split ballot question format experimentsLong vs. short listAgree/disagree vs. forced choiceRanking vs. ratingFull vs. end labelsShowcard vs. no showcardBranching vs. no branchingYes/no vs. code all that apply

Experimental design II

Each Q format experiment with several items which varied by

Inherent Q difficulty (content, wording)SensitivityType (satisfaction, other attitude, behaviour, other factual)Measurement level (ordinal, nominal)

Experimental design III

Follow-up to NatCen Omnibus and BHPSGreat Britain population aged 16+Restricted to respondents with web accessRandom allocation to CAPI, CATI, CAWI

Response rates and sample sizes

Sample sizes after exclusion of non-internet access or use cases from CAPI and CATI samples.

* Not available yet.

TodayBackground

1. Framework, design of experiments2. Cognitive interviewing follow-up study

Empirical results: the role of....3. ...visual and aural stimuli4. ...the interviewer5. ...mode specific question formats

Implications6. Questionnaire design for mixed mode surveys