The use of cognitive interviewing methods to evaluate mode effects in

survey questions

Michelle Gray Margaret Blake

Pamela Campanelli

Acknowledgements

• Members of the wider research team• Gerry Nicolaas (NatCen)• Steven Hope (University College London, Institute of Child

Health)• Peter Lynn (Institute for Social & Economic Research,

University of Essex)• Anette Jäckle (Institute for Social & Economic Research,

University of Essex)• Alita Nandi (Institute for Social & Economic Research,

University of Essex)

Outline of presentation

• Background and the wider research project• Aims of the cognitive interviewing stage• Novel implementation of CI method• Alterations in the CI techniques• Limitations of the methodology • Conclusions• Mode preferences• How cognitive interviewing findings fit in with

the rest of the analysis

Aims of the CI phase

• Used as a follow up methodology• To explore:

• how mode effects happen• how respondents process questions in different

modes• quantitative findings which relate to mode effects

(where a pattern was observed)

Novel use of CI?

• Used in the context of a methodological project• As a follow up methodology (after a survey)• Use of multiple modes in one interview • Sample selected based on satisficing behaviour• Goal was to provide insights into apparent mode

effects

Quantitative sample

• Came from a sample of survey participants (2008)• Those who agreed to give their email addresses were

• Randomly assigned to one of three experimental groups• Surveyed (as part of the experiment) in CAPI, CATI or CAWI• Questionnaire comprised of a mix of original 2008 survey Qs

+ new Qs which had been purposely designed to test research team’s hypotheses

• Initial quantitative analysis of results showed• Some hypotheses were supported• Whilst others were not (with some quite surprising findings!)

Thinking behind sample design for CI

• Importance of reflecting response behaviours in different modes

• Ideal would be to select (and then compare in CI)• Respondents who answer Qs in different modes

inconsistently• Respondents who show no mode effects

• However…mode effects detected at aggregate level • But there were two response behaviours which

could be detected at individual level – and when aggregated varied across mode 1. Agreeing to opposite statements 2. Non-differentiation on a ranking task

Satisficing

Sample design for CI

• Regression analysis suggested an over representation of Rs with lower socio-economic status and from minority ethnic groups amongst those who satisficed

• Three quota groups were decided upon1. “Acquiescers” (those who agreed to opposite statements)2. “Rankers” (those who misunderstood the ranking tasks)3. “Non-satisficers” (those who had not acquiesced & not

made ranking errors and had opposite demographic & socio economic profile to those who had satisficed)

Quotas for CI

CI sample(x37)

Acquiescers(x18)

Rankers(x10)

Non-satisficers(x9)

Format of the cognitive interview

• Three parts:1. Interviewers read a one page introduction to

the study2. Survey questions were administered in CAPI,

CATI and then CAWI 3. Interviewers carried out the ‘retrospective

think-aloud’• Whole of the interview audio recorded

(and later transcribed)

Survey administration1. Questions administered in CAPI

• Interviewer asking questions• Use of a paper questionnaire (to record responses)• Use of show cards (where necessary)

2. Next questions administered in CATI• Respondents & interviewers in separate rooms within the house• Use of paper questionnaire (to record responses)• Mobile (I) to landline (R) or mobile (I) to mobile (R) • Loudspeaker function on interviewer’s mobile phone

3. Finally questions administered in CAWI• Respondent on interviewer’s laptop• Offline version• Privacy replicated

Mode mimicking in CAPI

Mode mimicking in CATI

Mode mimicking in CAWI

Retrospective think-aloud (1)

• Face-to-face • Interviewers read out the following script to each R:

“Now we come to the part where I will ask you some questions about the Topical Issues questions you just answered. Mostly, Iwill be asking you to tell me what came to your mind while you were answering the Topical Issues questions. This is called ‘thinking out loud’. We’ve found that it helps to have some

practice at doing this. So let’s try an exercise which has nothing to do with the Topical Issues questions. For fun, I’d like you to: Think about how many windows there are in your house. As you count up the windows, tell me what you are seeing and thinking about. Well done. That’s what it is like to think out

loud.”

Retrospective think-aloud (2)

• Interviewers reminded respondent of:• The question they had answered• The mode it had been asked (in person, on the phone, on

laptop)• Response the respondent had provided

• Interviewers read out the following script to each R:“How we will proceed is that I will first remind you of a particular question

or set questions you have just answered and I will remind you of your answers and then I will ask you to think out loud and tell me how you

came up with your answers. While we are doing this, if you remember a specific question as being particularly easy or difficult or sensitive, it

would help if you could comment on this. Also if there are specific places where you felt that the way the question was delivered (that is, with me sitting here with you, over the phone, and on the computer)

made answering the question easier or more difficult, it would be helpful if you could comment on this.”

Retrospective think-aloud (3)

• Interviewer reminded respondent of any behaviour which could signify a problem• Hesitancies• Laughs• Additional comments

• Then, where appropriate, interviewer asked respondent a number of pre-scripted open probes• “Tell me your thinking behind choosing that category”

• Where there had been response behaviour displayed (such as choosing same category in ranking tasks) asked other probes• “I noticed you picked the same category (twice / three times / on all

4 questions). If you had to choose, which would be the most important?

CI techniques

• Use of pre-scripted probes• Lack of interviewer flexibility with probing• Less of the exploratory probing• For a few questions, asked the same survey question

in a different way• Focus of probing was:

• Not to assess performance of the questions• To gain insight into the thought processes of the respondent • Whether the mode of data collection impacted on the way

they answered

Data management & analysis

• Cognitive interview data audio recorded and transcribed word for word

• Data entered into qualitative data management programme (‘Framework’)

• Member of research team produced cognitive interview analysis plan• Built around set of original hypotheses• Analysis distributed between team• Analysis asked to look at findings which support hypotheses

as well as surprising findings • Produce bullet point summaries on the areas for

investigation

Reflections & limitations

• How effective was mode mimicking during the interviews?• How effective is ‘think aloud’ at capturing respondents’

thought processes without impacting on their observed behaviour?

• The quality of the questions included in the experiment• The style of probing used• The complexity of the task carried out by interviewers.• Technical difficulties (reliance on technology) • Analysis process (sometimes strayed into quantitative)• Confining the scope of analysis to mode effects

Conclusions

• Cognitive interviewing worked well as a method for uncovering response processes in different modes

• Explored quantitative findings in more detail• Mode mimicking (also) worked well

• CI modes felt similar to real modes (according to Rs) • Probing is necessary in person

• We would encourage the use of CI in novel ways such as• To look at how Qs perform in different modes• As a follow up methodology – after a quantitative survey (to

shed light on results)

Cognitive Interview Findings onMode Preferences

Mode PreferencesIn mode preference studies, it is typical for the mode the respondent experiences to be the one that the respondent says he or she prefers. • For a household survey example

see Groves and Kahn (1979)• For a school based survey example

see Brener et al (2006) • For a business survey example

see Tarnai and Paxson (2004)• For a longitudinal example

see Smyth and Olson (2010)

Cognitive Interview Questions

At the end of the cognitive interview, interviewers asked• “Overall what was it like to answer the questionnaire in person

versus over the phone versus by yourself on the computer?”• “What did you like or dislike about these 3 different ways you

were asked the questions?”

To our knowledge no previous study has administered multiple modes to the same respondents. • Should improve the validity of the mode preference data

Mode preference findings (1)

• CATI was the least preferred mode. • No Rs picked it as their first chose (and only a few

picked it as their second choice). • Reasons for Rs’ dislike of CATI

• Harder to concentrate / can be distracted by other things • Feeling rushed / pressured on the phone • One gives less attention on the phone or has to work harder to think

about the question and answer • Phone made the task harder because of the lack of visual stimulus and

difficulty in hearing• A general dislike of the phone as a mode of communication

Mode preference findings (2)

• Preferences for CAPI versus CAWI were mixed. This relates to . . . • R’s level of computer literacy• Valuing personal nature of face-to-face or privacy of CAWI• Controlling the pace of the CAWI interview, although some Rs felt

CAWI made them go too quickly without thinking• CAPI leading to better quality data because. . .

• One is more engaged (would “get more out of her”)• Answers more considered (more likely to answer and to give a genuine answer)• More committed to the interview and good answers

Findings from the Cognitive Interviewing

How they fit in with the rest of the analysis and presentations

Identifying mode effects

• Identified respondents who showed ‘suspicious’response patterns

• Compared them with the remaining respondents• Interrogated the data • Looking for explanations for these patterns

• Did not go through each question and respondent in turn; analysis combined questions and respondents

Examples of findings (1)

• Where quantitative methods suggested there was a problem (e.g. acquiescence bias) but a qualitative approach showed that respondents were able to justify their answers.

• Where quantitative methods identified unexpected response patterns or potential mode effects but only a qualitative approach can provide an understanding of what is happening

• Where quantitative methods do not identify any suspicious response patterns but a qualitative approach identifies potential problems.

Examples of Findings (2)

Where cognitive findings fit in

• Limited to investigating a subset of quantitative results

• Will be interspersed through out remaining presentations

• May specifically address issue in quantitative presentation

• May present a broader perspective• May present a narrower perspective (i.e., only a

subset of questions or only a particular aspect to share)