31
KRISTIN CONRADI NOVEMBER 30, 2012 [email protected] Examining Item Validity Using Cognitive Interviews
KRISTIN CONRADINOVEMBER 30, 2012
Examining Item Validity Using Cognitive Interviews
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
“How many times did you visit a doctor in the last year?”
What do you mean by last year?
What does “visit a doctor” mean?
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
“How do you feel about reading a book before you go to bed?”
Hate itIt’s
okayLove it
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
“How do you feel about reading graphic novels?”
What do you mean by graphic novels?
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Purpose: Present a rationale for conducting cognitive interviews in reading research.
Cognitive interviews are a technique researchers can employ to evaluate how participants understand questions screening for potential problems with interpretation
(Beatty & Willis, 2007; Conrad & Blair, 2004; Desimone & Floch, 2004; Presser et al., 2004; Willis, 2005; Willis, Royston, & Bercini, 1991).
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Why do cognitive interviews matter?
(Wiederhold & Block, Gray Silent Reading Test Story 1)
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
The boy is not big.But he helps Mother and Father.He puts his toys up after he plays with them.He tells the baby a funny story if she begins to cry.He sings to make her go to sleep.
Why do you think the boy likes to help?a) It makes him feel goodb) Mother and Father give him money.c) Father is too sick to work.d) Mother makes him do it.
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
• Cognitive Interviews• Reading Motivation Instruments
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
What are cognitive interviews?
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
The standardized approach creates a communication straightjacket.
(Gordon B. Willis, 2012)
Think-Aloud Probing
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Types of Probes ExamplesComprehension/Interpretation Probe What does the term “outpatient” mean
to you?
Paraphrasing Can you repeat the question I just asked in your own words?
Confidence Judgment How sure are you that your health insurance covers drug and alcohol treatment?
Recall Probe How do you remember that you went to the doctor five times in the past 12 months?
Specific Probe Why do you think that cancer is the most serious health problem?
General Probes How did you arrive at that answer? Was that easy or hard to answer? I noticed that you hesitated. Tell me what you were thinking.
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Reading Self-Concept Scale
(1995)
Motivation to Read Profile
(1996)
Elementary Reading Attitude
Survey (1990)
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Sample Questions
Are the books you read in class
too hard?
My best friends think reading is. .
.
How do you feel about reading on a rainy Saturday?
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Uses in Literacy
• Survey of Adolescent Reading Attitudes (McKenna, Conradi, Lawrence, Jang, & Meyer, 2012)
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
The Present Study
Are there any terminology/phrasing issues in reading motivation measures that threaten instrument validity?
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
DATA SOURCES
• Test administrators’ feedback• Respondents’ notes on surveys• Histograms
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
PROCEDURE
Data sources were analyzed to identify candidate items
Probing techniques were decided upon
One-on-one interviews were conducted
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
Subsample (N=16) of larger study (N=491)
Elementary Reading Attitude Survey
Motivation to Read Profile
Reading Self-Concept Scale
PARTICIPANTS & MEASURES
ANALYSIS
Item level response forms
3 reviewers read them and coded for issues
Response forms were read again for supporting or refuting examples
Written summaries were created
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
ANALYSIS
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
RESULTS
Terminological Situational
Logical
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
EXAMPLES OF PROBLEM-TYPES
Do you like playing word games in class? (RSCS)
Situational
Logical
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
EXAMPLES OF PROBLEM-TYPES
Terminological
When someone gives me a book
for a present, I feel (MRP)
Logical
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
EXAMPLES OF PROBLEM-TYPES
Terminological Situational
I read
• Not as well as my friends
• About the same as my friends
• A little better than my friends
• A lot better than my friends
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
When I grow up, I will spend. . .
A grown-up can do whatever they want!
Will probably read a lot because I read a lot
now
I will do more fun things instead of
reading
A grown-up will have a job and responsibilities
Maybe the job won’t require reading, so I
won’t spend time reading
Maybe the job will require reading, so I
will have to read.
Because I will be busy, I won’t have time
to read
Even if the job keeps me busy, I will still read and be sleep deprived
BACKGROUND & RATIONALE
LITERATURE METHOD RESULTS DISCUSSION
DISCUSSION & IMPLICATIONS
• Reading motivation measures
• Other reading measures: particularly as what we mean by literacy evolves (Pitcher et al. 2007)
