Post on 08-Feb-2022
transcript
Associations between Interviewer Observations and Proficiency ScoresMichael LemayValerie HsuRichard SigmanTom Krenzke
TSE 2015Baltimore, MD
PIAAC Study Programme for International Assessment of Adult
Competencies Organisation for Economic Co-operation and
Development (OECD) Largest literacy study ever undertaken
– Over 215,000 interviews (adults in households)– 33 countries– Up to 7 more slated to participate in upcoming 3rd round
Components of the PIAAC Interview Screener (if applicable) Background Questionnaire (BQ) Core Exercise (EX)
– Literacy– Numeracy– Problem-solving in technology-rich environment– Reading components
Observation Module (ZZ) For nonrespondents only: Non-Interview Report Form
(NIRF)
PIAAC Survey Outcomes Assessments were self-administered in official
language(s) of the country To reduce respondent burden, not all assessment items
were given to a respondent Scores estimated through Item Response Theory (IRT) 10 plausible values (scores) generated on a scale of 0 to
500 for each assessment domains IRT model uses BQ items as covariates
Research Questions
Proficiency
Respondents Behavior &
Characteristics
Circumstances during the
Assessment
5
Research Questions (cont’d)
Interviewer Observations
Proficiency
Participation
6
PIAAC Interv. Observation ModuleYes No Miss.
Anyone besides R present? 1,83038.1%
2,97061.9%
210
If so, provided assistance to R in BQ? 2034.2%
4,59995.8%
208
If so, provided assistance to R in EX? 851.8%
4,71798.2%
208
Second sample person present? (if applicable) (USA only) 8127.5%
21472.5%
4,715
R asked for clarification during interview? 1,85338.6%
2,94661.4%
211
(n = 5,010)
7
PIAAC Interv. Observation Module (cont’d)Yes No Miss.
R held conversation with someone? 80616.8%
3,99683.2%
208
R answered a phone call/text/email? 73315.3%
4,06984.7%
208
R was looking after children? 3948.2%
4,40891.8%
208
R was undertaking domestic tasks? 701.5%
4,73298.5%
208
Television/radio/game console/music player playing? 59112.3%
4,21187.7%
208
R was interrupted by other activity/task/event? 56511.8%
4,23788.2%
208
(n = 5,010)
8
Observation Module (cont’d)Upper Middle Working Poor Miss.
Household socio-economic status (SES) (observed by FI)(USA only)
57812.2%
1,48231.4%
2,07243.9%
59112.5%
287
(n = 5,010)
9
Results: Simple RegressionsLiteracy
Par. EstimatesNumeracy
Par. EstimatesProblem-solving
Par. Estimates
Presence of others -9.33** -12.89** -4.21**
Assist. in BQ -16.90** -22.75** -3.94**
Assist. in EX -14.24** -16.29** -4.93**
2nd SP in room -4.32** -2.84** 2.90**
Clarifications -13.93** -15.34** -8.35**
Conversation -5.70** -10.50** -3.38**
Phone call -0.96** -5.93** -4.98**
Children -6.76** -12.68** -5.62**
Domestic tasks -1.90** -7.26** -8.12**
TV, radio, etc. -12.72** -22.28** -11.67**
Other activity -3.57** -5.25** 0.30**
SES (ref. cat: Upper)- Middle- Working- Poor
-12.07**-38.76**-66.16**
-12.66**-44.21**-79.35**
-8.36**-26.57**-42.45**
Pearson’s Correlations among Covariatesrho Variables
.415 Presence of others Conversation
.292 Assistance in EX Assistance in BQ
.272 Assistance in BQ Presence of others
.250 Presence of others Children
.224 Conversation Children
< .2 All 50 other correlations
Results: Multiple RegressionsLiteracy
Par. EstimatesNumeracy
Par. EstimatesProblem-solving
Par. Estimates
Presence of others -- -- --
Assist. in BQ -9.01** -12.74** --
Assist. in EX -- -- --
Clarifications -9.91** -9.81** -7.07**
Conversation -- -- --
Phone call -- -- --
Children -- -- --
Domestic tasks -- -- --
TV, radio, etc. -- -13.13** --
Other activity -- -- --
SES (ref. cat: Upper)- Middle- Working- Poor
-11.94**-38.01**-64.08**
-11.98**-42.34**-75.26**
-8.17**-26.33**-41.50**
n; R2 4,723; 0.187 4,723; 0.209 4,005; 0.092
Nonresponse and SES
Nonrespondents(n=327)
Respondents(n=4,814)
High 19.0% 13.0%
Medium 37.6% 31.5%
Low 34.9% 42.7%
Very low 5.2% 11.3%
Could not determine 3.3% 1.5%
100.0% 100.0%
p value = 0.00121
SOURCE: Hogan, J., Montalvan, P., Diaz-Hoffman, L., Dohrmann, S., Krenzke, T., Lemay, M., Mohadjer, L., and Thornton, N. (2014). Programme for the International Assessment of Adult Competencies (PIAAC) 2012: U.S. Main Study Technical Report (NCES 2014-047). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Available at: https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014047.
13
Discussion Strongest predictors of proficiency scores:
– Interviewer observed SES– Whether or not the respondent asked for clarification while
undertaking the interview Weaker predictors of proficiency scores:
– Electronics in use in the immediate vicinity of the respondent
– Respondent received assistance from others in the room in answering the Background Questionnaire
Presence of others does not affect proficiency scores SES is related to participation in the study
Discussion (cont’d)
Minimize distractions (quiet room, ask to turn off TV, radio, music, etc.)
• Not always possible (small living quarter)• Do respondents comply?• Do field interviewers even ask?
Efforts to get cooperation from household of higher SES
15
Discussion (cont’d)
Observed SES could be used for nonresponse error reduction:– Associated with proficiency in all 3 domains– Associated with participation in study
Potential measurement error differences between:– SES observed for nonrespondents– SES observed after the interview
Observe SES at the time of first contact attempt?
17
Thank youMichaelLemay@Westat.com