Report on the 2008 International Total Survey Error Workshop (ITSEW)
Presented at Q2008 ConferenceRome, Italy July 11, 2008Presented by Paul Biemer
Acknowledge
Other Organization Committee members:
Roeland Beerten, ONSLilli Japec, Statistics SwedenMary Mulry, Census BureauAlan Karr, National Institute for Statistical ScienceJerry Reiter, Duke UniversityClyde Tucker, BLSBrian Meekins, BLS
Outline
• Definition of Total Survey Error• Highlights of the Recent Workshop on Total
Survey Error Workshop• Other Emerging Areas of Research• Future Directions of Research on TSE
Purpose of the ITSEWs
• To stimulate new research on total survey error (TSE)• Provide a forum for researchers to exchange ideas on
current research projects• Initiate new projects to advance the TSE field• To generate interest in the field among students and new
researchers
Organization of the Workshops
• Attendance by invitation only – although no one has ever been refused a seat at the workshop
• Focus is on on-going research and undeveloped ideas– Midcourse research– Ideas still in incubation– Exploratory research
• ITSEW series began 2005 in Washington, DC• ITSEW 2009 will be held in Tälberg, Sweden• National Institute of Statistical Science (NISS) provides
funding and staff support• Support from other organizations is welcome
2008 Total Survey Error WorkshopTheme: Multiple Error Sources and their Interactions
• Held in Research Triangle Park, NC on June 2-4, 2008• Organizing committee:
– Paul Biemer, RTI and UNC-CH– Roeland Beerten, ONS– Lilli Japec, Statistics Sweden– Mary Mulry, Census Bureau– Alan Karr, NISS– Jerry Reiter, Duke University– Clyde Tucker, BLS– Brian Meekins, BLS
• Sponsored by the National Institute for Statistical Science, RTI International and the Survey Research Methods Section of ASA
27 Presenters at the 2008 ITSEW
Joop HoxAnnica IsakssonPeter LundquistNoelle Molinari Donsig Jang Scott Fricker Ting YanSteven Cohen Wendy Hicks Barry Schouten Emilia PeychevaKristen OlsonJohn Dixon
Paul BiemerCatharine BurtAndy PeytchevBrian MeekinsMary MulryRoger TourangeauJerry ReiterLars LybergRita ThissenOztas Ayhan, Mojca Bavdaz, Steven MachlinEric Slud
Total Survey Error
• Sampling Error• Nonsampling Error
– Specification– Nonresponse– Noncoverage– Measurement– Data Processing
Mean Squared
Error (MSE)
TSE: the Concept
• Includes both measurement, analysis and reporting of multiple sources of error
–Quantify the major components of survey error for a survey–Assess contributions of each source to TSE–Combine these to estimate total MSE
• Does not include error reduction methods –E.g., questionnaire design, nonresponse reduction methods, etc.
Arguments for Estimating TSE
• To correct inferences based upon faulty theory
– Biased estimates– Biased measures of uncertainty
• To optimize the design of future surveys• To provide quality declarations to users
Arguments Against Estimating TSE
• Cost• Intractability• Inaccessible methodology• Lack of motivation• Fear
Topic Coverage
Coverage
Sampling
Nonresponse
.
Instrument
Mode
Interviewer
Respondent
.
Processing
0 2 4 6 8 10 12 14 16 18
Highlights
• Interaction between nonresponse and measurement error– Effect on measurement error of nonresponse followup– Statistical models for simulating the measurement error and
nonresponse interaction– Meta-analysis of literature on nonresponse-measurement error
interaction– Empirical studies of the relationship between nonresponse and
measurement error– Causal factors influencing both response propensity and
measurement error– Nonresponse propensity and under-reporting of sensitive topics
Other common themes
• Frame stratification errors and sampling errors• Frame coverage errors and measurement errors (e.g., cell
phone samples and measurement error)• Interviewer effects on nonresponse and measurement
errors
Gaps in the Research
• Data processing errors– Questionnaire design and editing errors– Information content and coding errors
• Establishment surveys• Specification error vs. measurement error
– i.e., measuring the wrong concept accurately vs. measuring the right concept inaccurately
• Need for meta-analysis of existing studies• Better indicators of data quality
– Schouten, Bethlehem, et al• More attention to costs of error reduction methods
Gaps in the Research (cont’d)
• Methods for developing countries• TSE issues in multi-cultural surveys• Methods for estimating variances that reflect multiple error
sources, not just sampling errors• Standardized indicators for comparing the quality of
survey data• A common language for TSE
Need for Data
• How can “true values” be obtained?• Paradata for TSE analysis
– E.g. level of effort to obtain a response in the broadest sense, condition or value of housing, encounters with gatekeepers
– need a systematic / theory-based framework – quality of paradata / “metadata for paradata”
• Data sets for TSE analysis are scarce
Emerging Research Areas
• Interviewer effects in surveys and their effects on TSE• Methods for predicting poor reporters (i.e., bad
respondents)• Language translation issues for TSE• Completing the feed-back loop between data processors
and survey designers• Methods for estimating the nonresponse-response error
interaction
Some Initiatives Underway for ITSEW 2009
• Lars Lyberg is organizing ITSEW 2009 in Talberg, Sweden for June 14-17, 2009
• Biemer and Lyberg will edit a special issue of Public Opinion Quarterly in 2010 on TSE.
• Special issue on TSE is also being planned for the Journal of Official Statistics in 2010.
• Alan Karr has offered NISS services to host a moderated Wikipedia on Total Survey Error
• Special Topic Sessions for ITSEW 2009– Nonresponse and measurement error interaction and practical
implications (Paul Biemer)– Shared data sets for TSE research (Barry Schouten)– Coverage error (Stephanie Eckman)– POQ and JOS have offered special issues in TSE
Some Initiatives Underway for ITSEW 2009 (cont’d)
Some Issues in Research on Nonresponse-Response Error Interaction
• Is there an interaction between the propensity to respond and the propensity to respond correctly?
• Are reluctant respondents more likely to be misclassified by survey questions?
• What can be done to obtain accurate data from reluctant respondents?
• If so, what are some strategies for nonresponse followup that minimize total survey error?
• What research designs allow analyst to separate nonresponse error and measurement error?
• What models are helpful for understanding the linkage between measurement error and nonresponse error?
P εMeasurement Process Model
ZεZp T
YP εCommon
Cause Model
Z T
Y P εTrue Value
Model
T
Y
Example: Kristen Olsen’s “Causal” Models
Summary
• Papers from the ITSEW 2008 will be available by August 15 on the NISS website
• Call for papers for ITSEW 2009 will be announced in September
• ITSEW 2009 will be held June 14-17 in Sweden• Papers that examine the interaction between two or more
error sources are encouraged.