1

Total Survey Error: Past, Present, and Future

Robert M. GrovesU.S. Census Bureau

Lars LybergStatistics Sweden and Stockholm University

2

Outline

1. Evolution of “total survey error” (TSE)

2. Weaknesses of the TSE framework

3. Strengths of the TSE framework

4. Future of surveys and research agenda ideas

3

Deming (1944) “On Errors in Surveys”

• American Sociological Review!

• First listing of sources of problems, beyond sampling, facing surveys

• The 13 factors

4

Deming’s 13 factors-The 13 factors that affect the usefulness of a

survey

-To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available

-To point out the futility of concentrating on only one or two of them

-To point out the need for theories of bias and variability that correlate accumulated experience

5

6

Comments on Deming (1944)

• Does include nonresponse, sampling, interviewer effects, mode effects, various other measurement errors, and processing errors

• Omits coverage errors• Includes nonstatistical notions (auspices)• Includes estimation step errors (wrong

weighting)• “Total survey error” not used as a term

7

Sampling Text Treatment of Total Survey Error

• Kish, Survey Sampling, 1965– Graphic on biases– 65 of 643 pages on various errors, with

specified relationship among errors

8

Sampling Biases

Frame biases

“Consistent” Sampling Bias

Constant Statistical Bias

Nonsampling

Biases

Noncoverage

NonresponseNonobservation

Field: data collection

Office: processingObservation

9

Sampling Text Treatment of Total Survey Error

• Särndal, Swensson, Wretman, Model Assisted Survey Sampling, 1992– Part IV, 124 pp. of 694, coverage,

nonresponse, measurement error; omits processing error

• Lohr, Sampling Design and Analysis, 1999– 34 of 500 pages, nonresponse, randomized

response

10

Other textbooks

• Cochran (1953). Sampling Techniques.40 pages in concluding chapter on “sources of

error in surveys”• Deming (1950). Some Theory of Sampling.

Starts with the 1944 factors but then continues with pure sampling

• Hansen, Hurwitz and Madow (1953). Sample Survey Methods and Theory, Vol 1. Nine pages on survey errors.

• Zarkovich 1966. Quality of Statistical Data.

11

Total Survey Error (1979)Anderson, Kasper, Frankel, and Associates

• Empirical studies on nonresponse, measurement, and processing errors for health survey data

• Initial total survey error framework in more elaborated nested structure

12

Total Error

VariableError

Sampling

Nonsampling

Field

Processing

Bias

Nonsampling

Observation

Field

Processing

Sampling

Frame

Consistent

Nonobservation

Noncoverage

Nonresponse

13

Survey Errors and Survey Costs (1989), Groves

• Attempts conceptual linkages between total survey error framework and– psychometric true score theories– econometric measurement error and selection bias

notions

• Ignores processing error• Highest conceptual break on variance vs. bias• Second conceptual break on errors of

nonobservation vs. errors of observation

14

Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode

Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode

Errors ofNonobservation

ObservationalErrors

Bias

Errors ofNonobservation

ObservationalErrors

Variance

Mean Square Error

construct validitytheoretical validityempirical validityreliability

criterion validity - predictive validity - concurrent validity

15

Nonsampling Error in Surveys (1992), Lessler and Kalsbeek

• Evokes “total survey design” more than total survey error

• Omits processing error

16

Components of Error Topics

Frame errorsMissing elements

Nonpopulation elements

Unrecognized multiplicities

Improper use of clustered frames

Sampling errors

Nonresponse errorsDeterministic vs. stochastic view of nonresponse

Unit nonresponse

Item nonresponse

Measurement errorsError models of numeric and categorical data

Studies with and without special data collections

17

Introduction to Survey Quality, (2003), Biemer and Lyberg

• Major division of sampling and nonsampling error

• Adds “specification error” (a la “construct validity”) or relevance error

• Formally discusses process quality

• Discusses “fitness for use” as quality definition

18

Sources of Error Types of Error

Specification error Concepts

Objectives

Data element

Frame error Omissions

Erroneous inclusions

Duplications

Nonresponse error Whole unit

Within unit

Item

Incomplete Information

Measurement error Information system

Setting

Mode of data collection

Respondent

Interview

Instrument

Processing error Editing

Data entry

Coding

Weighting

Tabulation

19

Survey Methodology, (2004) Groves, Fowler, Couper, Lepkowski, Singer,

Tourangeau• Notes twin inferential processes in surveys

– from a datum reported to the given construct of a sampled unit

– from estimate based on respondents to the target population parameter

• Links inferential steps to error sources

20

Construct Inferential Population

Measurement

Response

Target Population

Sampling Frame

Sample

Validity

Measurement Error

Coverage

Error

Sampling

Error

Measurement Representation

Respondents

Nonresponse

ErrorEdited Data

ProcessingError

Survey Statistic

21

Key Statistical Developments in Total Survey Error 1

• Errors of observers can be correlated (1902), Karl Pearson

• Interpenetrating samples (1946), Mahalanobis

• Criteria for true values (1951), Hansen, Hurwitz, Marks and Mauldin

• Essential survey conditions, correlated response variance (1959), H-H-Bershad

• BC survey model “mixed-error model”(1961), H-H-B

22

• Interviewer effects using ANOVA (Kish 1962)• Simple response variance via reinterviews (1964), H-H-

Pritzker• Relaxed assumptions of zero covariance of true values and

response deviations (1964, 1974), Fellegi• Errors of Measurement (1968), Cochran• Estimating model components via basic study schemes

using replication, interpenetreation and combinations of the two (1969), Bailar and Dalenius

• Estimating nonsampling variance using mixed linerar models (1978), Hartley and Rao

• “Error Profile” of Current Population Survey (1978), Brooks and Bailar

• Multi-method multi-trait models on survey measures (1984), Wothke…Browne

Key Statistical Developments in Total Survey Error 2

23

• Measurement of imputation error variance through multiple imputation (1987), Rubin

• Total error model for PES (1991), Mulry and Spencer• Measurement Errors in Surveys (1991)

Section (Biemer and Stokes..Colm..Fuller and others– attempts to juxtapose psychometric notions with

survey statistical notions of measurement error• Latent class model applications to survey errors (late

1990’s), Biemer, Tucker and others• Measurement error effects on analysis (1997) Biemer

and Trewin

Key Statistical Developments in Total Survey Error 3

24

• Certain components understudied– coverage error variance– nonresponse error variance– all processing errors– biases in general

• The movement from measurement of MSE to CBM, SOP thereby getting a situation where Var becomes an approximation of MSE (error-free processes)

• Simultaneous treatment of more than one error source (ITSEW)

Key Statistical Developments in Total Survey Error 4

25

Summary of the Evolution of “Total Survey Error”

• Roots in cautioning against sole attention to sampling error

• Framework contains statistical and nonstatistical notions

• Most statistical attention on variance components, most on measurement error variance

• Late 1970’s attention to “total survey design” (methods and theory for resource allocation to the control of various sources of error

• 1980’s-1990’s attempt to import psychometric notions, CASM

• Key omissions in research

26

Weaknesses of the Common Usage of “Total Survey Error”

– Notably a user perspective is missing– Key quality dimensions are missing in the TSE

paradigm– User often cannot or prefers not to question

accuracy– The complexity does not invite outside scrutiny of

accuracy– Users not really informed about real levels of error

or uncertainty– We don’t really know how users perceive

information on errors

27

Other Weaknesses of the Total Survey Error Paradigm 1

1. Lack of routine measurementsNo agency does thisError/quality profiles are useful but rare

2. Ineffective influence on professional standards Little expansion beyond sampling error in practice

Press releases on Federal statistics rarely contain even sampling errors

Survey error research compartmentalized rather than integrated

Methodologists tend to specializeRoot causes of error often still missingHow about OMB’s requirement of NR bias studies if

NR expects to exceed 20%?

28

Other Weaknesses of the Total Survey Error Paradigm 2

3.Large burden on design of some estimators

Interpenetration, reinterviews for variance estimation complicated and costly

Intractable expressions for some components

4. Some assumptions unrealistic

29

Strengths of the Total Survey Error Framework

1. Taxonomic decomposition of errors• nomenclature for different components

2. Separation of phenomena affecting statistics in different ways

• variance vs. bias; observation vs. nonobservation; respondent/interviewer/measurement task; processing

3. Conceptual foundation of the field of survey methodology• subfields defined by errors

4. Tool to identifying gaps in the research literature• e.g., where are the error evaluation papers on

processing?

30

Needed Steps in a Research Agenda for Total Survey Error 1

1. Integrating causal models of survey errors• cognitive psychological mechanisms (anchoring,

recall decay)

2. Research on interplay of two or more error sources jointly

• e.g., nonresponse and measurement error

3. Research on the interplay of biases and variances

• e.g., does simple response variance increase accompany some response bias reductions (self-administration effects)?

31

4. Guidance on tradeoffs between quality measurement and quality maximization and between measures and developing error-free processes- how much should we spend on quality enhancement vs. measurement of quality (Spencer, 1985)?

5. Integrating other notions of quality into the total survey error paradigm- if “fitness for use” predominates as a conceptual base, how can we launch research that incorporates error variation associated with different uses? Australian Bureau of Statistics

Needed Steps in a Research Agenda for Total Survey Error 2

32

Needed Steps in a Research Agenda for Total Survey Error 3

6. Exploiting a multiple-mode, multiple frame, multiple phase survey world

7. Need for methodological studies to assist the user

8. Costs and risks9. Develop theories for optimal design of

specific operations, design principles10. More standards?