Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | alexander-stevenson |
View: | 218 times |
Download: | 0 times |
A Comparison of Survey Reports Obtained via Standard
Questionnaire and Event History Calendar
Jeff Moore, Jason Fields, Joanne Pascale,
Gary Benedetto, Martha Stinson, and Anna Chan
U.S. Census Bureau
American Association for Public Opinion Research
May 14-17, 2009
Overview
Background:- SIPP; SIPP “re-engineering”- event history calendar (EHC) methods
Goals and Design of the SIPP-EHC Field Test
Preliminary Results
Summary / Conclusions / Next Steps
SIPPSurvey of Income and Program Participation
- income/wealth/poverty in the U.S.; program participation dynamics/effects
- interviewer-administered; longitudinal
- panel length = 3-4 years
Key Design Feature:
- 3 interviews/year, 4-month reference pd.
SIPP Re-Engineering
Implement Improvements to SIPP- reduce costs- reduce R burden- improve processing system- modernize instrument- expand/enhance use of admin records
Key Design Change:- annual interview, 12-month reference
pd., event history calendar methods
EHC Interviewing
Human Memory- structured/organized- links and associations
EHC Exploits Memory Structure- links between to-be-recalled events
EHC Encourages Active Assistance to Rs- flexible approach to help elicit an
autobiographical “story”
Evaluations of EHC Methods
Many EHC vs. “Q-List” Comparisons- various methods- in general: positive data quality results
BUT, Important Research Gaps- data quality for need-based programs?- extended reference period?
Field Test Goals & Design
Basic Goal:Can an annual EHC interview collect data of comparable quality to standard SIPP?
Basic Design:EHC re-interview of SIPP sample households
Design Details (1)
Sample:SIPP 2004 panel interview cases
- reported on CY-2007 in waves 10-12
EHC re-interview in 2008, about CY-2007
Design Details (2)
SIPP Sample Cases in Two Sites- Illinois (all)- Texas (4 metro areas)
N = 1,096 Wave 10-11-12 Addresses(cooperating wave 11 households)
IL: 487TX: 609
Design Details (3)
EHC Questionnaire- paper-and-pencil- 12-month, CY-2007 reference period- subset of SIPP topics (“domains”)- month-level detail
Sample of Addresses, Not People- post-interview clerical match to SIPP
Design Details (4)
1096 initial sample addresses
Outcomes:
- 935 household interviews (91%)
- 1,922 individual EHC interviews (99%)
- 1,658 EHC Rs matched to SIPP (86%)
FINAL ANALYSIS SAMPLE: 1,620
Primary Evaluation
Compare SIPP and EHC Survey Reports - same people- same time period- same characteristics
Differences Suggest Data Quality Effects
(later: use administrative records for a moredefinitive data quality assessment)
Main Research Questions
1. Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)?
2. Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?
Preliminary Results
3 Government “Welfare” Programs: Food Stamps
Supplemental Security Income (SSI)Women Infants & Children (WIC)
4 Other Characteristics:MedicareSocial Securityemploymentschool enrollment
Results in ContextAlmost All SIPP and EHC Reports Agree
- all characteristics, all months
- in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree
- worst case (employment): 92-94%
Disagreements are RARE EVENTS
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- no “main effect” for method
- no significant method difference in any month
WIC (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- no “main effect” for method
- no significant method difference in any month
Results Summary3 Patterns:
1. EHC = SIPP All Year SSI; WIC (IL)
2. EHC < SIPP All Yearreduced EHC data quality, but not due to longer recall period
15.0%
17.5%
20.0%
22.5%
25.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
MEDICARE -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is constant across months
SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
15.0%
17.5%
20.0%
22.5%
25.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is constant across months
WIC (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is constant across months
FOOD STAMPS (Illinois Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- significant “main effect” for method
- method difference (SIPP > EHC) is essentially constant across months
Results Summary3 Patterns:
1. EHC = SIPP All Year SSI; WIC (IL)
2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL)
3. EHC < SIPP, Early in the Year OnlyEHC data quality may suffer due
– to longer recall period
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
Analysis Summary
- no significant “main effect” for method
- BUT significant variation by month --
JAN-MAY: SIPP > EHC
later months: no difference (reversal?)
EMPLOYMENT -- % Working for Each Month of CY2007 According to the SIPP and EHC Reports
57.5%
60.0%
62.5%
65.0%
67.5%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% W
ork
ing
SIPP % w orking
EHC % w orking
Analysis Summary
- significant “main effect” for method (SIPP > EHC)
- BUT significant variation by month --
JAN-AUG (SEP): SIPP > EHC
later months: no difference
5.0%
7.5%
10.0%
12.5%
15.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% E
nrol
led
SIPP % enrolled
EHC % enrolled
SCHOOL ENROLLMENT -- % Enrolled in Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- no significant “main effect” for method
- BUT significant variation by month
JAN-APR: SIPP > EHC
JUN-JUL: SIPP < EHC
AUG-DEC: no difference
Field Test Overall SummarySuccessful “Proof of Concept”
Overwhelming Finding: SIPP-EHC Agreement
Valuable Lessons to Inform Next Test- larger, broader sample- “correct” timing of field period- automated questionnaire
Specific Data Comparisons are Instructive
Results ImplicationsPattern 1. EHC = SIPP All Year
SSI; WIC (IL)
No evident problems; no reason for concern about data quality in a 12-month EHC interview
Results ImplicationsPattern 2. EHC < SIPP All Year
Medicare; Social Security; WIC (TX); Food Stamps (IL)
Problems with data quality in the EHC treatment, but probably not due to recall length
- less effective screening questions (no D.I.; fewer probes; no local labels)- different definitions
Likely fixes in CAPI
Results ImplicationsPattern 3. EHC < SIPP, Early in the Year Only
Food Stamps (TX); employment;school enrollment
Most cause for concern; longer recall period may cause reduced data quality in the earlier months of the year
Additional research:- why these characteristics?- understand Field Test time lag effects