The Impact of Survey Design Modifications The Impact of Survey Design Modifications on Health Care Utilization Estimates in a on Health Care Utilization Estimates in a
National Longitudinal Health Care SurveyNational Longitudinal Health Care Survey Steven B. Cohen, Ph.D.Steven B. Cohen, Ph.D.
Trena Ezzati-Rice, M.S.Trena Ezzati-Rice, M.S.
Marc Zodet, M.S.Marc Zodet, M.S.
Presentation Presentation
Need for essential data on health care utilization to Need for essential data on health care utilization to inform health care policy and practiceinform health care policy and practice
Description of the Medical Expenditure Panel Survey Description of the Medical Expenditure Panel Survey (MEPS): purpose, design and analytical capacity(MEPS): purpose, design and analytical capacity
Nonresponse and post-stratification adjustmentsNonresponse and post-stratification adjustments Recent survey design modifications: Recent survey design modifications:
(1) CAPI upgrade; (2) Sample Redesign(1) CAPI upgrade; (2) Sample Redesign Evaluation ofEvaluation of impact of design modifications on health impact of design modifications on health
care utilization estimatescare utilization estimates Impact of design modifications on model-based Impact of design modifications on model-based
analyses of health care useanalyses of health care use DiscussionDiscussion
Medical Expenditure Panel Medical Expenditure Panel Survey (MEPS)Survey (MEPS)
Annual Survey of 14,000 households:Annual Survey of 14,000 households: provides national estimates of health care use, provides national estimates of health care use, expenditures, insurance coverage, sources of payment, expenditures, insurance coverage, sources of payment, access to care and health care qualityaccess to care and health care quality
Permits studies of:Permits studies of: Distribution of expenditures and sources of paymentDistribution of expenditures and sources of payment Role of demographics, family structure, insuranceRole of demographics, family structure, insurance Expenditures for specific conditionsExpenditures for specific conditions Trends over timeTrends over time
Key Features of MEPS-HCKey Features of MEPS-HC
Survey of U.S. civilian noninstitutionalized populationSurvey of U.S. civilian noninstitutionalized population Sub-sample of respondents to the National Health Interview Survey Sub-sample of respondents to the National Health Interview Survey
(NHIS)(NHIS) Oversample of minorities and other target groupsOversample of minorities and other target groups Panel Survey – new panel introduced each yearPanel Survey – new panel introduced each year
– Continuous data collection over 2 ½ year periodContinuous data collection over 2 ½ year period
– 5 in-person interviews (CAPI) 5 in-person interviews (CAPI)
– Data from 1st year of new panel combined with data from 2nd Data from 1st year of new panel combined with data from 2nd year of previous panelyear of previous panel
MEPS Overlapping PanelsMEPS Overlapping Panels(Panels 13 and 14)(Panels 13 and 14)
MEPS Household Component MEPS Panel 13
2008-2009
Round 2
Round 3 Round 4 Round 5
Round 1 Round 2 Round 3
MEPS Panel 142009-2010
1/1/2008 1/1/2009
Round 1
NHISNHIS20072007
NHISNHIS20082008 Round 4 Round 5
HC - PurposeHC - Purpose
Estimates annual health care use and expendituresEstimates annual health care use and expenditures Provides distributional estimatesProvides distributional estimates Supports person and family level analysisSupports person and family level analysis Tracks changes in insurance coverage and Tracks changes in insurance coverage and
employmentemployment Longitudinal design; linkage to National Health Longitudinal design; linkage to National Health
Interview Survey (NHIS)Interview Survey (NHIS)
Tool Chest of Methods to Maximize Tool Chest of Methods to Maximize Survey ResponseSurvey Response
Recruitment of experienced interviews and bilingualRecruitment of experienced interviews and bilingual 10+ days training (including procedures for obtaining 10+ days training (including procedures for obtaining
signed consents)signed consents) Uses of MEPS data as reference materials for interviewersUses of MEPS data as reference materials for interviewers Periodic retraining and special trainings (e.g. methods to Periodic retraining and special trainings (e.g. methods to
improve response rates)improve response rates) Respondent remunerationRespondent remuneration Advance mailings from co-sponsors of surveyAdvance mailings from co-sponsors of survey Monthly planning calendar and MEPS DVDMonthly planning calendar and MEPS DVD Daily emails to interviewers regarding interviewing Daily emails to interviewers regarding interviewing
progressprogress Multiple contacts for refusal conversionsMultiple contacts for refusal conversions
MEPS Response RatesMEPS Response Rates
Multiplicative response rates (RR): product of Multiplicative response rates (RR): product of NHIS RR and NHIS RR and MEPS RR (multiplicative function of round specific MEPS RR (multiplicative function of round specific
RR): RR): MEPS rounds 1-3 of MEPS rounds 1-3 of new panel (YR1 estimates) panel (YR1 estimates) MEPS rounds 3-5 of MEPS rounds 3-5 of old panel (YR2 estimates)panel (YR2 estimates)
MEPS Response Rates MEPS Response Rates (RR)(RR)
Overall annual RR (~65%)Overall annual RR (~65%) Highest RR 1Highest RR 1stst year, new panel (~66-71%) year, new panel (~66-71%) Lowest RR 2Lowest RR 2ndnd year, old panel (~63-65%) year, old panel (~63-65%) Post-survey nonresponse adjustmentsPost-survey nonresponse adjustments
– Dwelling unit levelDwelling unit level
– Person level survey attritionPerson level survey attrition
NHIS variables used as potential NHIS variables used as potential covariates in forming DU covariates in forming DU
NR adjustment cellsNR adjustment cells
DemographicDemographic Household Household CharacteristicCharacteristic
Socio-Socio-Economic Economic
StatusStatusGeographicGeographic Health Health
Age Age DU sizeDU size Poverty statusPoverty status Census regionCensus region Health statusHealth status
Race/ethnicityRace/ethnicity Has phoneHas phone EducationEducation MSA sizeMSA size Need helpNeed help
Marital statusMarital status Working/reason Working/reason not work (e.g., not work (e.g., attending attending school, retired, school, retired, etc,)etc,)
IncomeIncome MSA/nonMSAMSA/nonMSA
GenderGender Type of PSUType of PSU Employment Employment statusstatus
Urban/RuralUrban/Rural
Any AsianAny Asian
Any BlackAny Black
NEW NHIS variables added as potential NEW NHIS variables added as potential covariates in forming DU NR adjustment cellscovariates in forming DU NR adjustment cells
DemographicDemographic Household Household CharacteristicCharacteristic
Socio-Socio-Economic Economic
StatusStatusHealth Health
Interview Interview language language
Type of home – Type of home – house, Apt., etc.house, Apt., etc.
Category of Category of medical medical expenseexpense
Number of Number of nights in nights in hospitalhospital
U.S. CitizenshipU.S. Citizenship Time no phoneTime no phone Home Home ownershipownership
Healthcare Healthcare coveragecoverage
Born in USBorn in US
Adjustment factorAdjustment factor
Within each adjustment cell:Within each adjustment cell:
A(c) =A(c) =
ratio of the sum of weights of all eligible (E) units ratio of the sum of weights of all eligible (E) units in the cell to the sum of weights of only the in the cell to the sum of weights of only the respondents (R) in the cellrespondents (R) in the cell
)(1)(
)(1)(
iWiR
iWiE
ci
ci
Person Level Adjustments:Person Level Adjustments:Annual EstimatesAnnual Estimates
Each panel weighted separatelyEach panel weighted separately
Nonresponse adjustment for survey Nonresponse adjustment for survey attrition attrition
Final Poststratification adjustment –Final Poststratification adjustment –CPS 12/31:CPS 12/31:
age, race/ethnicity, sex, region, age, race/ethnicity, sex, region, MSA status, MSA status, poverty statuspoverty status
Person Level (survey attrition) Person Level (survey attrition) Nonresponse Adjustment CovariatesNonresponse Adjustment Covariates
Factors associated with survey attrition (after R1)Factors associated with survey attrition (after R1)
– Indicator for initial refusal to R1interview Indicator for initial refusal to R1interview
– Family sizeFamily size
– AgeAge
– MSA, census regionMSA, census region
– Marital status (family reference person)Marital status (family reference person)
– Race/ethnicityRace/ethnicity
– Education of reference personEducation of reference person
– Employment statusEmployment status
– Health insurance statusHealth insurance status
– Total expenditures (in yr 1 for yr 2 adj.)Total expenditures (in yr 1 for yr 2 adj.)
– # doctor visits (in yr 1)# doctor visits (in yr 1)
– Self reported health statusSelf reported health status
Round 1 Round 2 Round 3 Round 4 Round 5
2009 2010
Longitudinal Estimation StrategyLongitudinal Estimation Strategy
Individuals in the 2009 sample with positive weights that left the civilian population prior to 2010, with no return
2009 sample also responding in 2010 with complete information for both 2009 and 2010
&
MEPS Redesign in 2007MEPS Redesign in 2007
Re-engineered CAPI Interview: Windows-based Re-engineered CAPI Interview: Windows-based Platform replaces DOS-based system for Panel 12Platform replaces DOS-based system for Panel 12
New NHIS Sample Design Introduced in 2006: MEPS New NHIS Sample Design Introduced in 2006: MEPS Panel 12 selected from redesigned NHIS samplePanel 12 selected from redesigned NHIS sample
Year 2 of MEPS Panel 11 based on original MEPS Year 2 of MEPS Panel 11 based on original MEPS survey designsurvey design
The overlapping panel structure in MEPS allows for a The overlapping panel structure in MEPS allows for a comparison of survey estimates across the alternative comparison of survey estimates across the alternative designed for the same time period designed for the same time period
Evaluation of Concordance of Healthcare Evaluation of Concordance of Healthcare Utilization Estimates: Comparison of results Utilization Estimates: Comparison of results
from new and original designsfrom new and original designs
MEPS has overlapping panel design: 1st year of new MEPS has overlapping panel design: 1st year of new panel combined with data from 2nd year of previous panel combined with data from 2nd year of previous year’s panel to yield annual datayear’s panel to yield annual data
Multiplicative response rates: product of NHIS RR and Multiplicative response rates: product of NHIS RR and MEPS RR (multiplicative function of round specific RR: MEPS RR (multiplicative function of round specific RR: 3 rounds for new panel/5 rounds for old panel)3 rounds for new panel/5 rounds for old panel)
Detailed adjustments for survey nonresponse and Detailed adjustments for survey nonresponse and poststratification: poststratification:
CompareCompare 2007 health care utilization estimates based 2007 health care utilization estimates based on new designon new design (MEPS Panel 12 – Year 1) (MEPS Panel 12 – Year 1) with original with original designdesign (MEPS Panel 11-Year 2) (MEPS Panel 11-Year 2)
Testing for Survey Redesign EffectsTesting for Survey Redesign Effects
Comparisons of panel specific national Comparisons of panel specific national health care utilizationestimates derived from the MEPS for the following health careservices: ambulatory visits (office- based visits and outpatient facility
visits) in-patient stays ER visits dental visits prescribed medicine purchases For the overall population, and further subset by ageclassification (0-17, 18-64, 65+) Model-based tests for survey redesign effectsModel-based tests for survey redesign effects
Capacity of MEPS to Produce Capacity of MEPS to Produce Comparable NHIS Estimates of Comparable NHIS Estimates of
Health Care UtilizationHealth Care Utilization
The following NHIS measures of health care utilization The following NHIS measures of health care utilization were selected in support of these analyses:were selected in support of these analyses:
Have you been hospitalized OVERNIGHT in the past 12 months? Have you been hospitalized OVERNIGHT in the past 12 months? (yes; no; refused/not ascertained/DK)(yes; no; refused/not ascertained/DK)
How many different times did you stay in any hospital overnight or How many different times did you stay in any hospital overnight or longer DURING THE PAST 12 MONTHS? (#; refused/not longer DURING THE PAST 12 MONTHS? (#; refused/not ascertained/DK) ascertained/DK)
Altogether how many nights were you in the hospital DURING THE Altogether how many nights were you in the hospital DURING THE PAST 12 MONTHS? (#; refused/not ascertained/DK)PAST 12 MONTHS? (#; refused/not ascertained/DK)
During the past 12 MONTHS did you receive care from doctors or During the past 12 MONTHS did you receive care from doctors or other health care professionals 10 or more times? Do not include other health care professionals 10 or more times? Do not include telephone calls. (yes; no; refused/not ascertained/DK) telephone calls. (yes; no; refused/not ascertained/DK)
DURING THE PAST 12 MONTHS, have you delayed seeking medical DURING THE PAST 12 MONTHS, have you delayed seeking medical care because of worry about the cost? (yes; no; refused/not care because of worry about the cost? (yes; no; refused/not ascertained/DK) ascertained/DK)
DURING THE PAST 12 MONTHS, was there any time when you DURING THE PAST 12 MONTHS, was there any time when you needed medical care, but did not get it because you/the family needed medical care, but did not get it because you/the family couldn't afford it? (yes; no; refused/not ascertained/DK)couldn't afford it? (yes; no; refused/not ascertained/DK)
Options for aligning redesign-based estimates with the original design
OptionPeriod of Applicability
Constraints
Restrict time trend analyses to sample with old design
Year(s) with overlap between redesign and prior design
Loss in precision
Implementation of measure specific adjustments to a set of estimation weights
When the survey redesign is implemented and subsequent years
Introduction of greater variability in resultant survey estimates ; complicates model based analyses
Implementation of “bridging” adjustments to the primary survey estimation weight
When the survey redesign is implemented and subsequent years
Dependence on within survey adjustments or availability of comparable external data source for national control totals
No additional adjustments for redesign
When the survey redesign is implemented and subsequent years
Possibility of differences detected in trends partially attributable to redesign
SummarySummary
Need for accurate and reliable national data Need for accurate and reliable national data on health care utilization to inform policy and on health care utilization to inform policy and practice practice
MEPS design features and analytical capacityMEPS design features and analytical capacity Statistical, methodological and operational Statistical, methodological and operational
design features to adjust for nonresponse and design features to adjust for nonresponse and attritionattrition
Evaluation of impact MEPS redesign on health Evaluation of impact MEPS redesign on health care utilization estimatescare utilization estimates
Impact on model based studiesImpact on model based studies Some evidence of redesign effectSome evidence of redesign effect
Strategies to Improve Strategies to Improve AccuracyAccuracy
MEPS includes a linked survey of medical MEPS includes a linked survey of medical providers for expenditures: use of medical providers for expenditures: use of medical event information to evaluate household reports event information to evaluate household reports of health care useof health care use
MEPS data periodically linked to Medicare MEPS data periodically linked to Medicare claims data for evaluations: permits claims data for evaluations: permits examination of accuracy of household reported examination of accuracy of household reported data data
Implement additional improvements to the CAPI Implement additional improvements to the CAPI interview and enhanced post-survey adjustment interview and enhanced post-survey adjustment strategies strategies
