Using the Data that We Collect

Data Sources and Evaluation Tips

Mike Smith, MSPHMCH EpidemiologistDirector, Division of Research and PlanningSC DHEC(803) 898-3740smithm4@dhec.sc.gov

Overview

• Why do we need data?

• What types of MCH data are available and where can we get it?

• How can we evaluate our programs?

• So what?

Why do we need data?

Why do we need data?

• Knowing what the real issues are – e.g. smoking during pregnancy by race

• Knowing the potential for impact and targeting interventions

Why do we need data?

• Knowing whether or not our programs or interventions work at all (or where to improve them)

• Knowing whether or not our programs or interventions work in our setting or with out population

What types of MCH data are available?

SC Specific Data Sources

• Vital Records/registries – attempt to capture data around all events that occur to state residents– birth certificate – death certificate – reportable fetal deaths– induced terminations of pregnancy– birth defects registry

SC Specific Data Sources

• Surveys – attempt to collect more detailed data on fewer individuals that represent the larger population of interest– Pregnancy Risk Assessment Monitoring System

(PRAMS)– Behavioral Risk Factor Surveillance System (BRFSS)– Youth Risk Behavior Survey (YRBS)– National Survey of Children’s Health (NSCH)– National Survey of Children with Special Health Care

Needs (NSCSHCN)

Vital Records - Background• The National Center for Health Statistics (NCHS) creates

standard forms that they recommend US states and territories use to document vital events (but can not require states to use these forms).

• Periodically, NCHS releases revised versions of the forms and recommends that states implement them.

• Some states implement them earlier than others, leading to potential issues around comparability.

Example – Comparability• NCHS revised the standard birth certificate (BC) in 2003. SC

implemented revision in 2004.

• Some information from the SC BC from 2004-Present is not available for prior years (ex. pre-pregnancy weight, gestational diabetes).

• Some information from the SC BC is available before and after 2004, but is not directly comparable (ex. mother/father’s education, smoking status).

• Because some states have not moved to the 2003 NCHS standard birth certificate, across-state comparisons can be difficult for some information.

Vital Records Data Sources

• Birth Certificate– Includes information such as: maternal height

and weight, race/ethnicity, age, education, county of residence, smoking before and during pregnancy, delivery payment method, number of previous live births, risk factors, infections, delivery method, birthweight, gestational age, abnormal newborn conditions, congenital anomalies

Vital Records Data Sources

• Death Certificate– Can be combined with birth certificate data to

examine infant mortality

• Report of Fetal Death– Required to be reported only if beyond 20

weeks gestation and 350 grams or heavier– Some of the same information as on the birth

certificate; causes and conditions contributing to fetal death

Vital Records Data Sources

• Induced Termination of Pregnancy– Basic demographic data, gestational age,

informed written consent– Many induced terminations of pregnancy to

SC residents occur out of state

• SC vital records data available through the DHEC SCAN system: http://scangis.dhec.sc.gov/scan/index.aspx

Registry Data

• SC Birth Defects Program– Data available beginning in 2008– Birth defects recommended by the National

Birth Defects Prevention Network– Demographic and diagnostic data collected

through medical record abstraction

Where can we get data?

• SC vital records data available through the DHEC SCAN system: http://scangis.dhec.sc.gov/scan/index.aspx• Contact: Daniela Nitcheva, nitchedk@dhec.sc.gov

• SC Birth Defects Program data available through the DHEC Environmental Public Health Tracking Program: http://www.scdhec.gov/administration/epht/BirthDefects.htm• Contact: Mike Smith, smithm4@dhec.sc.gov

VR Data for MCH Research• Vital records data is very commonly used for MCH

research.

• Some strengths:– Standardized (mostly) national data– Data collected for all events– Available for county or smaller geographic levels

• Some weaknesses:– Much of the information is self-reported– Other data quality concerns– Staggered implementation of revisions

Survey Data

• Pregnancy Risk Assessment Monitoring System (PRAMS)– Survey data about maternal behaviors, attitudes, and

experiences before, during, and shortly after pregnancy

– Topic such as physical activity, breastfeeding, postpartum depression symptoms, oral health, stressful life events

– Available through DHEC SCAN system: http://scangis.dhec.sc.gov/scan/index.aspx

• Contact: Mike Smith, smithm4@dhec.sc.gov

Survey Data

• Behavioral Risk Factor Surveillance System (BRFSS)– Survey data about general health behaviors and

status for the adult population; often subset to women of reproductive age

– Topics such as healthcare access, tobacco use, disease screenings, immunizations, women’s health

– Available through SC BRFSS website: http://www.scdhec.gov/hs/epidata/brfss2010.htm

• Contact: Shae Sutton, suttonsr@dhec.sc.gov

Survey Data

• Youth Risk Behavior Survey (YRBS)– Survey data about the health and risk behaviors of

middle and high schoolers; conducted by the SC Department of Education

– Topics such as tobacco and alcohol use, diet, physical activity, sexual behaviors

– Available through SC YRBS website: http://ed.sc.gov/agency/se/Instructional-Practices-andEvaluations/SouthCarolinaYouthRiskBehaviorSurveyYRBS.cfm

• Contact: Kimberly Stewart, kwstewart@ed.sc.gov

Survey Data

• National Survey of Children’s Health (NSCH)– Survey data about the health of children (<18) by

state; collected by US DHHS HRSA– Topics such as physical and mental health status,

access to quality health care, family, neighborhood, social context

– Available through HRSA Child Health Data website: http://www.childhealthdata.org/

Survey Data

• National Survey of Children with Special Health Care Needs (NSCSHCN)– Survey data about the health and functional status of

children with special health care needs; collected by US DHHS HRSA

– Topics such as access to quality health care, care coordination of services, access to a medical home, transition services for youth, and impact of the chronic conditions(s) on the child’s family

– Available through HRSA Child Health Data website: http://www.childhealthdata.org/

Survey Data for MCH Research• Survey data is also very commonly used for MCH

research.

• Some strengths:– Often national data– Data collection on a variety of topics with flexibility for changing

topics and questions regularly

• Some weaknesses:– Nearly all information is self-reported– Often only available at the state level– Complex sampling and weighting schemes (applicable if you are

analyzing your own datasets)

Population-Based Data

• All of these vital records, registry, andsurvey data sources are great for determining what the real issues and planning and targeting interventions.

• These data sources are usually not very good for evaluating the effectiveness of programs that are implemented on smaller levels.

How can we evaluate our programs and intervention?

Does our program or intervention work at all?

Does our program or intervention work in our setting?

Types of Program Evaluation

• Process evaluation– Are we sticking to the planned model faithfully?– Measures such as: number of meetings held, number

of attendees, number of clients served, etc.

• Outcome evaluation– Have we impacted the outcome of interest?– Measures such as: knowledge gained, low birth

weight deliveries, contraceptive use– Often requires rigorous evaluation study designs

Basic Outcome Evaluation

• You have an intervention that you want to impact an outcome, so you implement the intervention (X) and observe (O) the outcome

X O

• Interventions and observations are the key elements to outcome evaluation

Basic Outcome Evaluation

• You want to be confident that: – your intervention is actually what causes

changes in the outcome (internal validity)– your intervention might result in similar

changes in the outcome in other populations or settings (external validity)

Basic Outcome Evaluation

• So, it is important to understand what may constitute threats to internal and external validity and how to design evaluations that avoid those threats.

Threats to Internal Validity• Ambiguous Temporal Precedence: Could the change in

outcome have occurred before the intervention?– Possible Evaluation Design: add a pre-test

O1 X O2

• Selection: Could those receiving the intervention be more or less likely to have the outcome than a random person (e.g. volunteering)?– Possible Evaluation Design: split volunteers into two groups and

apply intervention at different times

O1 X O2

O1 O2 X

Threats to Internal Validity• Testing: Could the change in outcome be due to

participants learning the test or guessing what the investigators are looking for in observation?– Possible Evaluation Design: add a series of tests and a

control group or have a long time between tests

O1 O2 X O3

O1 O2 O3

Other Threats to Consider• History: Could another event or program have

caused the difference in outcome?

• Maturation: Could natural changes (e.g. growth, fatigue, experience) have resulted in the outcome?

– Possible Evaluation Design: select intervention and comparison groups from the same geographic location and observing outcomes at about the same time so that groups are equally impacted by history

Other Threats to Consider• Regression Artifacts: If participants are selected

because they scored lower or higher than average, many will naturally regress back toward the average (e.g. periods of stress).

• Attrition: Are participants that drop out more or less likely to have a different outcome?

• Threats to internal validity can come in multiples.

Threats to External Validity

• Interaction of Causal Relationship with Units: Is the study population representative of the population of interest? Are participants of especially high/low risk?– Implication: You can only generalize to populations that

are comparable on key characteristics to those that complete your evaluation, so selection and attrition are very important to monitor.

Threats to External Validity

• Interaction of Causal Relationship with Outcomes: Can an evaluation result for one outcome be generalized for a similar outcome?– Example: can an evaluation of a program to increase high

school graduate rates be assumed to also improve SAT scores?

– Implication: Be cautious in overstating or over-interpreting the results of an evaluation project.

Threats to External Validity

• Interaction of Causal Relationship with Settings: Does an intervention work similarly in urban and rural areas? In public and private schools?– Implication: Be sure that an evaluation project is set in an

area that allows for broad participation across settings of interest or conduct evaluations in multiple sites.

Randomization is Magic

• Random sampling of individuals to participate in the evaluation from the population of interest is the most effective way to achieve high external validity.

• Random assignment of participants to intervention and control groups is the most effective way to achieve high internal validity.

• Of course, randomization is difficult in the real world.

So what?

Questions and Discussion