Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | rafe-curtis |
View: | 218 times |
Download: | 0 times |
Choosing measures and evaluation tools in social intervention
Methodologies for a new era summer school
School of Applied Social Studies, University College Cork
Wednesday 22nd June 2011
Matthew Morton, Jennifer Burton
Purposes of Measures
Implementation Adherence, quality, responsiveness, exposure
Outcome Changes (or baseline screening) in attitudes,
behaviors, wellbeing
Theory of change
“Blueprint of the building blocks needed to achieve the long-term goals of a social change initiative.”
Informed by… Existing evidence Psychosocial Theory Formative research Stakeholder input
Main Components of a Theory of Change
Component Description
Inputs Resources that go into a project/program (e.g., funding, staffing, equipment, curriculum materials)
Activities What we do.
Outputs What we produce. Tangible and can be counted. (e.g., # trainings, attendance, partnerships created, quality processes)
Outcomes Why we do it. Behavioral/attitudinal changes, result from outputs. (e.g., quit smoking, reduce depression, better parenting)
Impacts Long-term changes that result from accumulation of outcomes. Strategic. (e.g., reduce infant mortality)
Kusek & Rist 2004
ToC Assumptions
• External: The program is feasible, addresses real problems/needs, built on accurate beliefs of human nature, and culturally/politically appropriate.
• Internal: The program’s process and implementation will be implemented as intended by the TOC.
• Evaluative: The data captures the intended indicators/outcomes and is accessible. The methodology credibly assesses the TOC.
Exercise – 1. Topic clusters
1. Children2. Disabled3. Discrimination4. Elderly5. Environment6. Health7. Homelessness8. Immigration9. Youth
Exercise – Outcomes Mapping4 months 1 year 5 years
Level I (e.g., child)
Level II (e.g., family)
Level III (e.g., neighborhood)
Think: (a) changes, (b) specific, (c) realistic, (d) measureable
Types of MeasuresStrengths Weaknesses
Self-report (e.g., q’airre) Relatively easy to implement; helps with feelings/attitudes
Self-report biases
Third party (e.g., parent, teacher, psychologist)
Avoids self-report biases; alternative perspective
Limited perspectives; more complicated than self-report to implement
Observational (e.g., observer, video)
More objective; less subject to participant bias
Difficult & costly; harder to standardize coding
Institutional data (e.g., school or gov’t records)
Often directly policy-relevant; objectivity; can avoid burden of surveys
Can be limited; may not capture important “softer” psychosocial outcomes
What Makes a Good Measure?
• Matches Theory of Change• Appropriate for population (age, culture, language, etc.)• Well-tested
• Reliability – The extent to which a test can discriminate between one subject and another
• Validity - The extent to which a test measures what is is intended to measure
Reliability
INTERNAL CONSISTENCY
Measures level of agreement among individual test items
INTER-RATER RELIABILITY
Measures agreement between different raters
TEST-RETEST RELIABILITY
Measures agreement between successive ratings over time
Inter-rater Reliability
For continuous variablesThe Intra-Class Correlation
Coefficient (ICC) measures the extent to which overall test variance is attributable to variability between patients as opposed to random error variability (ie signal-to-noise ratio)
Inter-rater Reliability
For binary (dichotomous variable)Kappa measures the proportion of
agreement above that by chanceFor ordered categorical data,
weighted kappa quantifies degree of disagreement
Values range 0-1 (1=perfect agreement) 0.4-0.6 = Moderate >0.80 = Very high
Internal ConsistencyMeasures the extent to which items of the same
scale (or subscale) are inter-correlatedCronbach increases as (a) average inter-item
correlation rises or (b) number of items risesNo formal test statistic for
>0.5 = moderate >0.8 = excellent
Types of Validity
CONTENT (FACE) To what extent do the items reflect the construct being assessed?
CRITERION (CONCURRENT)
How does the test perform against the ‘gold standard’?
CRITERION (PREDICTIVE)
Do results on the test predict future outcomes? (i.e., why it’s important)
CONSTRUCT Do all the items (in a subscale) relate to the same concept?
CONVERGENT Do the test results correlate with those of other theoretically related tests?
DIVERGENT Is the correlation low between the test and others which should be unrelated?
Benefits of community based parenting groups for hard-to-manage
children: Findings from the Family Nurturing
Network Trial
Jenny BurtonFrances Gardner
University of OxfordCentre for Evidence Based Intervention
Funded by the Esmee Fairbairn Foundation
DEMOGRAPHIC INFORMATION
Parent How old? Status? Married, living as married, single, divorced etc. Employed? Educational achievement?
Child Gender? Age? Siblings?
PRIMARY AIMS OF INTERVENTION
Parents Help provide parents with necessary skills with which to deal with
their hard to manage children
Target Child Decrease child negative behaviour
eg. Non-compliance, tantrums, yelling,destructive, aggressive behaviours.
SECONDARY AIMS OF INTERVENTION
Parent
Improve sense of parenting competence Improve mood Improve their relationships Worst Sibling Decrease child negative behaviour in other siblings
MEASURES USED TO ASSESS PARENT OUTCOME AT BASELINE AND POST INTERVENTION (I)
Questionnaires
Parenting Scale(Arnold, O’Leary et al, 1993) Beck Depression Inventory (Beck, 1972)
Parent Sense of Competence (PSOC; Johnston & Mash, 1989) Dyadic Adjustment Scale (DAS; Spanier, 1976)
MEASURES USED TO ASSESS CHILD OUTCOME AT BASELINE AND POST INTERVENTION (II)
Parent-report questionnaires:
Eyberg Child Behavior Inventory (ECBI; Robinson et al,1980)
Parent interview: Semi-structured interview re conduct & hyperactivity: Parent Account of
Child Symptoms (PACS; Taylor et al,1986)
MEASURES USED TO ASSESS CHILD OUTCOME AT BASELINE AND POST INTERVENTION (III)
Teacher
Strength and Difficulties Questionnaire (SDQ, Goodman et al, 1999)
WHY USE DIRECT OBSERVATIONS? (I)
Direct observations
Planning interventions
Evaluate outcomes
Examine research questions about the mechanisms involved in parent-child interaction
WHY USE DIRECT OBSERVATIONS? (II)
Provide window on real processes and outcomes of interest, eg. parent skills and strategies and child problem behaviour in the home.
Behaviours defined by researcher rather than parent (therefore less easily influenced by parent mood or expectations of intervention)
Self-reported outcomes problematic as parents tend to overestimate change following intervention.
WHY USE DIRECT OBSERVATIONS? (III)
Bias due to expectancy effects less likely (distressed parents poor at manipulating their child into behaving well during home observation)
Observational measures can be particularly sensitive to change in parent and child behaviour following intervention
Observed behaviour a better predictor of long-term hard outcome measures such as arrest rates and incarceration.
DIRECT OBSERVATIONS IN PRACTICE (I)
Choose a visit protocol and coding system which address the questions which need answering:
Do parents use more positive parenting strategies? Does negative child behaviour decrease? In there an increase in positive interaction between parent
and child Do measures assess treatment change or group
differences?
DIRECT OBSERVATIONS IN PRACTICE (II)
Different observational tasks designed to study parent-child interaction in home:
Mildly stressful events eg: mealtimes
Problem solving tasks and tidy up tasks
Tasks where mother busy and child has nothing to do
Exposure (eg. how many sessions)
Adherence
Quality
Reponsiveness (Therapy Attitude Inventory)
IMPLEMENTATION FIDELITY
PARENTING INTERVENTION: THEORY OF CHANGE
• Hire staff• Train staff to implement tested intervention
• Carry out 14 week parenting intervention•Ensuretreatmentintegrity
• Maximiseparent Participation
• Decreasedchild negativebehaviour• Increasedparenting skills and strategies
• Less delinquency• Less incarceration
PARENTING INTERVENTION: THEORY OF CHANGE
• Hire staff• Train staff to implement tested intervention
• Carry out 14 week parenting intervention•Ensuretreatmentintegrity
• Maximiseparent Participation
• Decreasedchild negativebehaviour• Increasedparenting skills and strategies
• Less delinquency• Less incarceration
Exercise 2 - ToC
• Construct a basic TOC of an intervention you’ve worked with/have interest in, and focus on key outcomes
• Identify potential measures or types of measures for implementation and outcomes that match your ToC
Youth Program Assessment Tools
Implementation Guide to various instruments for Measuring Youth Program Quality by the Forum for Youth Investment
http://forumfyi.org/content/measuring-youth-program-quality-guide-assessment-tools-2nd-edition
E.g., Youth Program Quality Assessment (PQA)
OutcomesCART: Compendium of various psychosocial measures (esp., for youth) http://cart.rmcdenver.com/index.cgi?screenid=seldomain&autoid=91867 CORC: http://www.corc.uk.net/index.php?contentkey=81
Select references
Church, A. T. (2010). Measurement Issues in Cross-cultural Research. In G. Walford, E. Tucker & M. Viswanathan (Eds.), The Sage Handbook of Measurement (pp. 152-175). London: SAGE. (validity across cultures)
Streiner, D. L. (2003). Starting at the Beginning: An Introduction to Coefficient Alpha and Internal Consistency. Journal of Personality Assessment, 80(1), 99-103.
Imas, L. G. M., & Rist, R. C. (2009). The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, D.C.: The World Bank. (fesp. for Th of Chg)
Weiss, C. H. (1995). Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families. In J. P. Connell, A. C. Kubisch, L. B. Schorr & C. H. Weiss (Eds.), New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: Aspen Institute.