WRECMay 29, 2014Washington, D.C.

Learning “What Works” in Career Pathways Programming: The ISIS Evaluation

. David JudkinsAbt Associates, Inc

Abstract

2

Introductory review of new methodology for the study of the mediating pathways in the context of randomized experiments of social interventions– Pathways may contain both exposure to program

components and early outcomes of that exposure– Other talks in this session talk about randomizing to

program components– This talk is about how to analyze when randomization

of components is not selected as the evaluation methodology

Plans to apply this new methodology to ISIS

What is ISIS? Random assignment evaluation of nine “career pathways”

programs– Impact, implementation, cost-benefit studies

Evaluation funded by ACF; Abt Associates heading research team– Additional funding for program enhancements/scale up from

Open Society Foundations, Joyce Foundation, Kresge Foundation, Meadows Foundation, OFA/ACF (3 HPOG sites)

RA still underway; early impact results in 2-3 years Programs vary, but all promote access to and completion of

post-secondary education, targeting low-income, low-skilled adults and youth

Conducting nine separate studies– Overlap in research questions, measurement, analysis plans

3

Core Ideas of Career Pathways Programs

Address the wide range of skill and other needs of economically disadvantaged participants

Key “inputs” or elements: Assessment (academic and non-academic); basic and vocational skills training; supports; employment connections

Create manageable, well-articulated training steps Provide credentials valued in high demand

occupations/sectors Build effective partnerships

– Community colleges, employers, CBOs, WIBs

4

Challenge: Getting inside the “Black Box”

How to determine more effective program components?– Counseling– Tutoring– Financial assistance– Internships

How to determine intermediate outcomes on vital causal paths?– Self-efficacy– Strength of social network– Basic numeracy and literacy skills– Stress management skills

5

Benefits

Effective components are built into future interventions Ineffective components are dropped New components are invented to impact vital early

outcomes even more strongly Future evaluation findings can be obtained more quickly

if good early indicators are available

6

Familiar but Inadequate

Baron and Kenny (1986) mediation triangle

The indirect effect of treatment on Y mediated by M estimated as ab.

The direct effect of treatment on Y is d.

7

1 1

2 2

i i i

i i i i

M aT e

Y dT bM e

Why Do We Need Better Methodology?

Baron and Kenny assume independence of errors in two equations.

This will be violated if there are any common causes of the two errors such as:– Measured baseline characteristics – Unmeasured baseline characteristics– Other measured mediators– Other unmeasured mediators

Also, Baron and Kenny does not generalize well to categorical outcomes

8

Pearl’s Method

Judea Pearl has developed a much more powerful and general method.

However, he developed it in a graphical framework unnatural to statisticians in the Neyman-Rubin tradition.

Recently, Kosuke Imai and co-authors have recast it in the language and traditions of Neyman-Rubin.

Dramatically increased popularity Still very difficult literature to penetrate Sketch some features today We plan to use on ISIS

9

Pearl’s Method Capabilities

10

Challenge Pearl can Handle

Confounding due to measured moderators

Confounding due to other measured mediators

Confounding due to unmeasured moderators

Confounding due to unmeasured other mediators

Nonnormal outcomes such as binary outcomes

ISIS Nomenclature

Blend of nomenclatures proposed by Pearl and by Imai and coworkers.

Key is the concept of forcing an environmental stimulus while “blocking” one type of response to the stimulus

Three potential outcomes per personY0: person randomly assigned to controlY1: person randomly assigned to treatment and no changes are blockedY2: person randomly assigned to treatment but change in mediator M is “blocked”

11

Three Potential Outcomes

12

Example

Y0 = person’s degree attainment at 36 months if assigned to control

Y1 = person’s degree attainment at 36 months if assigned to treatment and no “natural” changes are “blocked”

Y2 = person’s degree attainment at 36 months if assigned to treatment but somehow we blocked change to self-confidence

13

Potential Outcomes Framework for Mediation

Total effect = average of Y1-Y0 Indirect effect mediated by M = average of Y1-Y2 Direct effect = average of Y2-Y0 Then total = indirect + direct Continuing example:

– Y1-Y0 = total effect of treatment on degree attainment– Y1-Y2 = indirect effect of treatment on degree attainment

via boosted self-confidence– Y2-Y0 = direct effect of treatment on degree attainment

Note: no need to reference linear models to define the estimands

14

Mediating Pathways

In ISIS, we are interested in multiple mediating pathways– Trying to decide which are the vital pathways for successor

programs to emulate Multiple mediation can take two forms:

– Serial (T causes M1 causes M2 causes Y)– Parallel (T causes both M1 and M2, both of which jointly

cause Y) Pearl’s framework flexible enough to handle

simultaneous parallel and serial mediation

15

Service Exposure and Early Outcomes

Both are mediators Let D1 and D2 be exposure levels to two program

components Let W1 and W2 be two early outcomes Treatment causes D1 and D2 to change, which causes W1 and

W2 to change (in addition to direct effects of T on them) All of which lead to changes in Y

16

Example Graph for Parallel and Serial Mediation

17

T

D1

D2

W1

W2

YX

Estimation Process

Build a series of models for every mediator and outcome in the system in terms of causally prior variables

Draw simulated values from system for each pathway of interest– What happens if some services are blocked and

some early outcomes are blocked Pathways with highest simulated Y values are the

most promising for future program developers to consider

18

Assumptions

Strong assumptions required Considerable debate about best way to express

them Oversimplified version

– Variation in M within each level of Y– No unmeasured prime joint causes of M and Y– This includes post-randomization latent outcomes

19

Example Violation

M = Self-confidence Y = Degree attainment Academic skill at unmeasured intermediate point

mediates treatment effects on both self-confidence and degree attainment

Easy to come to the mistaken conclusion that shortcut methods work that build self-confidence through means other than skill improvement

Need to measure skill to prevent this incorrect mediational inference

20

Challenges

Curse of dimensionality Need to identify graphs to be tested that correspond

to program-specific theories of change Also need to believe that we have measured the

necessary set of variables to unconfound the mediator(s) and outcome(s) in the graph of interest

21

Additional Information

David JudkinsPrincipal ScientistAbt AssociatesDavid_judkins@abtassoc.com

Websitewww.projectisis.org

Brendan KellyFederal Co-Project Officer, ISISACF/OPREBrendan.Kelly@ACF.hhs.gov

Molly IrwinFederal Co-Project Officer, ISISACF/OPREMolly.Irwin@ACF.hhs.gov