Development and Evaluation of Mentoring Programs for Youth

Development and Evaluation of Mentoring Programs for

Youth:Strategies for Fostering

Success in the Next Generation

David L. DuBoisUniversity of Illinois at

Chicago

Colloquium Presentation, University of Girona,

Barcelona, Spain June 20, 2013

Overview Why youth mentoring as an intervention strategy? Why be systematic/rigorous about developing (and

improving) mentoring intervention strategies and evaluating their effectiveness?

What is “best practice” when developing mentoring intervention strategies?

What are the most rigorous and informative methods for evaluating youth mentoring intervention strategies?

Concluding thoughts

Why Youth Mentoring as an Intervention Strategy? Fundamental role of mentoring in human development

Evolutionary basis• Mechanism for intergenerational knowledge transfer (Wong, 2009)• Adaptive advantage of altruism toward other group members (Bowles, 2006)• Value of shared responsibilities for caregiving: allo-parenting (Diamond, 2012) and

age-mixing (Ellis et al., 2012) Development and learning powered by dyadic relationships with 4 key

features (Bronfenbrenner, 2005; see also Li & Julian, 2012)• Emotional attachment• Reciprocity• Progressively more complex activities• Gradual shifting of power in favor of developing person

Lack of Access to Mentoring in Contemporary Society ~1 in 5 youth in the U.S. estimated be lacking supportive relationships

with caring adults• Least access among youth experiencing economic disadvantage

Evidence suggests proportion of youth without any “go to” adult may be increasing

Source: DuBois (2013).

Empirical Support for Youth Mentoring Supports Capacity of mentoring programs to promote youth outcomes

demonstrated through several meta-analyses of rigorous (quasi-experimental and experimental) evaluation studies (DuBois et al., 2002, 2011; Tolan et al., 2008; Wheeler et al., 2010)

Evidence of:• Benefits in multiple areas: social, behavioral, emotional, academic

Psych

olog

ical

/Em

otio

nal

Probl

em/H

igh-

risk

Behav

ior

Socia

l Com

pete

nce

Acade

mic

/Edu

catio

nal

All Out

com

es0

0.2

0.4

Youth Mentoring (Pre-1999)Youth Mentoring (1999-2010)

Eff

ect

Siz

e

Source: DuBois et al. (2002, 2011)

Evidence of:• Benefits in multiple areas: social, behavioral, emotional, academic • Magnitude of impacts that are comparable to those achieved by other forms of

youth intervention, but may be more holistic/multi-faceted

0

0.1

0.2

0.3

Effe

ct S

ize

Youth Mentoring(1999-2010)

Other YouthInterventions

Source: DuBois et al. (2011)

-0.5

-0.3

-0.1

0.1

0.3

0.5

Pre-Test Post-Test

Yo

uth

Ou

tco

me

s

Non-Mentored YouthMentored Youth

Promotion

Prevention

Evidence of:• Benefits in multiple areas: social, behavioral, emotional, academic • Magnitude of impacts that are comparable to those achieved by other forms of youth

intervention, but may be more holistic/multi-faceted• Both preventive and promotive effects (DuBois et al., 2011)

Why be systematic/rigorous about developing/improving & evaluating mentoring intervention strategies?

Evidence that systematic intervention development strategies (e.g., theory- and research-basis) tend to result in more effective programs

Systematic development likely to produce more replicable and scalable programs

Limited magnitude of current program effects indicates need to search for stronger programs/strategies

Not safe to assume effectiveness of a strategy/program based on prior evaluations due to potential differences in implementation quality, targeted youth population, local conditions, etc.

Need to grow knowledge base for the field and fill in gaps (e.g., long-term effects)

Consider drawing on one or more existing program planning frameworks (e.g., PRECEDE-PROCEED; Greene & Kreuter, 1999)

What is “best practice” when developing mentoring intervention strategies?

1. Assess Needs & Capacities of Population

2. Assess Causes, Set Priorities & Objectives3. Design &

ImplementProgram

4. Evaluate ProgramReassess causes

Redesign

Source: Green & Kreuter (2005)

Assess needs, assets, and priorities of the community to be served and design strategies accordingly. Potential sources of information:

• Existing: Research, public data bases• New: Key informant interviews, focus groups, surveys

Potential questions:• Who has least access to mentoring?• Who stands to benefit most from mentoring?• What kinds of mentoring are most needed?• What types of mentoring are already occurring?• What assets (human, organizational, etc.) could be leveraged to

provide effective mentoring?

Search for “leads” on promising intervention strategies Review scholarly literature:

• Theoretical models• Observational research• Evidence-supported strategies – SPECIAL EMPHASIS

– Priority on high quality experimental and quasi-experimental evaluations– Draw on findings of systematic reviews of such evaluations, especially

meta-analyses

DuBois & Karcher (2014) Herrera, DuBois, & Grossman (2013)

• A Few Recent Examples (among many!)– Youth-nominated mentoring associated with improved educational and

employment outcomes for marginalized older youth (Schwartz et al., in press)– Beneficial effects of community-based mentoring on depressive symptomatology

among higher-risk youth (Herrera et al.. 2013)– Emphasis on similarity of interests when matching mentors and youth predictive

of greater program effectiveness (DuBois et al., 2011)– Support for mentor assuming teaching and advocacy roles also linked to

enhanced effectiveness (DuBois et al., 2011)

Empirically-Supported Best Practices

-0.1

0

0.1

0.2

0.3

0.4

0.5

0 1 2 3 4 5 6 7

Number of Practices

Siz

e o

f E

ffec

t o

n Y

ou

th

Small Effect

Medium Effect

(Based on DuBois et al., 2002)

Outreach to practice community• Program models and intervention strategies already in use• Seek input and ideas on directions for program/strategy design

Obtain input from stakeholders• Acceptability, usability, feasibility, utility• Youth, mentors, parents, staff

Develop and pilot strategies/program components Build program logic model / theory of change Iteratively refine through multiple cycles of implementation Collect and analyze process evaluation data to guide

process Carefully document and “manualize” all

components/activities

Example: GirlPOWER! (DuBois et al., 2008) Mentoring program for ethnic minority girls ages 10-13

• Designed as enhancement of Big Brothers Big Sisters (BBBS) community-based mentoring (CBM) program

• 12 monthly, 3-hour group sessions in 10-15 mentor-youth pairs Assessment

• Informed by:– Interviews with youth development professionals, mentors, parents, and youth– Review of research

• Findings: – Needs -- Lack of evidence-based mentoring programs for girls– Assets -- Girls group-based programming model already in use by local BBBS

agency– Priorities – Girls’ need for support particularly great during transition to

adolescence

Research and theory basis• BBBS CBM program model is empirically supported • Incorporated practices linked to greater effectiveness in meta-

analysis of mentoring program effectiveness (DuBois et al., 2002)– Structured activities for mentors and youth– Ongoing mentor training– Parent support and involvement

• Built in theory-/research-based

gender-sensitive program

elements– Relational focused group format– Counter limiting stereotypes of girls– Focus on salient concerns for girls

during transition to adolescence

Program design process• Informed by interviews/focus groups with stakeholders (youth,

mentors, parents, program staff)– Example: Girls interested in competition, but did not want to resemble school

• Iterative piloting and refinement of group sessions and other program components

– Example: Build in more time for informal socializing based on input of mentors and youth

• Ongoing manualization of all aspects of program– Examples: Mentor training, group sessions, out-of-session activity guides, parent

communications

Example: Step-It-Up-2-Thrive (research in progress) Mentoring framework for youth ages 10-16

• Designed as enhancement of BBBS CBM program• Based on “Road Map” for thriving: Sparks, Growth Mindset, Thriving

Indicators, Goal Management/Pursuit Skills• Resources available on-line (http://www.stepitup2thrive.org/)

Assessment • Informed by:

– Dialogue with BBBS national program staff– Review of research

• Findings: – Needs – Lack of evidence-based programming to promote youth thriving through

mentoring– Assets -- BBBS program model includes regularly occurring support contacts with

mentors = important mechanism for program delivery– Priorities – Growing interest in moving beyond “prevention” model to “promotion”

model focused on helping youth to reach their full potential

Research and theory basis• Recent research findings point toward contribution of abilities and

experiences in each of the four areas of the “Road Map” to outcomes important for youth (e.g., academic success, avoiding involvement in problem/delinquent behavior) Informed by:

– Associations for most part relatively modest, suggesting that combination of strengths in all areas may be needed for optimal outcomes

– Theoretically, effects of factors in one area also may be dependent on those in others (e.g., need growth mindset to effectively pursue sparks)

• Emphasis on teaching/guidance role for mentor supported by findings of recent meta-analysis of mentoring program effectiveness (DuBois et al., 2011)

Positive effects of intervention teaching incremental theory (growth mindset) to 7th graders (Blackwell, Trzesniewski, & Dweck, 2007)2

Program design process• Drew heavily on existing resource materials from Thrive Foundation

for Youth• Driven by national BBBS staff and input from staff of 3 “lead design”

agencies (BBBS affiliates)• Piloting of mentor training and group activities• Manualization of all aspects of program

What are the most rigorous and informative methods for evaluating youth

mentoring intervention strategies?*

Planning Decide on primary purposes of evaluation Ensure adequate resources and technical expertise are

available Consult an accepted set of standards for program evaluation

(e.g., Program Evaluation Standards of Joint Committee on Standards for Education Evaluation)

Obtain approval from research review board

*See DuBois (2014) for a more detailed treatment of issues involved with evaluation of mentoring programs.

Process Evaluation Major purposes

• Determine whether program happens as planned• Obtain information to guide further development/refinement of

program• Accountability to stakeholders• Support good program delivery

– Monitoring procedures likely to improve implementation – “what gets measured, gets done” – and linked to better mentoring program effects (DuBois et al., 2002)

Should assess both fidelity of implementation and dosage• Fidelity = extent to which program/strategy was implemented as

planned– Assess for both staff- and mentor-implemented activities– Key components to assess:

» Adherence: extent to which services/activities are carried out according to design

» Exposure: level of intended services/activities provided » Quality of Delivery: how well services/activities are implemented» Responsiveness: how participants experience services/activities » Differentiation: extent to which the services/activities differ from those

associated with other similar programs

• Dosage = levels of exposure/participation for individual participants– Assess with respect to services provided to both youth and their mentor

Important to gather both quantitative and qualitative data• Quantitative: May be particularly helpful for gauging whether

implementation benchmarks are met and whether these improve over multiple implementations

• Qualitative: May be particularly helpful for identifying ways to improve program content and for detecting unexpected experiences of participants

Outcome Evaluation Major purposes

• Determine whether program has intended effects on participants• Test program’s theory of change• Obtain information needed to support economic evaluation of

program’s “return on investment”

Keys• Need credible comparison group of non-participants to obtain

accurate and unbiased estimate of program effects– Random assignment to program vs. comparison group = “gold standard”

• Need sufficiently large sample to detect effects• Need reliable and valid measures of program outcomes

– Use program logic model / theory of change as guide– Ideally measure key outcomes through multiple methods – “triangulate”

• Incorporate follow-up assessment of outcomes beyond the period of program involvement

• Analyze data for both overall program effects and effects on different subgroups of youth

Example: GirlPOWER! Small-scale (n = 40) randomized controlled trial comparing

GirlPOWER! to standard BBBS community-based mentoring Process Evaluation

• Agency records, mentor reports (Adherence, Exposure, Differentiation, Dosage)

• Direct observation (Quality of delivery)• Youth, mentor, parent feedback surveys (Responsiveness)

Outcome Evaluation• Youth (and their mentors) randomly assigned to participate in

GirlPOWER! or BBBS standard programming• Pre-test and one-year (end of program)• Youth, parent, and mentor surveys• School and juvenile system records

Selected Findings• Process Evaluation

– Strengthening mentor-youth relationships» Mutual sharing

» “we got to learn more about each other” [mentor]» “we communicated more” [youth]

» “Seeding” discussions and activities» “… it will be easier for us to discuss peer pressure & drugs” [mentor]» “… it gave me some ideas on how we can stay fit together” [youth]

– Facilitating group support and cohesion» Supportive group experience for youth

» “she really looks forward to … hanging with the Littles” [mentor]» “[performing in talent show helped me] break out of my shell” [youth];

» Mentor sharing and support» “meeting similar Matches makes the entire BBBS experience better”

[mentor]» “… the Bigs find it helpful to share info, concerns, etc.” [mentor]

» Match connections » “Bigs can share ideas & make plans together …” [mentor]

– Program improvements guided by findings» POWERSessions

» make activities more interactive, less didactic; free up time for casual socializing

» greater utilization of mentors as teachers and role models» POWERBuilders

» Encourage matches to design their own activities; be liberal in “crediting” activities as addressing different program topics

» Resource materials» Reduce length and organize in program binder

– Improvements evident in youth and mentor experiences of the program sessions

Spring 2005 Fall 2005

Average Ratings for "FUN"

1

2

3

4

5

Little (Self) Big (Little) Big (Self)

Respondent

Ratin

g

Average Ratings for "HELPFUL"

1

2

3

4

5

Little (Self) Big (Little) Big (Self)

Respondent

Ratin

g

• Outcome Evaluation– Improvements in mentoring relationship quality

» More frequent mentor-youth contact» Stronger instrumental/goal-oriented dimension of relationships» Mentor seen by youth as more disapproving of problem behaviors (e.g.,

unhealthy eating, substance use) – Improved youth outcomes (greater improvement than youth in standard

community-based mentoring)» Health behavior» Health knowledge & beliefs» Exercise» Emotional well-being and resilience» Perceived coping efficacy» Domain-specific and global self-esteem» Anxiety/depression» Academic» Intrinsic motivation for learning» Educational aspirations

Example: Step-It-Up-2-Thrive Randomized controlled trial comparing BBBS community-

based mentoring with and without integration of Step-It-Up-2-Thrive framework

10 BBBS agencies, ~800 youth ages 10-16 with one or more risk factors for engaging in delinquent behavior

Process Evaluation• Staff reports, agency records (Adherence, Exposure, Differentiation)• Implementation checklist completed by staff (Dosage)• Staff self-assessment, participant feedback (Quality of delivery)• Youth, mentor, parent feedback surveys (Responsiveness)

Outcome Evaluation• Youth (and their mentors) randomly assigned to participate in BBBS

CBM with addition of “Thrive” integration or standard programming• Pre-test and 15 months• Youth, parent, and mentor surveys

Benefits of “real time” analysis of process evaluation data• Unexpectedly low rates of participation in group activities for matches

(mentor-youth pairs) – Adjustment:

» Development of alternative formats» Activity guides for mentor-youth pairs to work through on their own» On-line format

• Challenges with having matches move through different stages of intervention activity in planned sequence

– Adjustment: » Allow flexibility in sequencing where fits with needs and interests of youth

(and mentor)

• Lack of case manager comfort with supporting intervention content

– Adjustments: » Increase their role as “trainers” of mentors» Provide greater opportunities for technical assistance from

national staff

Concluding Thoughts Promise of mentoring is arguably as well-established

as any form of youth intervention It would appear, however, that only a surprisingly small

portion of this potential has been realized More systematic and rigorous approaches to development

and evaluation of mentoring intervention strategies could help to narrow this gap

Important not to overlook need for complementary focus on improving extent & equity of youth access to effective forms of mentoring

Impact + Reach = Healthier Future Generations and Greater Global Prosperity

Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246-263.Bowles, J. (2006). Group competition, reproductive leveling, and the evolution of human altruism. Science, 314(5805), 1569–1572.Bronfenbrenner, U. (2005). Making human beings human: Bioecological perspectives on human development. Thousand Oaks, CA: Sage.Diamond, J. (2012). The world until yesterday. New York: Viking Press.DuBois, D. L. (2013). Trends and correlates of youth reports of not having a parent or other adult available to provide support with personal problems. Manuscript in prepartionDuBois, D. L. (2014). Program evaluation. In D. L. DuBois & M. J. Karcher (Eds.), Handbook of youth mentoring (2nd ed., pp. 481-498). Thousand Oaks, CA: Sage.DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157–197.DuBois, D. L., & Karcher, M. J. (Eds.). (2014). Handbook of youth mentoring (2nd ed.). Thousand Oaks, CA: Sage.DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57–91.

References

DuBois, D. L., Silverthorn, N., Pryce, J., Reeves, E., Sanchez, B., Silva, A., . . . Takehara, J. (2008). Mentorship: The GirlPOWER! program. In C. LeCroy, Craig Winston, & J. E. Mann (Eds.), Handbook of prevention and intervention programs for adolescent girls (pp. 326–336). Hoboken, NJ: Wiley.Ellis, B. J., Del Giudice, M., Dishion, T. J., Figueredo, A. J., Gray, P., Griskevicius, V., . . . Wilson, D. S. (2012). The evolutionary basis of risky adolescent behavior: Implications for science, policy, and practice. Developmental Psychology, 48, 598-623.Green, L.W.,&Kreuter, M.W. (2005). Health promotion planning:An educational and ecologicalapproach (4th ed.). New York: McGraw-Hill.Herrera, C., DuBois, D. L., & Grossman, J. B. (2013). The role of risk: Mentoring experiences and outcomes for youth with varying risk profiles. New York: A Public/Private Ventures project published by MDRC.Li, J., & Julian, M. (2012). Developmental relationship as the active ingredient: A unifying working hypothesis of ―what works‖ across intervention settings. American Journal of Orthopsychiatry, 82, 157-166.Schwartz, S., Rhodes, J., Spencer, R., & Grossman, J. (in press). American Journal of Community Psychology. Youth initiated mentoring: Investigating a new approach to working with vulnerable adolescents. American Journal of Community Psychology. Tolan, P., Henry, D., Schoeny, M., & Bass, A. (2008). Mentoring interventions to affect juvenile delinquency and associated problems. Campbell Systematic Reviews, 16. Retrieved from http://www.campbellcollaboration.org/lib/download/238/Wheeler, M. E., Keller, T. E., & DuBois, D. L. (2010). Review of three recent randomized trials of school-based mentoring: Making sense of mixed findings. Social Policy Report, 24(3).Wong, K. (2009). Twilight of the Neandertals. Scientific American, 301(2), 32–37.