Our Evaluation Our Evaluation Evolution! Evolution!

Santa Fe Mountain Santa Fe Mountain Center’sCenter’s

Journey toJourney to Evidence-Based Evidence-Based Research and EvaluationResearch and Evaluation

2008 REAP Symposium2008 REAP SymposiumSanta Fe, NMSanta Fe, NM

Sky Gray, MS, CTRSSky Gray, MS, CTRS Jenn Jevertson, MSJenn Jevertson, MS Michael Gass, PhDMichael Gass, PhDSanta Fe Mountain CenterSanta Fe Mountain Center Santa Fe Mountain CenterSanta Fe Mountain Center University of New University of New HampshireHampshiresky@santafemc.orgsky@santafemc.org jenn@santafemc.orgjenn@santafemc.org mgass@cisunix.unh.edu mgass@cisunix.unh.edu

I. History of SFMC Evaluation

II. Anti-Bullying Initiative Evaluation Design

III. Results from ABI Research and Evaluation

IV. Q&A

Overview

SFMC Evaluation History• 1979

– Since inception, Santa Fe Mountain Center has engaged in various research and evaluation.

• 2005– Began learning about new trends in evidenced based research in

New Mexico, at AEE and beyond.

• 2005-2006– Created new evaluation tool to measure outcomes with

assistance from Dr. Susan Carter

• 2006– Created the Experiential Adventure Based Resiliency Model and

published our new Program Manual with assistance from Project Adventure

Why Bother?

I have a organization and

program to run!!!

External Motivation• Following suit with New Mexico's Behavioral

Health Redesign• Following local and national trends in the

evidence-based evaluation and research world

• Satisfying the requirements of our largest funding source of the SFMC

• Contributing to the body of knowledge in the Experiential Adventure Based profession

Internal Motivation– Evaluation tool outdated and ineffective

(therefore not embraced by staff) – Recognition of the value and importance of

good evaluation and research– Desire to contribute to the field– Desire to create stronger program

outcomes– FUNDING and SURVIVAL!!!

Learnings• This process was at times intimidating for the

organization, and there was a need for staff training at all levels

• We consciously entered into new territory and had to ask for help from academic institutions (UNH) and other related experts

• Deepened our understanding of Evaluation and Research and other related principles

• Joining forces with the Symposium and now with CORE has pushed us further into our learning curve.

• How to conduct quasi-experimental evaluation designs within our program… Anti-Bullying Initiative

Anti-Bullying Initiative

An Experiential Adventure Based Approach to Increasing Resiliency and

Decreasing Bullying Behavior

Anti-Bullying Initiative

•Intended Outcomes1. Classrooms are safer and more productive learning

environments.

2. Students possess skills and strategies to deal with being the target of bullying behavior.

3. Students possess skills, strategies, and confidence to intervene appropriately in bullying situations.

Anti-Bullying Initiative• Program Design

– One identified at-risk elementary school• Low-income, predominately Hispanic, high percentage of

recent immigrant children

– Entire 5th grade level • 5 classrooms; total of 82 children• 2 bilingual classrooms

– Dosage throughout school year• Intro/Closing Session• 8 two-hour experiential sessions @ School• 3 full-day experiential adventure programs @ Santa Fe

Mountain Center ropes course

Evaluation ToolA. Demographics

• anonymous

B. Internal Assets• adapted from the California Healthy Kids Survey: Resilience Module (WestEd). • goals and aspirations, problem-solving, empathy, and self-efficacy. • corresponds to the Search Institute’s 40 Developmental Assets

C. Attitudes and beliefs about violence and bullying• adapted from the Safe Haven Survey (WestEd). • assess students’ beliefs and attitudes about violent and bullying behavior in several

categories: communication, school environment, and conflict management.

D. Violence and bullying experienced by respondent• adapted from Bullyproofing Your Schools program. • assesses students’ direct of experience of violent and bullying behavior in the past

month in the classroom.

E. Violence and bullying witnessed by respondent• adapted from Bullyproofing Your Schools program. • assesses students’ observation of violent and bullying behaviors experienced by

other students in the past month in classroom.

Developed by Dr. Susan Carter and Jenn Jevertson, 2006

Evaluation DesignYear 1•Pre/post surveys (356)

– 72 Tx Group– 87 Comparison Group

•Client Satisfaction (post only)•Student Essays•Teacher post surveys (post only)

Year 2•Follow-up pre/post surveys in Fall•Focus Groups (teachers/students) in Fall

A Step Farther…Research

• Collaboration with UNH – Jesse Beightol, Master’s student– Dr. Michael Gass

Resilience

“a class of phenomena characterized by patterns of positive adaptation in the context of significant adversity or risk”

(Masten & Reed, 2002, p.75)

Why Resilience?

School Performance• Low Assets = Low API Scores

(Hanson & Austin, 2003)

• Resilient Students perceive a more positive learning environment and have fewer challenges with schoolwork

(Padron et al., 1999)

Bullying • High resilience = Less risk taking behavior• Low resilience = More risk taking behavior and twice

as likely to be a victim of bullying

(Donnon & Hammond, 2007)

RYDM Framework

Current Research

1. Does participation enhance resilience compared to a control group?

2. Is there a sustained effect on levels of resilience at 4-months post-treatment?

3. If so, what aspects of the program may have contributed to this resilience enhancement?

4. Do levels of resilience vary according to gender or ethnicity?

Research Design

QUANData

Collect

QUANResults

QUALData

Collect

QUALResults

QUANData

Analysis

QUALData

Analysis

InterpretationQUAN + QUAL

Compare and

Contrast

RYDM

Frame-work

Modified Triangulation Design: Convergence Model

(Adapted from Creswell and Plano Clark, 2007, p.63)

Demographics

• Treatment Group– 52 Students

• 26 Male• 25 Female

• Control Group– 55 Students

• 29 Male• 25 Female

Quantitative Measure

• The Anti-Bullying Initiative Survey

(Carter & Jevertson, 2006)

– Goals and Aspirations

– Problem Solving

– Empathy

– Self Efficacy

Focus Groups, Interviews, and Program Observation

• Focus groups with students and teachers

• Semi-structured interviews with SFMC facilitators and school principal

• One day of program observation at the Santa Fe Mountain Center

Goals and Aspirations

Time321

Mea

ns

3.80

3.60

3.40

ControlTreatment

Goals and Aspirations by GroupSignificant increase for the treatment group from pre-test to follow-up (p = .010), with a small to medium effect size (r = .25).

Significant increase from post-test to follow-up, (p = .031), with a small to medium effect size (r = .21).

Goals and Aspirations

• “One of my goals is to go to college and become a doctor…because, like, the Mountain Center taught us…to help people out.”

• “My goal was to go to college and be a nurse because the Mountain Center taught me to help people.” When I asked how the Mountain Center taught her that she wanted to help people, she said, “To be like, have more responsibility and be more friendly to the other person.”

Self Efficacy

Time321

Mea

n

3.80

3.60

3.40

ControlTreatment

Self Efficacy by Group Significant increase for the treatment group from pre-test to post-test (p = .008), with a small to medium effect size (r = .26).

Self Efficacy

• “Yeah my uncle really picks on me and I don’t do much about it” to a few months later “Hey I used this stuff with my uncle that picked on me and it worked.”

• “Before they came…I saw people getting bullied but I didn’t do anything, like I just walked away. Cause I thought like if I tried to do something they would start bullying me. So I didn’t do anything…they told me about all this stuff, like the HAHASO and all that, um, I started helping those people out.”

• “They gave you the option of going down, but they also said ‘Are you sure you want to try another step’ and then when you did you realized you thought different, like you didn’t want to go down yet, and you ended up going up to the top.”

Average Resilience by Gender

Time321

Mea

n

3.80

3.60

3.40

3.20

FemaleMale

Treatment Group Average Resilience by Gender

Treatment group females higher Average Resilience scores than males 4 months after treatment.

Goals by Gender

Time321

Mea

n

4.00

3.80

3.60

3.40

FemaleMale

Treatment Group Goals and Aspirations by Gender

Significant increase in treatment group females from pre-test to follow-up (p = .009), with a medium effect size (r = .37).

Significant increase from post-test to follow-up (p = .017), with a medium effect size (r = .34).

Female Goals by Group

Time321

Mea

n

4.00

3.80

3.60

3.40

ControlTreatment

Female Goals and Aspirations by Treatment or Control Significant difference between females in Goals and Aspirations at follow-up.

Self Efficacy by Gender

Time321

Me

an

s

3.80

3.60

3.40

FemaleMale

Treatment Group Self Efficacy by Gender

Significant difference in treatment group at follow-up (p=.028), with a medium effect size (r = .31).

Significant increase for females in the treatment group from pre-test to post-test (p = .025), with a small to medium effect size (r = .32).

Comparison to AggregateTreatment Control

Aggregate

AverageHigh 94.2 90.9 65Medium 5.8 9.1 33Low 0 0 2

Goals and AspirationsHigh 94.2 92.7 83Medium 5.8 7.27 17Low 0 0 1

Problem SolvingHigh 80.8 83.6 38Medium 19.2 12.7 55Low 0 3.6 7

Comparison to Aggregate Treatment Comparison Aggregate

EmpathyHigh 75 70.9 44Medium 21.2 23.6 50Low 3.8 5.5 6

Self EfficacyHigh 98.1 92.7 NAMedium 1.9 7.27 NALow 0 0 NA

External Assets- Physical and Emotional Safety

-Unique and Consistent Definition of Resilience

Enhancing External

Internal Assets

Improved Outcomes

Tools and Common Language

Successful Experiences in a Variety of Activities

Consistency and Continuity

Courage to Implement Tools in Real Life Situations

Increased Responsibility

Enhancing External Assets at School

Experience and Development

Transfer of Lessons

Continuation TheResilience

Cycle

Conceptual Model from Qualitative Data

Ideas to Consider

• Reliability and Validity of Survey• Subscales, Age Appropriateness, Ceiling

Effects

• Gender Differences

• Complexity of a Topic like Resilience

• Importance of a Holistic Program

Select ReferencesCarter, S., & Jevertson, J. (2006). Anti-Bullying Initiative Survey.

Constantine, N., & Benard, B. (2001). California Healthy Kids Survey: Resilience Assessment Module Technical Report. Berkeley, CA: Public Health Institute.

Creswell, J. W., & Plano Clark, V. L. (2007). Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage.

Donnon, T., & Hammond, W. (2007). Understanding the Relationship Between Resilience and Bullying in Adolescence: An Assessment of Youth Resiliency from Five Urban Junior High Schools. Child and Adolescent Psychiatric Clinics of North America, 16, 449-471.

Masten, A., & Reed, M.G.. (2002). Resilience in Development. In Snyder, C. R. and Lopez, S. Handbook of Positive Psychology. Oxford University Press: New York, 74-88.

Technical Report: 5th Grade. (2006). WestEd and California Safe and Healthy Kids Program Office.