Our Evaluation Evolution! Our Evaluation Evolution! Santa Fe Mountain Center’s Santa Fe Mountain Center’s Journey to Journey to Evidence-Based Evidence-Based Research and Evaluation Research and Evaluation 2008 REAP Symposium 2008 REAP Symposium Santa Fe, NM Santa Fe, NM Sky Gray, MS, CTRS Sky Gray, MS, CTRS Jenn Jevertson, MS Jenn Jevertson, MS Michael Gass, PhD Michael Gass, PhD Santa Fe Mountain Center Santa Fe Mountain Center Santa Fe Mountain Center Santa Fe Mountain Center University of University of New Hampshire New Hampshire [email protected][email protected][email protected][email protected][email protected][email protected]
Sky Gray, MS, CTRSSky Gray, MS, CTRS Jenn Jevertson, MSJenn Jevertson, MS Michael Gass, PhDMichael Gass, PhDSanta Fe Mountain CenterSanta Fe Mountain Center Santa Fe Mountain CenterSanta Fe Mountain Center University of New University of New [email protected]@santafemc.org [email protected]@santafemc.org [email protected][email protected]
I. History of SFMC Evaluation
II. Anti-Bullying Initiative Evaluation Design
III. Results from ABI Research and Evaluation
IV. Q&A
Overview
SFMC Evaluation History• 1979
– Since inception, Santa Fe Mountain Center has engaged in various research and evaluation.
• 2005– Began learning about new trends in evidenced based research in
New Mexico, at AEE and beyond.
• 2005-2006– Created new evaluation tool to measure outcomes with
assistance from Dr. Susan Carter
• 2006– Created the Experiential Adventure Based Resiliency Model and
published our new Program Manual with assistance from Project Adventure
Why Bother?
I have a organization and
program to run!!!
External Motivation• Following suit with New Mexico's Behavioral
Health Redesign• Following local and national trends in the
evidence-based evaluation and research world
• Satisfying the requirements of our largest funding source of the SFMC
• Contributing to the body of knowledge in the Experiential Adventure Based profession
Internal Motivation– Evaluation tool outdated and ineffective
(therefore not embraced by staff) – Recognition of the value and importance of
good evaluation and research– Desire to contribute to the field– Desire to create stronger program
outcomes– FUNDING and SURVIVAL!!!
Learnings• This process was at times intimidating for the
organization, and there was a need for staff training at all levels
• We consciously entered into new territory and had to ask for help from academic institutions (UNH) and other related experts
• Deepened our understanding of Evaluation and Research and other related principles
• Joining forces with the Symposium and now with CORE has pushed us further into our learning curve.
• How to conduct quasi-experimental evaluation designs within our program… Anti-Bullying Initiative
Anti-Bullying Initiative
An Experiential Adventure Based Approach to Increasing Resiliency and
Decreasing Bullying Behavior
Anti-Bullying Initiative
•Intended Outcomes1. Classrooms are safer and more productive learning
environments.
2. Students possess skills and strategies to deal with being the target of bullying behavior.
3. Students possess skills, strategies, and confidence to intervene appropriately in bullying situations.
Anti-Bullying Initiative• Program Design
– One identified at-risk elementary school• Low-income, predominately Hispanic, high percentage of
recent immigrant children
– Entire 5th grade level • 5 classrooms; total of 82 children• 2 bilingual classrooms
– Dosage throughout school year• Intro/Closing Session• 8 two-hour experiential sessions @ School• 3 full-day experiential adventure programs @ Santa Fe
Mountain Center ropes course
Evaluation ToolA. Demographics
• anonymous
B. Internal Assets• adapted from the California Healthy Kids Survey: Resilience Module (WestEd). • goals and aspirations, problem-solving, empathy, and self-efficacy. • corresponds to the Search Institute’s 40 Developmental Assets
C. Attitudes and beliefs about violence and bullying• adapted from the Safe Haven Survey (WestEd). • assess students’ beliefs and attitudes about violent and bullying behavior in several
categories: communication, school environment, and conflict management.
D. Violence and bullying experienced by respondent• adapted from Bullyproofing Your Schools program. • assesses students’ direct of experience of violent and bullying behavior in the past
month in the classroom.
E. Violence and bullying witnessed by respondent• adapted from Bullyproofing Your Schools program. • assesses students’ observation of violent and bullying behaviors experienced by
other students in the past month in classroom.
Developed by Dr. Susan Carter and Jenn Jevertson, 2006
Evaluation DesignYear 1•Pre/post surveys (356)
– 72 Tx Group– 87 Comparison Group
•Client Satisfaction (post only)•Student Essays•Teacher post surveys (post only)
Year 2•Follow-up pre/post surveys in Fall•Focus Groups (teachers/students) in Fall
A Step Farther…Research
• Collaboration with UNH – Jesse Beightol, Master’s student– Dr. Michael Gass
Resilience
“a class of phenomena characterized by patterns of positive adaptation in the context of significant adversity or risk”
(Masten & Reed, 2002, p.75)
Why Resilience?
School Performance• Low Assets = Low API Scores
(Hanson & Austin, 2003)
• Resilient Students perceive a more positive learning environment and have fewer challenges with schoolwork
(Padron et al., 1999)
Bullying • High resilience = Less risk taking behavior• Low resilience = More risk taking behavior and twice
as likely to be a victim of bullying
(Donnon & Hammond, 2007)
RYDM Framework
Current Research
1. Does participation enhance resilience compared to a control group?
2. Is there a sustained effect on levels of resilience at 4-months post-treatment?
3. If so, what aspects of the program may have contributed to this resilience enhancement?
4. Do levels of resilience vary according to gender or ethnicity?
Research Design
QUANData
Collect
QUANResults
QUALData
Collect
QUALResults
QUANData
Analysis
QUALData
Analysis
InterpretationQUAN + QUAL
Compare and
Contrast
RYDM
Frame-work
Modified Triangulation Design: Convergence Model
(Adapted from Creswell and Plano Clark, 2007, p.63)
Demographics
• Treatment Group– 52 Students
• 26 Male• 25 Female
• Control Group– 55 Students
• 29 Male• 25 Female
Quantitative Measure
• The Anti-Bullying Initiative Survey
(Carter & Jevertson, 2006)
– Goals and Aspirations
– Problem Solving
– Empathy
– Self Efficacy
Focus Groups, Interviews, and Program Observation
• Focus groups with students and teachers
• Semi-structured interviews with SFMC facilitators and school principal
• One day of program observation at the Santa Fe Mountain Center
Goals and Aspirations
Time321
Mea
ns
3.80
3.60
3.40
ControlTreatment
Goals and Aspirations by GroupSignificant increase for the treatment group from pre-test to follow-up (p = .010), with a small to medium effect size (r = .25).
Significant increase from post-test to follow-up, (p = .031), with a small to medium effect size (r = .21).
Goals and Aspirations
• “One of my goals is to go to college and become a doctor…because, like, the Mountain Center taught us…to help people out.”
• “My goal was to go to college and be a nurse because the Mountain Center taught me to help people.” When I asked how the Mountain Center taught her that she wanted to help people, she said, “To be like, have more responsibility and be more friendly to the other person.”
Self Efficacy
Time321
Mea
n
3.80
3.60
3.40
ControlTreatment
Self Efficacy by Group Significant increase for the treatment group from pre-test to post-test (p = .008), with a small to medium effect size (r = .26).
Self Efficacy
• “Yeah my uncle really picks on me and I don’t do much about it” to a few months later “Hey I used this stuff with my uncle that picked on me and it worked.”
• “Before they came…I saw people getting bullied but I didn’t do anything, like I just walked away. Cause I thought like if I tried to do something they would start bullying me. So I didn’t do anything…they told me about all this stuff, like the HAHASO and all that, um, I started helping those people out.”
• “They gave you the option of going down, but they also said ‘Are you sure you want to try another step’ and then when you did you realized you thought different, like you didn’t want to go down yet, and you ended up going up to the top.”
Average Resilience by Gender
Time321
Mea
n
3.80
3.60
3.40
3.20
FemaleMale
Treatment Group Average Resilience by Gender
Treatment group females higher Average Resilience scores than males 4 months after treatment.
Goals by Gender
Time321
Mea
n
4.00
3.80
3.60
3.40
FemaleMale
Treatment Group Goals and Aspirations by Gender
Significant increase in treatment group females from pre-test to follow-up (p = .009), with a medium effect size (r = .37).
Significant increase from post-test to follow-up (p = .017), with a medium effect size (r = .34).
Female Goals by Group
Time321
Mea
n
4.00
3.80
3.60
3.40
ControlTreatment
Female Goals and Aspirations by Treatment or Control Significant difference between females in Goals and Aspirations at follow-up.
Self Efficacy by Gender
Time321
Me
an
s
3.80
3.60
3.40
FemaleMale
Treatment Group Self Efficacy by Gender
Significant difference in treatment group at follow-up (p=.028), with a medium effect size (r = .31).
Significant increase for females in the treatment group from pre-test to post-test (p = .025), with a small to medium effect size (r = .32).
Courage to Implement Tools in Real Life Situations
Increased Responsibility
Enhancing External Assets at School
Experience and Development
Transfer of Lessons
Continuation TheResilience
Cycle
Conceptual Model from Qualitative Data
Ideas to Consider
• Reliability and Validity of Survey• Subscales, Age Appropriateness, Ceiling
Effects
• Gender Differences
• Complexity of a Topic like Resilience
• Importance of a Holistic Program
Select ReferencesCarter, S., & Jevertson, J. (2006). Anti-Bullying Initiative Survey.
Constantine, N., & Benard, B. (2001). California Healthy Kids Survey: Resilience Assessment Module Technical Report. Berkeley, CA: Public Health Institute.
Creswell, J. W., & Plano Clark, V. L. (2007). Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage.
Donnon, T., & Hammond, W. (2007). Understanding the Relationship Between Resilience and Bullying in Adolescence: An Assessment of Youth Resiliency from Five Urban Junior High Schools. Child and Adolescent Psychiatric Clinics of North America, 16, 449-471.
Masten, A., & Reed, M.G.. (2002). Resilience in Development. In Snyder, C. R. and Lopez, S. Handbook of Positive Psychology. Oxford University Press: New York, 74-88.
Technical Report: 5th Grade. (2006). WestEd and California Safe and Healthy Kids Program Office.
