Meet the Evaluation Team: RAND Corporation
CalMHSA Statewide Program Partners
Statewide Coordination Workgroup Meeting
A9604-AB-2 12/2011
Overview
• Background on team and state-wide evaluation
• Approach to evaluation
• Current state of research in three initiative areas
• Evaluation team and program partners: Relationship building
A9604-AB-3 12/2011
PEI Statewide Evaluation Purpose
To evaluate to what extent and how the strategies of PEI Statewide Projects are effective in:
• Preventing Suicides
• Improving Student Mental Health
• Reducing Mental Health Stigma and Discrimination
A9604-AB-4 12/2011
Objectives of the Statewide Evaluation
Key objectives of PEI Statewide Evaluation:
• Establish baselines and community indicators
• Conduct thorough program evaluations
• Identify innovative programs for replication
• Promote continuous quality improvement efforts
A9604-AB-5 12/2011
PEI Statewide Evaluation Process
• CalMHSA developed a Request for Qualifications
• Statements of Qualifications reviewed by a team of experts that selected the RAND team
• CalMHSA and RAND team negotiated a Statement of Work
• RAND team develops an Evaluation Strategic Plan for discussion with CalMHSA and the SEE
• Statewide evaluation implementation and reporting
A9604-AB-6 12/2011
Components of Statewide Evaluation
• Goal-based, process-based, and outcomes-based evaluations
• Conducted at three levels: – Individual programs – Each of three initiatives – Overall CalMHSA effort statewide
• Coordination and leveraging across PEI initiatives and programs
A9604-AB-7 12/2011
Who Is on the RAND Team?
• RAND Corporation – independent nonprofit public policy research organization based in Santa Monica, CA
• SRI International Policy Division – independent, nonprofit research institute based in Menlo Park, CA
• Field Research Corporation – marketing and public opinion research firm based in San Francisco, CA
• UCLA/RAND/USC NIMH – Partnered Research Center for Quality Care
A9604-AB-8 12/2011
Key Players at RAND
• Audrey Burnam – Project Leader (Admin and Design)
• Sandy Berry – Co-Project Leader (Data Acquisition)
• Rajeev Ramchand – Suicide Prevention
• Becky Collins – Stigma and Discrimination
• Bradley Stein – School Based Mental Health
• Patricia Ebener – Technical Assistance
• Mary Vaiana – Communications and Dissemination
A9604-AB-9 12/2011
What Is RAND?
• Chartered in 1948 as a nonprofit organization To further and promote scientific, educational, and charitable purposes, all for . . . public welfare and security . . .
• Mission To help improve policy and decision making through research and analysis
• Core values Quality and objectivity
A9604-AB-10 12/2011
Civil Justice
Education Substance Abuse
U.S. National Security
Energy and
Environment
Health and Health Care
Transportation and
Infrastructure
International Affairs
Science and
Technology
Population and
Aging
Terrorism and
Homeland Security
Children and
Adolescents
Mental Health
• Providing practical solutions to complex problems
• Supporting evaluation and implementation
• Enhancing policy debates
• Training future policy analysts
• Improving knowledge and analytic methods
RAND’s Principal Research Areas
A9604-AB-11 12/2011
Our Research Is Typically Characterized by . . .
An analytic approach that is: • Integrative, collaborative,
and multidisciplinary • Empirical, with technical
depth and methodological rigor
• Innovative, but informed by past findings
• Buttressed by demanding standards of quality and objectivity
RAND strives to build long-term relationships with its clients
Issues that involve: • Competing objectives
and perspectives • Intersection of public/
private interests • "Messy" data, major
uncertainties • Implications for the
future
A9604-AB-12 12/2011
Toward a Culture of Consequences Performance-Based Accountability Systems for Public Services Brian M. Stecher, Frank Camm, Cheryl L. Damberg, Laura S. Hamilton, Kathleen J. Mullen, Christopher Nelson, Paul Sorensen, Martin Wachs, Allison Yoh, Gail L. Zellman
Recent RAND Research Addresses Many Critical Societal Issues
Invisible Wounds of War Psychological and Cognitive Injuries,
Their Consequences, and Services to Assist Recovery TERRI TANIELIAN AND LISA H. JAYCOX, EDITORS
Oil Shale Development In the United States
James T. Bartis, Tom LaTourrette, Lloyd Dixon, D.J. Peterson, Gary Cecchine
Prospects and Policy Issues
Early Childhood
Interventions
Lynn A. Karoly M. Rebecca Kilburn
Jill S. Cannon
Proven Results, Future Promise
A9604-AB-13 12/2011
Overview
• Background on team and state-wide evaluation
• Approach to evaluation
• Current state of research in three initiative areas
• Evaluation team and program partners: Relationship building
A9604-AB-14 12/2011
Evaluation Approach: Structure-Process-Outcomes
STRUCTURE
What PEI capacities &
resources are PPs developing and implementing?
• Networks • Needs assessment • Service expansion • Outreach • Training & technical
assistance • Screening • Educational resources • Marketing campaigns • Cross-system
collaboration • Policies & protocols
PROCESS
What intervention activities are delivered,
and to whom?
• Participation in training & education
• Exposure to outreach • Exposure to media • Use of services • Integration of
services across organizations
SHORT TERM OUTCOMES
What are immediate targets of change?
• Knowledge • Attitudes • Normative behavior • Mental & emotional
well-being • Help-seeking
KEY OUTCOMES
What negative outcomes are
reduced?
• Suicide • Discrimination • Social Isolation • School drop-out
A9604-AB-15 12/2011
Purposes of Evaluation
• Formative Evaluation – Provide early feedback on measures of process and
proximal outcomes – Feedback can inform program partners
implementation efforts, and suggest changes in structure of PEI initiatives
• Summative Evaluation – Assess short-term and key outcomes to evaluate
overall impact – Examine relationships between process and short-
term outcomes to understand what had impact – Examine relationships between short-term and key
outcomes to test social change assumptions
A9604-AB-16 12/2011
Overview
• Background on team and state-wide evaluation
• Approach to evaluation
• Current state of research in three initiative areas – Suicide prevention – Stigma reduction and prevention – Student mental health
• Evaluation team and program partners: Relationship building
A9604-AB-17 12/2011
What We Know About Suicide Prevention
• CalMHSA evaluation presents a significant opportunity to move the field forward because few suicide prevention initiatives have been evaluated
However
• Suicide is rare, and it is difficult to show universal approaches reduce deaths by suicide
• Evaluations often focus on short-term outcomes that are logically or empirically linked to suicide
A9604-AB-18 12/2011
Types of Programs Evidence base Informational Resources • No evidence of effect
• Generally considered a ‘good idea’, but is it? Gatekeeper Trainings • Changes in knowledge and attitudes among certain trainees
Hotlines • Lower levels of suicidality and hopelessness among suicidal callers after calls
Universal Screening • Little evidence of effect Training Providers • Training primary care providers on depression awareness may
reduce suicides Treatment Interventions • Certain treatments lead to reductions in suicide or attempts
for: suicide attempters, persons with bi-polar disorder, and patients with major depression
Social/Policy Interventions • Evidence that means restrictions lead to reductions in suicide
Comprehensive Approaches
• Multi-component efforts implemented in organizations and institutions have reduced suicide rates
Evidence of Varying Effectiveness for Suicide Prevention Initiatives
A9604-AB-19 12/2011
The Rareness of Suicide and Data Lags Often Impede Timely Outcomes Evaluations
• Two sources of national and state-level suicide data – NCHS Vital Statistics System (3 year lag) – NVDRS for 16 states (1.5 year lag)
• CA is not a participating state in NVDRS
USA
CA
0
2
4
6
8
10
12
14
2000 2001 2002 2003 2004 2005 2006 2007 2008
Suicide rate per 100,000
Suicide in the US and California*
* Age-Adjusted, with 2000 as standard year
3775 suicides in 2008
A9604-AB-20 12/2011
Outcome Short Description Strengths Weaknesses Suicide Did an intervention lead to a reduction in
deaths by suicide? Key Outcome - Rare/Power
- Data Availability
Suicide Attempt
Did an intervention lead to a reduction in suicide attempts (objective reports or self-reported)?
Best predictor of suicide (5-15%)
Half of all suicides have no history of attempt
Suicide Ideation
Did an intervention lead to reductions in thinking about killing oneself?
Self-report suicidal thoughts
Relationship to suicide is unclear
Change in Functioning
Did an intervention lead to reductions in mental health functioning linked with suicide?
Self-reports of thoughts thought to relate to suicide
Relationship to suicide is unclear
Referral Did an intervention increase referrals or utilization of other, specific resources?
Often is the intended outcome
Relies on referral to prevent suicide
Knowledge and Attitudes
Did an intervention change the way people think about suicide, or how care professionals manage suicidal individuals?
May lead to increased referrals
Unclear whether changes in K&A are actionable
Other Behavior Change
Did an intervention change functioning in other domains that may indirectly impact suicide (e.g., promote responsible drinking)?
May eventually lead to reductions in suicides
Predictive link with suicide is often weak
Varying Outcomes Used to Evaluate Suicide Prevention Programs
A9604-AB-21 12/2011
Overview
• Background on team and state-wide evaluation
• Approach to evaluation
• Current state of research in three initiative areas – Suicide prevention – Stigma reduction and prevention – Student mental health
• Evaluation team and program partners: Relationship building
A9604-AB-22 12/2011
What Do We Mean by Stigma?
• Defined as – Prejudice – Negative stereotypes – Discrimination against an individual or group
• Two manifestations – Internalized/private – Social/public
A9604-AB-23 12/2011
Why Are Stigma and Discrimination Mental Health Issues?
• Reduce help-seeking, service utilization, and treatment compliance
• Can adversely affect self-perceptions/mental health
• Cause social exclusion/limit opportunity – Limit family/peer support – Reduce healthcare access – Create employment/housing/school issues – Result in negative interactions with criminal justice
A9604-AB-24 12/2011
Multi-Level Interventions Are Thought to Be Most Effective
Individual Change
Social Change
Policy/Practice Change
A9604-AB-25 12/2011
Individual Change
Social Change
Policy/Practice Change
Outcomes at Each Level of Change
Change in knowledge, attitudes, behavior
Shifts in community discussion,
norms
Change in organiza-
tional policies, practices
A9604-AB-26 12/2011
Three Kinds of Programs Are Used to Reduce Stigma and Discrimination
• Training presentations for police, health care providers, in schools, and in workplaces
– Education only – Video “contact” – Direct/In-person contact
• Media Interventions – Deliberate communication/media campaigns – Advocacy and education to affect media coverage
• Policy and practice reviews
A9604-AB-27 12/2011
Evaluations of Existing Training Interventions
• A handful of small pre-post studies of police and workplace trainings have been conducted
• There is a more substantial literature on SDR trainings in college student and other school-based populations
• These evaluations suggest trainings involving contact, education, or both are effective, with contact a priority
• The changes demonstrated are limited to short term shifts in attitudes
A9604-AB-28 12/2011
Little Is Known About the Effectiveness of Media Campaigns
• Background noise: hard to distinguish effects of a campaign from other change-producing factors and unrelated secular trends
• Message spillover into the control group
• Media effects are gradual and cumulative (a “drip” effect) unless the information communicated is novel
• Matching study sample to target audience can be difficult
A9604-AB-29 12/2011
Considerations Specific to Evaluating Media Campaigns
• Test potential - are the messages effective ones?
• Document implementation – are multiple well-chosen channels used?
• Collect exposure metrics – is substantial portion of target audiences reached and with high frequency?
• Measure immediately targeted outcomes – is awareness, knowledge affected among those reached?
A9604-AB-30 12/2011
Overview
• Background on team and state-wide evaluation
• Approach to evaluation
• Current state of research in three initiative areas – Suicide prevention – Stigma reduction and prevention – Student mental health
• Evaluation team and program partners: Relationship building
A9604-AB-31 12/2011
What Is Student Mental Health?
• Programs and interventions to address needs of students with emotional and behavioral problems or enhance students social and emotional learning
• Commonly focused on students, but may also involve parents and siblings
• May be staffed by school personnel, or by individuals from other organizations working in the schools; treatment services provided in schools or in local community
• Knowledge and evidence base greater for K-12 schools than for colleges and universities
A9604-AB-32 12/2011
Common Types of Student Mental Health Programs
• Clinical services – Efficacy depends on nature of clinical treatment provided--
school-based CBT programs have strongest evidence base
• Targeted intervention/prevention programs – CBITS efficacious for trauma symptoms – FRIENDS/Coping Cat efficacious for anxiety
• Primary prevention programs – Project ALERT – efficacious for substance use – SOS- quasi-randomized- effective for decreasing suicide
attempts
• Holistic/school-wide approaches to student mental health – PBIS – improved behaviors in pre-post and non randomized
comparison groups
A9604-AB-33 12/2011
What We Know About Student Mental Health Evaluation
• Relatively few student mental health programs have been rigorously evaluated
• Intervention processes are most easily assessed; clinical and academic outcomes are more challenging
• Most rigorous mental health evaluations have focused on specific programs and subsets of outcomes; clinical programs more often evaluated than prevention or curricular programs
• School-based substance abuse programs have a history of more widespread evaluations when compared to mental health programs
A9604-AB-34 12/2011
Student Mental Health Programs Often Rely on Broader Community Resources
• Programs to enhance identification and referral of students for treatment may depend on factors beyond the program’s control
– Are resources available in the community to provide treatment?
– Can student or family pay for treatment if necessary?
– Is student and/or family willing to accept referral?
• Information needed to evaluate referral component of identification and referral program may be difficult to obtain
A9604-AB-35 12/2011
Issues Specific to College Mental Health
• Field is generally not as developed as K-12 mental health
• Common types of programs – On campus clinical services – General population programs to increase awareness and
decrease stigma – Targeted programs for specific populations or issues
• Suicide prevention • Recognition of onset of schizophrenia • Early identification and referral (recently often tied to
concerns about violence) • Drug/alcohol related, Intimate partner violence, other
at risk groups
A9604-AB-36 12/2011
Student Performance Is Key Outcome, but Data Challenges Exist
• School data: grades, attendance, and test scores can be used to evaluate programs, but schools vary widely in the data available
– Electronically captured (and if captured available) – Time intervals (quarter, trimester, semester) – Metrics (A, B, C, D, F vs. E, S, NS vs. 1, 2, 3) – What information is kept and for how long – Program may span grading periods – Grades reflect cumulative performance over a
period of time
A9604-AB-37 12/2011
Implications for CalMHSA Evaluation
STRUCTURE
What PEI capacities & resources are PPs
developing and implementing?
• Networks • Needs assessment • Service expansion • Outreach • Training & technical
assistance • Screening • Educational resources • Marketing campaigns • Cross-system
collaboration • Policies & protocols
PROCESS
What intervention activities are delivered,
and to whom?
• Participation in training & education
• Exposure to outreach • Exposure to media • Use of services • Integration of
services across organizations
SHORT TERM OUTCOMES
What are immediate targets of change?
• Knowledge • Attitudes • Normative behavior • Mental & emotional
well-being • Help-seeking
KEY OUTCOMES
What negative outcomes are
reduced?
• Suicide • Discrimination • Social Isolation • School drop-out
A9604-AB-38 12/2011
Overview
• Background on team and state-wide evaluation
• Approach to evaluation
• Current state of research in three initiative areas
• Evaluation team and program partners: Relationship building
A9604-AB-39 12/2011
Next Steps in Statewide Evaluation
• Compile information from SOWs to learn about Program Partners
– Contact information, schedules, resources – Goals, elements of programs, target populations
• Conduct “get-to-know-you” visits or calls with each Program Partner
– Clarify and update plans, understand logic models – Review evaluation plans, including data collection – Discuss technical assistance needs and options
Part 2
A9604-AB-40 12/2011
Development of the Strategic Plan
• Collaborate and coordinate with Program Partners
• Assess Program Partner interest and need for technical assistance
• Understand Program Partner plans for collecting baseline data and existing program evaluation plans
• Seek guidance from the SEE, CalMHSA, Program Partners, Counties and other stakeholders
A9604-AB-41 12/2011
Data for Evaluation
• May include use of data collected by individual programs, existing data, and data RAND collects
• RAND will: – Work with Program Partners to develop
specifications for data to be provided – Provide templates and secure methods for
providing data – Provide reports on data completeness and quality
• Goal is to avoid duplication and excessive burden
A9604-AB-42 12/2011
Timeline for Statewide Evaluation
• Request for Qualifications issued August, 2011
• Proposals due and RAND selected October, 2011
• Negotiation of RAND SOW concluded November, 2011
• “Get To Know You” meetings December, 2011 —January, 2012
• Draft SESP to be available for review April, 2011
• Final SESP to be adopted June, 2011
• Contract concludes June, 2014