Evaluating Innovation - mhsaresources.orgmhsaresources.org/.../05/EvaluatingInnovationPart2.pdf ·...

Evaluating Innovation, Part2 012512 1

January 25, 2012Cricket Mitchell, PhD

CiMH Consultant

Evaluating InnovationPart 2 of 2

Introduction• At the cohort meeting in December we

reviewed key components of Innovation evaluation– Articulating program elements and goals,

measurement, appropriate points of comparison, utilizing a Logic Model, and selecting evaluation strategies

• Today we will take a closer look at applying these key components to representative examples within each of the three cohort areas

2

Overview

• Brief review of Part 1

• Example of Innovation Evaluation: New roles for peers– A closer look at the Logic Model example

from Part 1

• Example 2: Culturally relevant services

• Example 3: Primary care and behavioral health integration

• Questions and discussion

3


A note…

• Examples are provided for illustrative

purposes; they are not intended to be exhaustive or precise.

4

Summary of Part 1• Innovation evaluation combines both process

evaluation and outcome evaluation, with the ultimate goal of contributing to learning– Evaluation for all Innovation projects should include

measurement of both process and outcome learning goals

– And, in order to understand data collected, the most appropriate point(s) of comparison should be identified

• As a flow chart representing the components of an Innovation program, a Logic Model serves as a guide for evaluation

5

Building and Utilizing a Logic Model: Review

• A Logic Model is a pictorial representation,

a flow chart, of the hypothesized relationships between program Inputs,

Activities, Outputs, and Outcomes

– Provides a guide for measurement and data collection activities

6



• Inputs– Resources necessary to achieve objectives

• Activities– What the program does with the resources to

meet the objectives

• Outputs– Direct products of program activities

• Outcomes– Changes that result from the program’s activities

and outputs

7


• Linking the findings from process and

outcome learning goals back to independent program elements is the

heart of evaluation for Innovation

– The ultimate goal is to identify program elements that are related to the achievement

of desired goals and can be adopted or replicated by others

8

Diverse stakeholder

input is important in

all aspects of

Innovation evaluation

9


10

Example: Evaluating a New Roles for Peers Innovation Project

A Closer Look at the Logic Model Example from Part 1

Logic Model Example

• Let’s say we have an Innovation program

that aims to improve outcomes for youth leaving foster care by pairing a TAY peer

mentor with an older youth as they are

discharged from the foster care system...

11

Articulating Program Elements

• Questions to ask:

– Who is being served?

• Older youth discharging from foster care

– Is this Innovation intended to bring change to

anyone else / anywhere else?

• Publicly-funded service agencies?

– e.g., by coordinating available services and supports

• Young adults with histories of foster care experience (i.e., peer mentors)?

12




– What are this program’s activities?

• Pairing a peer mentor with older youth discharging from

foster care

– What are the key strategies or processes by which the activities will be achieved?

• How will youth and mentors be paired?

• What training and supervision are mentors provided?

• How will collaborating agencies work toward coordinated services and supports?

13

Articulating Program Elements• Questions to ask:

– What is the timeline?• Is there a limited or proscribed amount of time that the

mentor will work with a peer?

• When would one expect to see change as a result of this Innovation?

– Who delivers the activities?• What qualifications or attributes are necessary for peer

mentors to be hired?

• Who will oversee the program? Who will train and supervise the peer mentors?

• How will representatives from collaborating agencies be recruited for participation? What agencies will be involved?

14

• Process learning goals and outcome

learning goals combine to inform a broader set of

findings and recommendations from Innovation

programs

15

Articulating Program Goals



• Possible outcome learning goals…– Does this increase knowledge about available

services and supports?

– Does this increase coordination of available services?

– Does this increase participation in available services and supports?

– Does this increase rates of employment, education/vocation, independent living?

16


• Possible process learning goals…– What mentor attributes are most likely to

contribute to success?

– What aspects of collaboration among community-based agencies are most likely to contribute to success?

– What information do older youth leaving foster care find most useful?

– What activities, services, and supports are identified as most helpful?

17

Logic Model Example

•TAY Peer Mentors

•Reps from Collaborating Agencies (e.g., CW/SS, MH/BH, Ed, WFTD)

•Guiding principles, policies, and training

TAY Peer Mentor paired with TAY at time of d/c from FC

Peer-provided assistance with forms

Peer-provided emotional support

Coordinated service options

Coordinated service delivery

Inputs Activities OutcomesOutputs

TAY engagement in emotional and informational supports from Peer Mentor

TAY engagement in coordinated services and supports from community-based agencies

Increased utilization of services and supports by TAY exiting the FC system

Increased enrollment in educational and/or vocational opportunities Increased rates of

Independent Living

Increased rates of employment

Improved Quality of Life

18


Logic Model Example

– Inputs:

• TAY peer mentors

• Representatives from collaborating

agencies

• Guiding principles, policies and training

19

Examples of Measurement for Inputs

•TAY Peer Mentors



Inputs •How many TAY Peer Mentors are hired?

•Are they retained/do they stay employed?

•How many agencies are represented in the collaboration?

•How many representatives participate from each agency?

•Are they retained/do they consistently participate?

•How are the Guiding Principles, Policies, and Training protocols

developed and enacted?20

Logic Model Example

– Activities:

• Pairing TAY with peer mentor at time of discharge from FC

• Peer-provided emotional support, information, and assistance with completing paperwork and forms

• Coordination of service options by collaborating agencies

• Coordination of service delivery by collaborating agencies

21


Examples of Measurement for Activities






Activities •How is the older TAY discharging from Foster Care paired with a Peer Mentor?•How often does the Peer Mentor provide emotional support?•Does the emotional, informational, and instrumental support provided meet the needs of the TAY?•How are service options presented to the TAY?•Are services delivered in a coordinated manner?

22

Logic Model Example

– Outputs:

• TAY engagement in supports offered by peer mentor

• TAY engagement in coordinated services and supports from collaborating agencies

23

Examples of Measurement for Outputs

Outputs



•Does the TAY discharging

from Foster Care access

and utilize the emotional,

information, and

instrumental supports

offered/provided by the Peer

Mentor?•Does the TAY find them

helpful?

•Does the TAY engage in any of the coordinated

service options presented?•Does the TAY stay

enrolled in any of the services?

24


Logic Model Example

– Outcomes:

• Increased utilization of services and supports by TAY exiting the foster care system

• Increased enrollment in educational and/or vocational opportunities

• Increased employment

• Increased rates of independent living

25

Examples of Measurement for Outcomes

Outcomes



Independent Living



•Does the TAY enroll in any

educational or vocational

opportunities?

•Does the TAY seek employment?•Does the TAY gain employment?

•Does the TAY retain employment?

•Does the TAY utilize available

community-based services and supports?

•What is the TAY’s living environment

status six months, 12 months, 24

months after discharge from Foster Care?

•What is the TAY’s perceptions of

his/her Quality of Life?

26

Process Goals

•TAY Peer Mentors













Independent Living



27


• Hold focus groups, conduct interviews, and/or develop questionnaires to get the perceptions

of those involved with the Innovation on the hypothesized relationships between program

Inputs, Activities, Outputs, and Outcomes

– TAY participants

– Peer mentors

– Representatives from collaborating agencies

– Other stakeholders

Examples of Measurement and Data

Collection for Process Goals

28

• Tracking system (e.g., Excel workbook)– Include dates of critical activities

• e.g., recruiting representatives from partner agencies, hiring peer mentors, developing policies & procedures, conducting trainings, introduction of TAY and peer mentor, enrollment in services

– Include numbers• e.g., # of interviews for peer mentors, # of

successful hires, # of trainings, # of contacts/encounters between TAY and mentor, # of contacts by type of service or support

29

How Might These Data be Collected? (1 of 3)


• Interviews– TAY

• What was helpful? What was not helpful? Did services appear to be coordinated? What else could have been done/offered/provided?

– Peer mentors• Did the TAY participant accept/engage in all

contacts? What seemed to be most successful? What seemed to be least successful? What else could have been done/offered/provided?

30



• Interviews

– Representatives from public agencies

• What processes have changed within and across agencies as a result of this Innovation? What worked? What didn’t?

• Standardized Quality of Life questionnaire

31

Appropriate Points of Comparison

• Historical data: Foster youth discharged in two years prior to implementation of Innovation– Rates of service utilization, enrollment in

educational and/or vocational opportunities, employment, independent living

• Current data: Foster youth discharging currently who don’t have access to or don’t participate in the Innovation– Rates as above, plus Quality of Life questionnaire

32

Employing Evaluation Strategies: Use of Descriptive Statistics

• Side-by-side comparisons of rates of desired outcomes (frequency bar charts)– Data from prior years and data since

Innovation implementation

– Data from Innovation group and current data from non-Innovation group

• Responses from participants regarding what worked and what didn’t (percentages)

33


• Descriptive Statistic: Frequency Counts,

Bar Chart

34

4245

75

2 Years Prior to Innovation 1 Year Prior to Innovation 1 Year After Innovation

Gainful Employment of 20 hrs per Week or More

• Descriptive Statistic: Percentages, Pie

Chart

45%

35%

13%

7%

"How helpful was the peer mentor in assisting you with completing forms?"

Very helpful

Somewhat helpful

A little helpful

Not helpful

35

Utilizing Your Innovation Logic Model

• The Logic Model is not a static document

• Timelines should be established for review of the logic model, likely linked with the timing of measurement

– Document the progression of each review of the logic model - including evidence that supports proposed relationships as well as evidence that either does not support or suggests unexpected relationships

• Modify and update key components and proposed relationships in the Logic Model as data are collected and lessons are learned

36


37

Questions and Discussion

38

Example 2: Evaluating a Culturally-Relevant Services Innovation

Project

Logic Model Example 2


that aims to increase access and utilization of mental health services among

Native American youth...

39




– Who is being served?

• School-age Native American children and youth

– Is this Innovation intended to bring change to

anyone else / anywhere else?

• The Native American tribe/community

• Schools serving Native American children and youth

40


– What are this program’s activities?• Tribal leaders promote education, awareness, and

identification

• School staff promote education, awareness, and identification

• School-based screenings


• What training materials will be used?

• How will gatekeepers be trained and supervised?

• Will there be ongoing technical assistance for tribal communities and schools?

• How will screenings be conducted and managed?

41


• Questions to ask:– What is the timeline?

• Is there a limited or proscribed amount of time that structured activities will include education and awareness information?


– Who delivers the activities?• How are champions at the tribal and school levels

selected?

• How are gatekeepers identified? Will they receive supervision, ongoing technical assistance?

• How will positive screenings and/or identified need for services be handled?

42



• Possible outcome learning goals…– Does this increase knowledge and decrease

stigma about mental health issues?

– Does this increase identification of those in need of services?

– Does this increase access and utilization of available services and supports?

– Does this increase mental health and social role functioning?

43


• Possible process learning goals…– What types of information promoted increased

awareness and knowledge?• In what settings? Delivered by whom?

– What setting characteristics within the tribal community and schools contributed to greater success of the Innovation?

– What factors were most influential in youth pursuing (or not pursuing) referred services and supports?

44


•Tribal leaders as “Champions”

•Screening tools and educational materials adapted for cultural relevance

•Culturally-relevant trainings for school- and community-based mental health professionals and “gatekeepers”

Awareness and anti-stigma info woven in to structured tribal activities

School-based screenings


Tribal-based identification of need and referrals to available services and supports

School-based identification of need and referrals to available services and supports

Increased utilization of available services and supports

Increased access to available services and supports

Improved mental heath functioning

Increased engagement

Improved social role functioninge.g., improved academic performance

45

Awareness and anti-stigma info woven in to structured school-based activities



– Inputs:

• Tribal leaders who will function as “champions”

• Screening tools and educational materials modified for cultural relevance

• Culturally-relevant trainings for school- and community-based mental health professionals and “gatekeepers”

46





Inputs

47

•How many tribal leaders will serve as

champions?•How are they recruited?•Do they have different responsibilities as champions?

•Who is involved with adapting materials

(screening tools, educational materials,

training curriculum) for cultural

relevance?•What is the process for determining or agreeing upon changes?•How are the materials tested?

•Who will be trained as gatekeepers?•How will their roles be tracked?


– Activities:

• Awareness and anti-stigma information woven into structured tribal activities

• School-based screenings

• Awareness and anti-stigma information

woven into structured school- based activities

48




Activities

49



•How many tribal and school-

based events will include

awareness and anti-stigma

information?•What is their frequency?•Will different information be presented at different events?•Will the presenters vary?

•How many screenings are

conducted?•What is the frequency with which screenings will be conducted?


– Outputs:

• Tribal-based identification of need and referrals to available services and supports

• School-based identification of need and referrals to available services and supports

50

Outputs



51

Examples of Measurement for Outputs•How many youth are identified

in need through members of

the tribal community?•How many youth are

referred?

•What referrals are provided?

•How many school-based

screenings identify need?•How many referrals result

from screenings?

•How many youth are identified

in need through school-based

gatekeepers?•How many youth are

referred?•What referrals are provided?



– Outcomes:

• Increased access to available services and supports

• Increased engagement in available services and supports

• Increased utilization of available services and supports

• Improved mental health functioning

• Improved social role functioning– e.g., improved academic performance

52

Outcomes






53


•How many youth access referred


•How many youth return after initial access?

•How many youth are retained in

services and supports until

identified goals are met?

•What is the average length of time

that youth utilize referred services and supports?

•What proportion of youth have

improved functioning after

utilization?•What is the average amount of improvement?

•In what areas/domains?














54


Process Goals


Appropriate Points of Comparison

• Historical data: Community and school-based data from prior years– Access and utilization of services

• Also compare to prevalence estimates

• Current data: Youth who access and utilize referred services as a function of Innovation– Academic performance, historical and current

– Mental health functioning, pre- and post-• i.e., using a standardized questionnaire

55

56


57

Example 3: Evaluating a Primary Care and Behavioral Health

Integration Innovation Project




that aims to improve the physical and mental health of its community members

by implementing bi-directional integration

of physical and behavioral health services in a rural area...

58


• Questions to ask:– Who is being served?

• Primary care clients with mental and/or behavioral health issues

• Mental health clients with physical and/or behavioral health issues

– Is this Innovation intended to bring change to anyone else / anywhere else?

• Primary care clinics

• Mental health clinics

• Staff of varying disciplines within each set of clinics

59


– What are this program’s activities?• Trainings and ongoing technical assistance in

integration using an established curriculum– Implementation and sustainability activities

• Screening, assessment, interventions, referrals


• How will clinics be recruited? Are there minimum characteristics or qualifications?

• How will trainings and T.A. be delivered and tracked?

• How will screenings, assessments, interventions, and referrals be coordinated and tracked?

60



– What is the timeline?• How long is the training phase? How long is technical

assistance provided?


– Who delivers the activities?• Who are the expert trainers and advisors?

• What set of systems need to be in place in each clinic to provide oversight for the new activities?

– Who will be “in charge” of the Innovation in each clinic?

– What new hires are necessary?

61


• Possible outcome learning goals…– Does this increase identification of primary care

clients in need of mental health services and mental health clients in need of physical health services?

– Does this increase the capacity for clinics to provide comprehensive care?

– Does this increase access and utilization of available services and supports?

– Does this improve physical and mental health functioning of clients served?

62


• Possible process learning goals…– What aspects of the training, technical assistance,

implementation and sustainability activities are perceived as most helpful?

– What clinic characteristics are associated with increased likelihood of success?

– Are there differences across disciplines and/or clinics with regard to acceptance and implementation of integration?

– Are there client characteristics that are better served in one clinic vs. another? (e.g., primary care vs. mental health)

63



•Primary care clinic(s) interested and willing to integrate mental health services

•Mental health clinic(s) interested and willing to integrate physical health services

•Cross-training and integration curriculum (with established history of success in rural areas)

Trainings and ongoing technical assistance to recruited clinics

Screenings and assessments


Increased rates of identification of needed services and supports

Increased education and awareness of physical and behavioral health issues among patients and clients served





Improved physical health functioning

64

Participation in implementation and sustainability activities with experts from established , successful bi-directional integration


– Inputs:

• Primary care clinic(s) interested and willing to integrate mental health services

• Mental health clinic(s) interested and willing to integrate physical health services

• Cross-training and integration curriculum with

an established history of success in rural areas

65




Inputs

66


•How many clinics will participate in the

Innovation?•How are they recruited?•What are the reasons sites choose to participate (or not)?•What is the staff make-up (e.g. by discipline) within each clinic?

•Are there any modifications or adaptations to the cross-training and integration curriculum?

•How are these decided?•How are they tested?



– Activities:

• Trainings and ongoing technical assistance to

recruited clinics

• Screenings and assessments

• Participation in implementation and sustainability activities with experts from established, successful bi-directional

integration

67



Activities

68



•How many trainings, technical assistance contacts, and other implementation and sustainability activities are offered?

•What are the number of participants across disciplines and/or clinics for each?

•How many screenings are

conducted?•What is the frequency with which screenings are conducted?•What proportion of screenings lead to assessments?


– Outputs:

• Increased rates of identification of needed services and supports

• Increased education and awareness of physical and behavioral health issues among patients and clients served

69


Outputs



70

Examples of Measurement for Outputs

•How many assessments lead

to referrals?•How many referrals are pursued?

•How many clients report greater awareness of physical and mental health issues?

•How many clients report

increased understanding of how to manage / care for their

physical and/or mental health

issues?•How many clients report

better management of their

health?


– Outcomes:

• Increased access to available services and supports

• Increased engagement in available services and supports

• Increased utilization of available services and supports

• Improved mental health functioning

• Improved physical health functioning

71

Outcomes






72


•How many clients access referred


•How many clients return after initial access?

•How many clients are retained in

services and supports until

identified goals are met?

•What is the average length of time

that clients utilize referred services and supports?

•What proportion of clients

demonstrate improved health

status?•What is the average amount of improvement?

•In what areas/domains?















73


Process Goals

• Data from the established bi-directional integration in its original implementation– At the systems/clinic level

• Staff make-up (e.g., by discipline), numbers participating in training and T.A. events, systems changes to accommodate integration

– At the client or patient level• Numbers served by integration (screenings,

assessments, interventions, referrals), rates of improved self-management and/or improved physical and mental health functioning

74

Appropriate Points of Comparison (1 of 2)

Appropriate Points of Comparison (2 of 2)

• Patients and clients who participate in

integration activities

– Physical health functioning, historical and current

• e.g., using standardized measures of health status

– Mental health functioning, pre- and post-

• i.e., using a standardized questionnaire

75


Summary

• A Logic Model should be used to guide

Innovation evaluation

– Clearly articulate program elements and program goals (process learning goals and outcome learning goals)

– Build a Logic Model by identifying the key components of inputs, activities, outputs, and outcomes, as well as the hypothesized

relationships among key components

76

Summary (cont’d)

– Utilize the Logic Model throughout the Innovation

• Collect data from multiple sources and examine the extent to which the proposed relationships in the Logic Model hold

– What works, what doesn’t

– What should be continued, what should be changed

• In evaluating outcomes of Innovation, ensure that appropriate points of comparison are employed

– Data in and of themselves don’t tell us much… Comparisons allow greater understanding and

interpretation of data collected

77

78



79

The End

Contact Information

•Cricket Mitchell, PhD•Email: [email protected]

•Cell phone: 858-220-6355

Date post:	04-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Evaluating Innovation - mhsaresources.orgmhsaresources.org/.../05/EvaluatingInnovationPart2.pdf ·...

Documents