Implementing evidence-informed practice
A practical toolkit
Learning Collaboration Leadership
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 2
Table of contents
Introduction 4
Implementing evidence-informed practice 6
Implementation science 6
Evidence-informed practice 7
Learning organizations 7
Framework for implementation 9
Planning phase: Getting started 10
I. Building leadership support Organizational implementation capacity 11
Leadership support 12
Senior management charter 13
Creating a budget 14
II. Engaging stakeholders Building staff buy-in and readiness 15
Engaging youth and families 20
Communicating change to stakeholders 21
Maintaining momentum 22
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 3
III. Managing and leading change Choosing a team 23
Regular team meetings 25
Assessing the evidence 27
Selecting an evidence-informed practice 31
Creating an implementation plan 32
Selecting a purveyor 34
Ongoing evaluation and monitoring 35
Implementation activities and timeline 36
Preparing for the next phases of implementation 37
General tips and strategies for implementing EIPs 39
Additional resources 40
References 40
Acknowledgements 42
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 4
HELPFUL INFORMATION
About the Centre
As a leader for child and youth
mental health in Ontario, we build
connections, share knowledge
and draw upon our expertise to
enhance the skills of front line service
providers. We passionately
promote and share the benefits
of both organizational learning
and evidence-informed practices.
Together, with our partners, we are
working to strengthen Ontario’s
mental health programs and services
for all children, youth, families and
caregivers.
Other available toolkits
“Try to take things one day at a time. If you rush the process you may overlook important things or not be truly prepared for your implementation..”
— Implementation support service
(ISS) agency
Introduction• Are you planning on introducing a new program to
address the mental health needs of children and youth in your community?
• Are you wondering what it would take to sustain a program or practice in your organization?
• Are you looking for tools and resources to kickstart the implementation process?
Welcome to the implementation toolkit
Implementation is a continuous process that includes a specified set of activities designed to put into practice an activity or program of known dimensions (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005). Knowing about implementation is one thing, but doing it is another. As with learning any new skill, only through practice will individuals and organizations become proficient in implementation processes.
As you begin on your implementation journey, be sure to consider the
following:
• Research indicates that it takes two to four years to implement a new practice/service into your organization (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005).
• Rushing or skipping steps in the implementation process may result in a false start or an inability to sustain the practice.
• Without a well thought-out plan, implementation can take longer, cost more money, have a negative impact on your people and/or become too costly to maintain.
• While the phases of implementation (planning, doing and sustaining) may overlap, each takes six to 12 months to complete.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 5
This toolkit contains practical insight, strategies and resources for the planning phase of implementation. It couples theory and research findings with practical strategies and real-life experiences from the field that may be relevant to your organization.
Implementation support services at the Ontario Centre of Excellence for Child and Youth Mental Health
The Ontario Centre of Excellence for Child and Youth Mental Health (the Centre) builds connections, shares knowledge and draws upon expertise to enhance the skills of direct service providers and agencies across the province. The Centre passionately promotes and shares the benefits of both organizational learning and evidence-informed practice. Together with our partners, the Centre is working to strengthen Ontario’s mental health programs and services for all children, youth, families and caregivers.
The Centre’s implementation support services are designed to meet the needs of Ontario’s child and youth mental health agencies and their communities. We work with agencies to achieve sustained success in the implementation of evidence-informed practices using an integrated program of education, training and ongoing consultation. We offer variable levels of support to match the needs of diverse organizations, and our team can work with agencies at any point in the change continuum.
For more information on available services please visit our website.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 6
Implementing evidence-informed practiceLet’s start with the big picture. When planning a change in the services you provide, it helps to understand the concepts of implementation science, evidence-informed practice and learning organizations and consider how they can work together to help you succeed.
Implementation science
The science of implementation draws from a multidisciplinary set of theories and methodologies that bridge the gap between research and practice. It is essential to consider the processes involved when integrating new practices within your organization.
Identifying an effective practice and training staff to deliver it are not enough to achieve successful implementation. A program that has been shown to improve mental health outcomes for children and youth can be poorly implemented and fail to produce good outcomes – this is referred to as an implementation gap. Either the practice is not used as designed or isn’t applied consistently over time.
Current research and practical evidence tell us that factors called implementation drivers influence the successful implementation of EIP (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005). These drivers include leadership support, clinical supervision, performance management, policies and procedures. When implementation drivers are in place, the risk of there being a gap between research and practice is reduced.
www.excellenceforchildandyouth.ca 6
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 7
Evidence-informed practice
When implementing evidence-informed practices into the real world setting, it can be difficult to match scientific research findings to the specific needs of an organization and the people it serves. Where evidence exists, it may have been developed in a research environment but does not produce the same results in a less controlled evironment. Research findings may also exclude many promising practices that have not yet had a chance to be systematically evaluated. Finally, the practice may simply not make sense for a given population.
Evidence-informed practices are provided to clients in a way that combines the best available research with the experience and judgment of practitioners and the unique preferences of children, youth and their families to deliver measurable benefits. When an agency provides an evidence-informed practice for clients, it is more than finding and implementing an off-the-shelf manualized intervention. It’s about continuously gathering, sharing and using evidence from the research literature, practitioners experience and feedback from children, youth and families. By combining these elements, organizations can strengthen their services and improve outcomes for the children, youth and families they serve.
Although the terms evidence- based practice (EBP) and evidence-informed practice (EIP) are used interchangeably in the literature, the Centre favours evidence-informed practice. In order to properly reference the research, we will continue to use the same terminology (EBP or EIP) suggested by the respective author(s).
Learning organizations
Learning organizations are on a constant journey of discovery and organizational development. They are willing to take risks, embrace creativity and enjoy a culture that promotes continuous improvement and professional growth. Learning organizations provide a practical and purposeful way to embrace evidence-informed practice, share knowledge and improve outcomes for children, youth and families.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 8
Gather information: Evidence is everywhere. You can find it in peer-reviewed journals, stories of successes and challenges, your daily work and the real-life experiences of the people you serve.
Use knowledge to grow: Gathering diverse and relevant information is critical but it’s what you do with it that matters most. Evaluation offers a way to arrive at your ultimate destination: better outcomes for children and youth. Implementation is the deliberate but flexible path to use evidence effectively.
Share your story: We can learn a lot from each other. Sharing research evidence is important but so is telling stories from the field. Stories are where we find meaning. They present real-life evidence that helps us find common ground. By sharing your story, you provide evidence for someone else.
The Centre asked Ontario mental health organizations what strategies they use to foster a culture of learning. Here are some of their responses:
• Be open to questions.
• Seek staff input for training ideas.
• Keep people up to date on what’s going on provincially/nationally.
• Have clear clinical goals.
• Encourage performance appraisals.
• Encourage community experience.
• Model positive leader attitudes.
• Foster an environment for people to ask questions.
• Use team meeting time as a time to learn.
• Provide learning opportunities.
• Develop partnerships.
• Use specific staff strengths and build on their knowledge and experience.
• Recognize staff as your best resource.
• Research connections with universities.
• Appreciate different skill sets.
• Acknowledge efforts.
• Conduct literature reviews relevant to each program.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 9
Framework for implementation
The Centre’s implementation framework was adapted from the implementation model of the National Implementation Research Network (NIRN) (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005), the clinical transformation model of Kinark Child and Family Services (Moore, Cunning & Roundwaithe, 2010), the Partnerships for Success program in the state of Ohio (Julian, Ross & Partridge, 2008; Kerns, Rivers & Enns, 2009) and the Replicating Effective Programs framework by the US Centers for Disease Control (Kilbourne, Neumann, Pincus, Bauer & Stall, 2007).
The goal of the planning phase is to consider evidence-informed practice implementation an organizational change initiative and get people and systems ready for that change.
ORGANIZATION LEARNINGORGANIZATIONAL LEARNING
PLANNING PHASEGetting people and systems ready for change01
Focus on: • laying the groundwork among
teams and systems• organizational readiness and
needs assessment
• assessing evidence • leadership buy-in and support• evaluation framework
DOING PHASETraining on specific practices, implementing, adapting and evaluating02
Focus on: • professional development and training on specific programs or practices• coaching, clinical supervision and communities of practice• monitoring and evaluation
Focus on: • ongoing monitoring and evaluation• ongoing coaching and clinical supervision of new staff• knowledge exchange and dissemination
SUSTAINING PHASELearning to continually use new evidence to improve practice03
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 10
During the planning phase, you may want to conduct an organizational implementation capacity assessment, engage staff readiness for change, consider youth and family feedback before practice change and select the new practice to address the current gap and/or needs. You want to focus on the core element of leadership commitment and support right from the beginning, as well as having evaluation practice in place within the organization before implementation begins.
Although the major activities of this framework are set into three stages, the process should be thought of as cyclical. That is, the framework functions like all continuous quality improvement approaches and the activities should be regularly revisited and refined.
For the timeline and activities of the planning phase of the implementation, check out the section on Implementation activitiesand timeline.
Planning phase: Getting started”There is so much to learn about implementation. The nuances and preparation are so important in the process”
— Implementation support service (ISS) agency
The following section describes each component in the planning phase and provides tools and tips from the mental health field to help you on your implementation journey. As you go through each part, you will find that success (or challenge) in one area is linked to success (or challenge) in another. While not sequential during the planning phase, the following elements should all be considered:
I. Building leadership support II. Engaging stakeholders III. Managing and leading change
Organizational implementation capacity
Leadership support
Senior management charter
Creating a budget
Building staff buy-in and readiness
Engaging youth and families
Communicating change to
stakeholders
Maintaining momentum
Choosing a team
Regular team meetings
Assessing the evidence and
selecting an EIP
Creating an implementation plan
Selecting a purveyor
Ongoing evaluation and
monitoring
Implementation activities and timeline
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 11
I. Building leadership support Organizational implementation capacity
Implementing an evidence-informed practice will involve change. The first step is to determine organizational capacity (ability to support) and readiness (preparedness and availability) for implementation. This process will consider things such as identifying staff attitudes and skills, assessing financial resources and competing demands (e.g. accreditation), as well as selecting the right EIP to meet organizational needs.
Implementation capacity checklistThe Centre has developed and adapted an implementation capacity checklist that can help organizations identify their capacity to implement an EIP. It assesses key characteristics, skills and processes that learning organizations use to facilitate ongoing integration of evidence into practice. Some items on the checklist were adapted from the Checklist to Assess organizational Readiness for Implementation (CARI) (Barwick, 2011). It examines the degree to which organizations and their staff have these key components:
• access and use of evidence
• youth, family and community engagement
• evaluation practices
• learning culture
• management of new or ongoing initiatives in implementation
• training and implementation plan for an evidence-informed practice
• strengths and limitations in using evidence
Who should fill out the checklist? The implementation capacity checklist should be completed by three to five people from a variety of disciplines within the organization (e.g. direct service provider, manager and senior leader).
Why is the checklist important? Organizational capacity and readiness is an under-emphasized but critical part of the implementation process. If organizations do not have the current capacity or are not ready to implement an evidence-informed practice, they will likely be unsuccessful. The checklist is also used as a measure to recognize where implementation capacity has grown over time. For example, it can help you:
• engage staff in implementation planning (start the conversation).
• set priorities and strategic directions.
• measure organizational readiness to implement a new practice.
• identify strengths and weaknesses.
• generate actionable items for planning purposes.
• lay the structure and groundwork, including developing any necessary processes, to prepare for change.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 12
Tips and resources
The implementation capacity checklist:
• is completed from the perspective of the organization as a whole.
• will provide a realistic snapshot of your organization.
• will help your organization review where you are in your implementation journey and should be completed annually.
Leadership support
“…Successful [organizational change] transformation is 70 to 90 percent leadership and only 10 to 30 percent management.” — John P. Kotter
What do we mean by leadership support?Leadership support refers to the explicit acknowledgement and buy-in from senior management within an organization, including the board of directors. This requires their understanding of the need for change and support for the organizational commitment to making it happen.
Why is leadership support important?Having leadership support is critical when a gap in service has been identified and the decision must be made to bridge it in an evidence-informed way. Senior leaders who buy in to – or drive – change efforts understand the value of evidence-informed practice, acknowledge the importance of thoughtful implemention and align organizational goals to facilitate the change process. Leadership support serves as a driving force to steer and shape the implementation, particularly in providing necessary resources.
Tips and resources
Consider the following:• Use findings from the implementation capacity checklist to create a presentation to senior management.
• Consider who your presenters are, because sometims the messenger is as important as the message. Is there a champion who can co-present to the leadership group?
• Highlight the rationale behind the change (the why) during the discussion with senior leaders. Explain the clear benefit to the organization, and by extension to staff, leadership and clients.
• Identify the critical success factors for implementation within the organization.
• Identify any potential barriers for implementation that may need to be considered.
• If possible, identify examples where your organization has successfully implemented an evidence-informed practice and how you can build upon it.
• Highlight alignment with your organization’s vision and values.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 13
Senior management charter
What is a senior management charter?The senior management charter outlines the key activities that are expected during your two- to three-year implementation journey. It is sometimes referred to as a terms of reference or project proposal, and it may be viewed as the contract for the core team. The charter needs to be agreed upon and supported by senior management within the organization. The document should explain why the implementation is important at this time, identify the areas of strength within the organization and areas that may require some support.
Who should complete the charter?The senior management charter should be completed by members of the core team who will be leading the implementation process.
Why is a senior management charter important?The charter is an important communication tool between the core team and senior leadership about the implementation process. Just like a blueprint for building a house, a senior management charter is an important step for initiating the implementation process. Through the charter, the leadership group will gain a general sense of the intent and scope of the implementation plan and the responsibilities of the core team members.
How do you create a senior management charter?After you’ve identified what program(s) you want to implement:
• Bring together a core team.
• Identify someone who will write the charter.
• Brainstorm the key elements of the implementation plan.
• Circulate the document for the core team to review.
• Have someone outside of your agency provide feedback.
Tips and resources
When writing a charter:• keep your audience in mind - senior management and board members.
• consider the results of your implementation capacity checklist.
• consider the implementation drivers as they are critical to the success ofyour plan.
• consider key community stakeholders.
• consider gathering input from clients and families you serve.
• Remember that this document will evolve with your project and should
be reviewed on a regular basis.
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
Here is an example of a senior
management charter with headings
and key questions that may
be incorporated into your own
document. You may use other terms
or components to reflect your own
organization.
Download example
Example from the field
A senior management charter
from Aisling Discoveries Child and
Family Centre: Facing your Fears:
Implementing a new evidence-
informed practice to improve
outcomes for children with anxiety
(and their parents)
Download example
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 14
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
Here is an example of the budget
template.
Download worksheet
Example from the field
In-kind time tracking sheet from Child
Community Resources,
Download example
Creating a budget
What is an implementation budget?Implementation requires resources – human and financial. When you consider allocating resources to the implementation effort, financial support is a crucial component to successful implementation. Without sufficient resources, change could cause frustration that impedes the implementation from moving forward.
Who should be involved in creating the budget?• all core team members in collaboration with senior leadership
• a financial officer could assist with estimating and tracking in-kindcontributions
How do you go about creating a budget?• Start by conducting a needs assessment to determine resources required
for the implementation initiative.
• From the senior management charter, list any items that may require budgetconsiderations (e.g. backfill, personnel costs, training costs, information andtechnology support, administrative costs as well as other resources).
• Consult with financial staff within the organization and discuss potentialresources that can be dedicated to the implementation.
• Include estimated in-kind contributions.
• Begin discussions on how to integrate implementation expenses into annualoperational costs.
• Keep accurate documentation of actual costs.
• Update and revise the projected budget at the end of the planning phase.
Tips and resources
Consider the following:• Ask staff outside the core team to review the budget to ensure it makes
sense.
• Consult with other staff/organizations that may have experience inimplementing EIPs.
• Combine in-kind costs (e.g. computer or equipment) with other relevantinitiatives to maximize the impact of implementation.
• Use technology for communication to minimize cost for face-face meetings.
• Anticipate and project the costs associated with on-going supervision andcoaching (internal and external).
• Anticipate and project the travel costs for staff consultations andcommunication throughout the process.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 15
II. Engaging stakeholders Building staff buy-in and readiness
Who should be involved?
Establishing buy-in for implementation of an EIP is necessary at all levels including practitioners, leaders, and policy-makers. Gaining commitment involves identifying the advantage of the practice (e.g. “What are the benefits for me and my client?” and “How will it make my job easier?”) and demonstrating the availability of the resources, training and leadership required to support and sustain change.
You should consider assessing staff attitudes to improve the effectiveness of implementation efforts. This will increase your success during the planning process and throughout your implementation journey.
What tools and strategies are available?
Various tools are available to understand and assess individual and organizational commitment, buy-in, readiness and attitudes related to implementing a new evidence-informed practice.
• Commitment curve
The commitment curve (Conner & Patterson, 1982) explains why someone is willing (or unwilling) to participate in daily activities. It is closely related to motivation within oneself and interaction with the environment (i.e. extrinsic and intrinsic motivation, reinforcement).
People usually go through three stages of commitment:
• Awareness and understanding: where some people may resist the idea. This is mostly due to lack of information or understanding of the situation.
• Engagement: where people see the benefits and want to get involved, and are more comfortable with the idea of change.
• Commitment: where people own the benefits of change and are actively involved and contribute to the change effort.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 16
You can use this tool to assess organizational commitment throughout the implementation process. This can be done individually or as a group.
7
6
5
4
3
1
2
It’s mine
I’m making it happen
I think it’s a good thing
I understand
I need to know more
I’m aware
6
ORGANIZATIONAL COMMITMENT CURVE Diagram modified from the Patterson-Conner Commitment Curve, Conner & Patterson (1982).
COMMITMENT
ENGAGEMENT
AWARENESS & UNDERSTANDING
I want it to happen
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 17
• Evidence-Based Practice Attitude Scale (EBPAS)
The EBPAS (Aarons, 2005) is a 15-item measure that describes an individual’s global attitude toward the adoption of evidence-based practice (EBP). It was developed by Gregory Aarons as a preliminary exploration of mental health service provider attitudes toward the adoption of EBP in community mental health settings in the U.S.
The EBPAS includes four scales:
Appeal scale: assesses the extent to which a provider would adopt an EBP if it were intuitively appealing, could be used correctly or was being used by a colleague who was happy with it.
Requirement scale: assesses the extent to which a provider would adopt an EBP if it was a requirement of their agency.
Openness scale: assesses the extent to which the provider is generally open to trying new interventions and would be willing to try or use EBPs.
Divergence scale: assesses the extent to which the provider perceives EBPs as not clinically useful and less important than clinical experience.
Sample questions (Yes/No responses):
1. I like to use new types of therapy/interventions to help my clients.
2. Clinical experience is more important than using manualized therapy interventions.
3. I would try a new therapy/intervention even if it were very different from what I am used to doing.
Information here is taken from Aarons, G. A. (2005). Measuring Provider Attitudes Toward Evidence-Based Practice:
Consideration of Organizational Context and Individual Differences. Child and Adolescent Psychiatric Clinics of North America.
14(2): 255–271.
• Implementation Capacity Checklist (ICC)
The Implementation Capacity Checklist can be used as a tool to identify organizational readiness and capacity in implementing EIPs. Please visit the section Organizational implementation capacity in this toolkit for more information.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 18
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
A comparison of the Implementation
Capacity Checklist, the
Organizational Readiness Checklist
and the Readiness for Organizational
Learning and Evaluation Instrument
Download example
• Organizational Readiness to Change Scale (ORC)
The Organizational Readiness for Change (ORC) assesses an agency’s readiness for change (Lehman, Greener & Simpson, 2002). The ORC scales are good indicators of overall strengths and weaknesses within the areas assessed and are useful for highlighting potential areas for improvement.
The ORC includes 18 scales from four major domains:
1. Motivational readiness (three areas)
2. Institutional resources (five areas)
3. Staff attributes (four areas)
4. Organizational climate (six areas)
Information here is taken from the Lehman, E., K.W.; Greener, J.M.; and
Simpson, D.D (2002). Assessing organizational readiness for change. Journal
of Substance Abuse Treatment, 22, 197-209.
• Readiness for Organizational Learning and Evaluation Instrument(ROLE)
The Readiness for Organizational Learning and Evaluation Instrument (ROLE) assesses the level of readiness of an organization to implement processes and practices in learning and evaluation within the organization (Preskill, H. & Torres, R.T. 1999). The ROLE can be used to determine the level oforganizational learning and its characteristics within an organization. It canalso identify existing strengths and needs in terms of organizational changeand evaluation processes. It can measure staff interest in participating inevaluation and provide an overall direction for improving organizationallearning for both individuals and teams.
The ROLE includes 81 items that are embedded in subscales as well as some additional information including:
1. Culture (27 items)
2. Leadership (12 items)
3. Systems and structures (12 items)
4. Communication (eight items)
5. Teams (11 items)
6. Evaluation (eight items)
7. Additional information (three items)
Information here is taken from the Preskill, H. & Torres, R.T.(1999). Evaluative
Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 19
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Go to TED Talks to access a Simon
Sinek clip on How great leaders
inspire action.
Example from the field
Utilizing EBPAS and sharing results
with staff - Central Toronto Youth
Services
Download example
How do we consolidate information gathered from the readiness tools?
• Identifiying barriers and strengths
Information gathered from readiness tools can help determine organizational strengths and areas to improve. It is important to identify barriers and strengths within your organization that may influence the implementation process. A change initiative can fail if the weight of the barriers is greater than the strengths or benefits. If the weight of barriers is greater than the strengths, it is important to identify actions to shift the balance.
Common examples of barriers include scepticism from staff, long waiting lists, staff shortages and poor communication. Examples of strengths include effective and functional teams, leadership support, financial security and key timing.
Key questions to consider:
• Do people consider something to be a barrier because they do not fullyunderstand it?
• Will better communication change perceptions?
• Can the effect of a barrier be reduced? For example, could you extendthe implementation period to allow people more time to adjust?
• Is an additional strength needed? For example, additional training, newequipment, improved facilities and incentives.
• What aspects of the organizational culture can you build upon?
Tips and resources
Consider the following:• Use one or more of the above tools or strategies to assess
staff readiness in the beginning, middle and the final stages ofimplementation.
• Staff commitment could fluctuate throughout the implementationprocess; you may want to revisit the commitment curve at variouspoints.
• Work with key stakeholders to identify strengths and barriers to theproposed change.
• General tips on dealing with resistance:
• Provide information (the why, what and how) and what itmeans to staff.
• Reflect concerns: acknowledge and validate the person’sopinion.
• Roll with resistance: each person’s perspective is valuable.It may provide insight to the challenges associated with thechange initiative.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 20
Engaging youth and families
What is family engagement?
Family engagement is an active partnership between families and service providers. Family engagement in community child and youth mental health services is increasingly considered best practice for providing quality service and support (Chovil, 2009). Involvement of families in their child’s treatment can be related to successful treatment outcomes (Hoagwood, 2005). As well, engagement interventions implemented during first contact with youth and families boosts service use (McKay et al., 2004).
For service providers, this means listening to what families think, engaging them in two-way communication and involving them as essential allies in decision-making so that their involvement is meaningful and has a purpose. Effective family engagement requires the service provider to develop a relationship-building process focused on listening. Engaging families early in the process of implementing a new evidence informed practice can increase their buy in and support towards the new practice which can help in it’s success. Engagement in the form of information sessions or focus groups early in the exploration of a new practice will allow families to have input into the treatment being considered for their children.
What is youth engagement?
Youth engagement is about empowering all youth as valuable partners in addressing and making decisions about issues that affect them personally and/or they believe to be important. Engaged youth play an active role in program planning and decision-making. They can include current and former clients, or those who express a genuine interest in the issues.
An adult ally is an adult assigned by the organization that supports, advocates for and works alongside youth. Youth are meaningfully engaged when they are involved in activities that they believe to have purpose, when they show commitment to what they are doing and they demonstrate gained knowledge of the activity.
How can youth and family engagement enhance the implementation process?
The inclusion of key external stakeholders such as children, youth and their families is recommended in the implementation process. Below is a summary of comments provided by Ontario child and youth mental health agencies about why children, youth and their families should be viewed as active partners (rather than recipients) in the implementation process:
• Buy-in is better if they are active partners.
• Creating relevant services to meet client needs.
• Clients/parents are the experts on their children’s needs. They will be more motivated to participate if they feel their input has been heard.
• The client’s voice is essential in planning, implementing and evaluating EIPs.
• Client input is essential in helping us make decisions about delivery of services.
• It’s about the clients. Their perspective will be valuable and the client experiences will define our success. We need the continual voice of clients.
• You are not doing something for them, you are doing it with them.
• They have valuable points of view/expertise and can become a voice for us as advocates.
• They help keep process goals clear and real.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 21
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
The Centre has developed a handy
tip sheet on communicating change:
Download tip sheet
Here is one example of Centre’s
work to complete the feedback loop
with stakeholders. This provides an
opportunity for stakeholders to have
their input, indicate the next steps
and create a continuous learning
framework.
Download example
Example from the field
Engaging with community partners -
Children’s Centre Thunder Bay
Download example
Tips and resources
• You may not be able to include all members at once: considerincremental steps.
• You may start with existing committees. For example, a youth advisorycommittee
Visit the Centre’s website for more information on youth and family engagement.
Communicating change to stakeholders
Stakeholders are individuals who have a direct or indirect interest in an organization, such as staff, clients, and community partners. It is essential to include stakeholders in the implementation process and establish ways to communicate the processes and outcomes. This involves identifying who the stakeholders are, sharing the rationale for change, and explaining how they may be impacted. Communication with stakeholders is critical, but is not always easily or effectively done.
Tips and resources
Consider the following:• Consult with clients around language to use.
• Inform via a newsletter.
• Have parent representative(s) on the implementation team.
• Hold an education session with clients/stakeholders on benefits ofimplementation of EIP.
• Have informal discussions with clients.
• Introduce consumer (parent/youth) into EIP committee/sub-committee
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 22
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tips and tricks
The Centre has captured
suggestions from provincial mental
health agencies across Ontario
at the planning phase of EIP
implementation
Download tip sheet
Example from the field
Implementation newsletter –
Woodview Children’s Mental Health
Download example
Maintaining momentum
Maintaining momentum within core teams and organizations is a key ingredient for effective implementation of evidence-informed practice. This requires ongoing communication and dissemination of information to ensure the implementation initiative remains a priority for all staff and ultimately becomes embedded in the organizational culture.
Why is it important to maintain momentum?Having strategies in place to build momentum will help the core team, implementation team, stakeholders and agency staff to stay motivated, energized and engaged throughout the process. Given that full implementation may take two to four years, momentum may fluctuate over that time. By keeping staff up to date with key milestones in the process, you will help to move the implementation forward.
Who should be involved in maintaining momentum? The core team should be thinking about ways to build and sustain momentum within the team and agency from the very beginning of EIP implementation.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 23
III. Managing and leading changeChoosing a team
“Select a team that has the widest range of representation from programs and/or employment status as possible (e.g. front line, senior management)”— Implementation support service (ISS) agency
Teamwork and collaboration are important for change and essential to the implementation process. Success depends on a cohesive and effective team that carries through the implementation from the beginning to the end.
Why is it important to establish a team?A team approach allows the work to be shared and enriches the change process. It can create a common vision to support and sustain the project, and team members act as ambassadors for the practice being implemented. They educate others, bring everyone on board and prepare the organization for change. A team approach will help to identify challenges and opportunities and pool resources to overcome them.
What teams should be established?There are two types of teams that contribute to the implementation process: the core team and the implementation team. The number of representatives on either team depends on the scope of your initiative.
A core team is established at the beginning of the planning phase and oversees the entire implementation process including the development of the implementation plan. An implementation team is assembled once the planning phase is complete. This team assists in carrying out the implementation plan and moving the selected practice or program through the stages of implementation.
Who should be on the core team?A minimum of three people should be included on the core team. These members should represent various areas within your organization including direct service providers, clinicians, senior leadership, middle management, technical support, administrative support, family and youth. These individuals utilize organizational strengths and can help eliminate barriers that may impede the implementation process.
Who should be on the implementation team?The implementation team is formed once the initial exploration and planning phase has been completed. This team is crucial in the execution of the implementation plan. Similar to the core team, members of the implementation team should be representative of the organization. Some core team members may step in and become part of the implementation team, while others may remove themselves as the initial implementation is undertaken.
Individuals may be chosen for the implementation team because of their expertise in implementation, the specific EIP, evaluation, interest, coaching and supervision. Some examples of potential implementation team members include project managers, practitioners, clinical management and leadership, family members and youth, EIP trainers/purveyors and evaluators.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 24
Tips and resources
Consider the following:• Invest time in selecting and building your core and implementation teams.
• Include enough members for diversity of opinion.
• Review existing committees (i.e. best practice, evaluation). Identify opportunities for alignment.
• Identify people you may not typically include (e.g. technical support and human resources personnel).
• Involve members of your organization that are considered champions.
• Focus on building trust and relationships.
• Build on the strengths that each member can bring to the table.
• Communication is key.
• Develop terms of reference for the team.
• For the smaller agencies, seek opportunities to partner within the community.
• Survey core and implementation teams on a regular basis (i.e. six months/one year).
• If your agency has multiple sites, ensure that you have adequate representation from each of your sites.
Sample matrix for selecting core team members in a multi-site agency
Site 1 Site 2 Site 3
Youth ü
Family member ü
Clinical staff/ service provider ü
Management ü
Administrative staff (IT, communications) ü
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 25
Regular team meetings
“Never underestimate the strength of your selected team; they have immense knowledge about your agency and selected program, and can assist in propelling your project forward.” — Implementation support service (ISS) agency
Team meetings are critically important and are effective ways to maintain communication in the implementation process. Depending on the scope of implementation and nature of the practice, the following types of meetings may be used:
• Core team meetings
• Implementation team meetings
• Internal stakeholder meetings
• External stakeholder meetings, including community partners and families
Core team and implementation team meetings
Team meetings can occur on a regular basis with pre-determined dates and agenda items. They can also happen in an informal way that encourages dialogue and open discussion of ideas.
Internal stakeholder meetings
These meetings are opportunities to involve stakeholders within the organization in the change process. They can increase staff awareness and knowledge of the implementation initiative and create a forum that reinforces the learning organization approach.
Ideas include:
• small department/unit meetings
• lunch-and-learn
• road show Q and A hour
• annual staff meetings or retreats
• knowledge exchange and learning events
• on-line staff chat room
• ask an expert
• monthly learning circle
These types of meetings also refer to informal opportunities such as show and tell, water cooler chats, or one-on-one meetings with supervisors.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 26
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
Sample implementation team
meeting agenda
Download example
Example from the field
A sample terms of reference: CBT
community of practice - Pathways
Children and Youth
Download example
External stakeholder meetings, including community partners and families
These meetings are opportunities to involve stakeholders outside the organization in the implementation process. Ideas include:
• town hall meetings
• community of practice
• specific committees and collaborative initiatives
Tips and resources
Consider the following:• In the planning phase, core teams meet more frequently. Most
organizations meet on average 1.5 hours every two weeks in the first sixmonths.
• Identify:
• who will attend the regular meetings.
• the frequency of the meetings.
• the objectives of the meetings.
• Determine meeting logistics in advance (i.e. dates and time, location, themeeting chair, meeting agenda, follow-up plan and note taking).
• Establish ground rules, roles and responsibilities.
• Establish meeting agendas and preview documentation prior to themeetings.
• Discover ways to maintain effective meetings and ongoingcommunication.
• Create opportunities to celebrate small wins.
• Share successful stories during the meetings.
• Evaluate and monitor the effectiveness of the team meetings on aregular basis.
• For organizations with multiple sites and locations:
• find a central location for easy access.
• take advantage of video capacity to cut down cost and traveltime (e.g. GoToMeeting, OTN).
• Keep in mind that the initial phase of implementation planning mayrequire more frequent meetings to build teams and establish routines.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 27
Assessing the evidence
What does it mean to assess the evidence?Before introducing a new practice into an organization, it is critical to go through a process of assessing and selecting an evidence-informed practice (EIP) that will be considered. From the initial needs assessment, the team may come to conclusion that: 1) there is a gap between current services and clients’ needs; 2) a more comprehensive approach is required to apply evidence into practice and 3) without considering organizational context, simply bringing in an off-the-shelf manualized intervention doesn’t guarantee success in implementation.
Evidence-informed practice (EIP) embraces the three sources of evidence to bring theory into practice. This includes research evidence, practitioner experience and judgment and child, youth, family and caregiver preference. When assessing the evidence and selecting an EIP to implement, it is essential to be continuously gathering, sharing and using evidence from these three sources.
Here are some key questions for the core team to consider when assessing the evidence: • How will information be generated and incorporated into the overall program practice?
• How do we capture child, youth, family and caregiver preferences into our current practices?
• Do we have inputs from the child, youth family and caregiver throughout the continuum of the client engagement?
• How do we embrace practitioner experience and judgment in our current practices?
• How do we loop back the information to these three sources?
Child, youth, family and caregiver preferences
Practitioner experience and judgement
Evidence-informed practice
Research evidence
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 28
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
Literature review guide
Download example
Who should assess the evidence? The core team, with the input of senior leadership and other identified stakeholders, should examine the evidence. Some organizations use the help of students to complete this task, whereas some may utilize existing committees within the organization to assist with the information.
When and why should the evidence be assessed? The evidence should be assessed during the early stages of implementation to ensure the selected EIP addresses the identified needs.
How should evidence be assessed?
1. Assessing research evidenceCore and implementation teams should reference current literature and researchbefore and during the implementation of programs and practices. This willensure decision-makers and frontline staff are using the most current and up-to-date knowledge to inform their work. There are a number of approaches thatcould be used to look at research such as completing a literature review andconducting a critical appraisal.
Conducting a literature reviewWhen conducting a literature review, it typically follows four key stages:
1. Search the literature.2. Assess the findings.3. Summarize the findings.4. Write the review.
Conducting a critical appraisalCritical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, its value and its relevance in a particular context (Burls, 2009). Although the process of conducting a critical appraisal may be similar to that of a literature review, it usually follows a more systematic and thorough examination of research evidence; as well, the results of the appraisal are analyzed and documented in an organized matter. For example, drawing comparison of studies, categorizing or utilizing rating scales to assess against the criteria.
The National Collaborating Centre for Methods and Tools (NCCMT) has a critical appraisal module that demonstrates how to assess the quality of an intervention (National Collaborating Centre for Methods and Tools, 2010). Click to see NCCMT website.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 29
Tips and resources
Consider the following:• Organizations that have a partnership with post-secondary institutions may have student placement programs that
include conducting a literature review as part of their placement.
• It is important to make researching the evidence a shared task among team members. This will build capacity within the organization and increase access to research information.
• Safeguards Training has some information that is helpful for the literature search.
2. Assessing evidence from child, youth, family and caregiver preferenceThe second source of evidence to consider is the preference of children, youth and their families and caregivers - the recipients of your services. By involving youth and their families as partners in treatment planning, you improve your services by ensuring their needs are met and creating solutions that encourage their commitment. Client characteristics may also impact their preferences and responses to the interventions they received, factors such as social and culture differences of these preferences would require attention during the EIP selection. Ultimately, the reason for implementing a new EIP is to improve services and adapt to these diverse needs.
Tips and resources
Consider the following:• Discuss how to include client feedback and preferences, and discover ways to facilitate the involvement of their input
from the planning stage.
• Use pre-surveys to gather client perspectives (preference) before considering a new practice. Conduct a post survey after implementing a new EIP.
• Look for opportunities to communicate how the selected EIP will meet the needs of the client.
• Draw from examples of agencies who are using the proposed EIP and consider the outcomes they have achieved.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 30
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Examples from the field
A staff survey for CBT practice -
Pathways for Children and Youth
Download example
A staff survey on DBT - Woodview
Children’s Mental Health
Download example
3. Assessing evidence from practitioner experience and judgmentThe third source of evidence to consider is practitioner experience andjudgment. Establishing evaluation practices that capture the collectivepractitioner clinical experience and judgment can increase implementationcapacity and fidelity. In addition, this can support strategies to decreasepractitioner’s feelings of isolation.
Tips and resources
Consider the following:• This is an opportunity to engage all staff in the process of
implementation. Organizations can develop surveys to solicit feedbackfrom staff and identify emerging trends through staff feedback.
• Create opportunities for practitioners to ask questions, share experienceor express concerns for the proposed practices/intervention.For example, host an open house, lunch & learn or other on-linecommunication tool to encourage feedback.
• Utilize existing communication strategies to promote feedback loopsand share data results. Present the information through team meetings,year-end staff meetings or other forums.
• Create opportunities to draw upon the experience of champions.
• Tap into existing committees or community-of-practice groups to gathercurrent knowledge of the subject.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 31
Selecting an evidence-informed practice
The process of selecting an EIP can bring about rich conversations that include diverse perspectives. By including the voices of multiple stakeholders, this will ensure the best EIP is selected and will increase staff commitment.
Selecting an evidence-informed practice is a multi-step process with four main steps:1. Select what approach you will take to determine the appropriate EIP. For example, using a critical appraisal or literature
review to gather evidence.2. Identify resources that can help with successful implementation.3. Identify appropriate implementation sites.4. Identify key features of the implementation that must be closely adhered to and monitored.
Similar to the implementation capacity checklist, organizations may also consider other tools, such as a hexagon tool.
An example: Hexagon toolThe Hexagon tool uses six factors to evaluate new and existing interventions and may assist organizations in systematically identifying the right EIP to implement.
Needs: how well the programs or practice might meet identified needsFit: overall fit with current initiatives, priorities, structures and supportResource availability: for training, staffing technology supports, curricula, data systems and administrationEvidence: indicating the outcomes that might be expected if the program or practices are implemented wellReadiness for replication: including expert assistance, number of replications accomplished, exemplars available for observation, and how well the program is operationalizedCapacity to implement: as intended and to sustain and improve implementation over time
Tips and resources
• A thorough process focused on the proposed program or practice will help your team have a productive discussion related to the six areas listed above.
• The information gathered can be used to communicate and egage all staff within the organization.
For more information and tips on how to use the Hexagon Tool, go to the Active Implementation Hub website.
Information from Blasé, K., Kiser, L., and Van Dyke, M. (2013). The Hexagon Tool: Exploring Context. Chapel Hill, NC: National Imple-
mentation Research Network, FPG Child Development Institute, University of North Carolina at Chapel Hill.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 32
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
A template designed by the Centre
to help you identify key factors within
your organization.
Download template
Creating an implementation plan
Creating an implementation plan is essential in order to address potential implementation gaps. Effective implementation planning and execution takes an average of three years, provided there is fidelity to the plan as well as to the EIP intervention (Fixsen, Blasé, Timbers and Wolf, 2007).
An implementation plan lets you anticipate challenges (e.g. cost, commitment, responsibility), create a common understanding among staff (e.g. regardless of level of involvement, everyone understands the goal of the program), and tie together all of the steps involved in moving from early planning to sustaining a new practice.
Who should be involved in creating the plan? The core team should be involved in creating the plan. The implementation team may assist with adjusting the plan along the way.
When and how should the plan be used? An implementation plan should be created early on in the implementation process to provide consistency in communication and interpretation for all staff. It is a living document that requires ongoing adjustments and revisions as necessary.
What should be considered when creating an implementation plan?Before creating a plan, it’s important to use a framework that provides an organizing structure that helps make sense of complicated processes. It provides a visual so you can more easily organize and monitor where you are at in terms of the implementation process framework. The following information can be used throughout implementation:
1. Key factors of implementationThere are several frameworks that identify key factors of implementation.At the Centre, we base our description of key factors on the ConsolidatedFramework for Implementation Research developed by Damschroder et al.(2009). This framework outlines five domains affecting the implementationprocess:
• Inner setting
• Outer setting
• Intervention characteristics
• Characteristics of individuals
• Process of implementation
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 33
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
A template designed by the Centre
to help you identify implementation
drivers within your organization.
Download example
A sample implementation plan
template.
Download example
Examples from the field
Implementation plan from Youth
Services Bureau: Trauma Focused-
Cognitive Behaviour Therapy (TF-
CBT).
Download example
Implementation framework from
Woodview Children’s Mental Health:
Dialectic Behaviour Therapy (DBT).
Download example
2. Implementation drivers
Implementation drivers are core features that contribute to successful implementation initatives. The drivers are integrated, occurring simultaneously in order to maximize their impact on practitioner behaviour change. The drivers also compensate for one another whereby a weakness in one component can be offset by strengths in another.
There are three types of implementation drivers (adapted from The National Implementation Research Network, 2013):
• Competency drivers: They help develop, improve and sustain anagency’s ability to implement an intervention. They include staff selection,training, ongoing consultation and coaching as well as staff performanceassessment.
• Organizational drivers: These are the mechanisms that create andsustain favourable organizational and system environments for effectiveservice. They include a data system to support decision making andprogram evaluation, facilitative administration supports and systemsinterventions.
• Leadership drivers: Successful implementation requires thecommitment, involvement and accountability of leaders in theimplementation process. Leaders drive the use of outcome data to informdecision making and keep staff motivated.
Find more information on implementation factors and implementation drivers on the Active Implementation Hub website.
Tips and resources
Consider the following:• In addition to the Centre framework, consider the NIRN implementation
framework for further resources and tips.
• Use concrete terms and graphs to visualize implementation activitiesand timelines.
• Use the organizational tools that staff are familiar with to layout thetimeline.
• Some implementation plans may require a more detailed layout andadditional actions to carry out activities.
• Evaluation should be considered part of the implementation plan.
• When deciding on the timeline for the implementation initiative,be mindful of other concurrent initiatives and priorities within theorganization.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 34
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
Questions to ask your purveyor -
National Registry of Evidence-based
Programs and Practices
Download Example
Selecting a purveyor
What is a purveyor? A purveyor is an individual or group of individuals representing a program or practice. These individuals work to develop and implement that program or practice with fidelity and good effect (Fixsen & Blase, 2008). A purveyor, often referred to as a trainer, has knowledge and expertise in a specific area and can be someone from within or outside the organization.
When to select a purveyor?After you have selected an EIP, selecting an appropriate purveyor for the EIP intervention is the next step. A purveyor should be selected before confirming the implementation plan and training dates. The purveyor may provide input to the implementation plan during the planning phase and assist with choosing training dates that are suitable for your organization.
Who should be involved in selecting a purveyor? The core team along with other key staff is involved in selecting the right purveyor. It may be helpful to ask other organizations that have implemented the same EIP for recommendations when choosing purveyors.
Tips and resources
Consider the following:• The Centre of Excellence has an online resource hub where
organizations can access useful information on evidence-informedpractices.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 35
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Tools and reference
The following are a few useful
templates when completing an
evaluation.
Sample logic model
Sample evaluation framework
Sample evaluation questions
Sample fidelity resources
Ongoing evaluation and monitoring
Program evaluation is the backbone of providing evidence-informed services. It helps you assess what you’ve done, determine where you’re at and plot a course of action for the future. Monitoring your outcomes along the way gives you a real-time snapshot of how far you’ve come. Evaluation allows you to assess how well the program has been implemented in order to meet its objectives. It also shows if and how your clients are changing as a result of the intervention.
Why is evaluation important? Evaluation is a valuable strategy for measuring the success of an EIP, not only in terms of assessing child and youth mental health outcomes, but for measuring the success of the implementation process itself. An effective intervention may fail to provide expected positive outcomes due to a lack of fit within your context, or due to a lack of fidelity to the core intervention components necessary to realize the anticipated outcomes. By evaluating both the process of implementation and the outcomes for your clients, you can ensure you capture the most accurate depiction of your program.
Who should be involved in evaluation? The core team along with any other staff with knowledge or experience in evaluation or research should take part in both the development and ongoing activities of evaluating the program. While many organizations have an evaluation person or team, even those who do not can build internal capacity in this area. The Centre has a number of supports and tools that can assist in the building of such capacity. It is also recommended that organizations consider the inclusion of a student to support with a number of activities including the data collection, analyses and interpretation of results.
What should be evaluated?Evaluation activities should be focused on both the implementation process (including fidelity) as well as the child and youth mental health outcomes of the program/intervention being implemented.
A process evaluation is an important first step as it will describe how the program was developed, including the important factors of implementation as well as how it is operating once implementation is complete. It asks the “what” of the program. For example, typical process evaluation questions would include: What staff competencies were anticipated? Do program staff have those competencies? What are the characteristics of the intended clients to treat? Did you reach the intended target population?
An outcome evaluation assesses the effects that taking part in the program has on the clients you serve, or what the impact was on their mental health.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 36
HELPFUL TOOLS
DOWNLOAD
Use the links provided to access
and download examples and
templates that could prove helpful
throughout your process.
Sample from the field
Engaging staff in understanding a
logic model - Central Toronto Youth
Services
Download example
Tools and reference
A chart suggesting activities and
timelines for the planning phase.
Download example
It asks the “so what” of the program. This type of evaluation is important and should be conducted once your EIP has been fully implemented in order to provide important information about how the clients are improving or not based on their participation in the intervention.
Both implementation and outcome evaluations should be seen as working together to create a more complete picture of what is going on in your program.
When should we evaluate?Evaluation should begin from the early stages of implementation planning and continue through to the full implementation stage. In the early stages, the evaluation should focus more on process questions to assess the success of implementation as it is recommended that a program be fully implemented for at least six months before beginning an evaluation of the outcomes.
Tips and resources
Consider the following:• If you are implementing an EIP that is in a school, consider alignment
of training/ implementation/ evaluation with school calendars and anyrequired ethics reviews.
• Begin thinking about evaluation early on in the implementation process.
• Create an evaluation plan and incorporate it into your implementationplan.
• A program logic model can help to understand the overall structure of aprogram when you are implementing the new practice/intervention.
• Identify key evaluation questions, indicators and outcomes.
• Utilize an outcome measurement database to help inform yourevaluation activities.
The Centre has created an Evaluation Toolkit. Click here for more information.
Implementation activities and timeline
The activities and timeline for implementation in the planning phase will vary depending on the program itself (e.g. during or frequency of program activities), staff capacity (e.g. availability, knowledge and skills on implementation) and organizational resources (e.g. access to research literature, information technology resources). The chart in the helpful tools section presents activities and timelines as milestones that the core teams may consider.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 37
Preparing for the next phasesof implementation
Doing phase
Things to consider when preparing for phase two of implementation:
• evaluation framework and measures
• ongoing communication with stakeholders
• ongoing regular core team and implementation team meetings
• intervention staff selection
• training on EIP
• trainer/coach follow-up (consultation, practice, and model)
• implement/adjust supervision model
• implement/adjust coaching system
• ongoing leadership support
• process and system alignment (e.g. changing clinical pathways and referral systems)
• initial process and outcome evaluation (pilot)
• link with experts and communities of practice
• fine tune budget for phase three of implementing EIP(s) (sustaining phase)
• develop a full implementation plan
• update the implementation capacity checklist
• develop a sustainability plan
• develop a knowledge exchange plan
Sustaining phase
Things to consider when preparing for phase three of implementation:
• ongoing communication with stakeholders (e.g. status reports)
• ongoing regular core team and implementation team meetings
• execute full implementation plan (scale up or refine)
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 38
• develop/implement knowledge exchange and dissemination plan
• ongoing linkage with experts and communities of practice
• implement a sustainability plan
• align supervision model
• align coaching system
• ongoing leadership support
• process and system alignment
• ongoing process and outcome evaluation
• program reports
• mentor other agencies in implementing EIPs
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 39
General tips and strategies on implementing EIPsFinally, here are some general tips and strategies suggested by agencies that reflect their experience in the implementation of EIPs during the planning phase:
“What factors contributed to your success in terms of activities in the planning phase of your implementation journey?”
• A sense that the middle management group is a key facet in the sustainability of an EIP over time
• The core team believed that training key staff early on would both motivate staff and provide useful and cautionary information that would be helpful in avoiding future problem areas
• The use of a collaborative approach / teamwork
• Input from staff who occupy different roles in the agency - all of our thoughts and ideas were heard and considered
• Provide a respectful, safe atmosphere - very understanding team
• Stability and guidance from a dedicated and committed leader
• Existing culture where team members can safely take a risk and challenge other team members
• Comprehensive, accessible and useful tools, training and resources from the Centre
• Having a valid and credible implementation model to follow was KEY
As you go through the planning phase of the implementation, keep the following things in mind:
• Recognize that implementation takes time.
• Shrink the change into manageable steps.
• Use a strength-based approach.
• Become a learning organization.
• Maintain the senior leadership commitment and buy-in throughout the entire process.
• Ensure readiness for change and implementation - create a sense of urgency.
• Celebrate the short-term wins.
• Communicate, communicate, communicate.
• Build a solid implementation plan.
• Track every step of the implementation - monitor the change.
• Embrace teamwork and collaboration.
• Be flexible.
• Have an open communication policy on the core team.
• Share the workload.
• Welcome new and energetic staff to the team.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 40
Additional resourcesDisclaimer
The Centre’s toolkit contains links to sites developed and managed by others. The Centre assumes no responsibility or liability for material on
linked sites and recognizes the material may not be available in both official languages.
Centre implementation resources
Evaluation toolkits
Measures database
Child/youth and family engagement in evaluation
Harvard Family Research Project
ReferencesAarons G. A., Glisson, C., Hoagwood, K., Kelleher, K., Landsverk, J., Cafri, G. (2010). Psychometric properties and U.S. national norms of the Evidence-Based Practice Attitude scale (EBPAS). Psychological Assessment, 22, 356–365.
Aarons, G. A. (2005). Measuring Provider Attitudes Toward Evidence-Based Practice: Consideration of Organizational Context and Individual Differences. Child and Adolescent Psychiatric Clinics of North America. 14(2): 255–271.
Austin, J., M., Claassen, J. (2008). Implementing Evidence-Based Practice in Human Service Organizations: Preliminary Lessons from the Frontlines. Journal of Evidence-Based Social Work. 5(1/2), 271-293.
Barwick, M. A. (2011). Checklist to assess organizational readiness (CARI) for EIP. Hospital for Sick Children University of Toronto. http://melaniebarwick.com/wp-content/uploads/2019/01/CARI-Checklist_for_Assessing_Readiness_for_Implementation-BARWICK.pdf
Burls, A. (2009). What is critical appraisal? Retrieved July 10, 2012, from www.whatisseries.co.uk
Chovil, N. (2009). Engaging Families in Child and Youth Metal Health: A Review of Best, Emerging, and Promising Practices. Retrieved July 10, 2012, from http://www.forcesociety.com/sites/default/files/Engaging%20Families%20in%20Child%20%26%20Youth%20Mental%20Health.pdf.
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 41
Conner, D. R., & Patterson, R. W. (1982). Building commitment to organizational change. Training and Development Journal, 36, 18-30.
Damschroder, L., Aron, D., Keith, R., Kirsh, S., Alexander, J., Lowery, J.(2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science. 4(1),50.
Fixsen, D.L., Blasé, K, Timbers, G.D. and Wolf, M.M. (2007).In Search of Program Implementation: 792 Replications of the Teaching-Family Model. The Behavior Analyst Today, 8(1), 96-114.
Fixsen, D. L., & Blase, K. A. (2008). Drivers framework. Chapel Hill, NC: The National Implementation Research Network, Frank Porter graham Child Development Institute, University of North Carolina.
Fixsen, D. L., Naoom, S.F., Blase, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation research. A Synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHL Publication #231).
Godley, S. H., Garner, B. R., Smith, J. E., Meyers, R. J., & Godley, M. D. (2011). A large-scale dissemination and implementation model for evidence-based treatment and continuing care. Clinical Psychology: Science and Practice, 18, 67-83.
Hoagwood, K. E. (2005). Family-based services in children’s mental health: A research review and synthesis. Journal of Child Psychology and Psychiatry, 46, 690-713.
Julian, D.A., Ross, M., & Partridge, C. (2008). Challenges in supporting community implementation of science-based programs: A critical review of local partnerships for success plans. American Journal of Community Psychology, 41(3), 351-360.
Lehman, E., K., W., Greener, M., J., Simpson, D. D.(2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment. 22(4), 197-209.
Kilbourne, A.M., Neumann, M.S., Pincus, H.A., Bauer, M.S., & Stall, R. (2007). Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implementation Science, 2:42.
McKay, M. M., Hibbert, R. Hoagwood, K. E., Rodriguez, J., Murray, L., Legerski, J., et al. (2004). Integrating evidence-based engagement interventions into ‘‘real world’’ child mental health settings. Brief Treatment and Crisis Intervention, 4, 177-186.
Moore, P., Cunning, S., & Roundwaithe, J. (November 2010). Implementation Evidence-Informed Practice – Kinark’s Clinical Transformation Journey. Presentation paper at Children’s Mental Health Ontario Conference, Toronto, ON.
National Collaborating Centre for Methods and Tools. (2010). Critical Appraisal of Intervention Studies. Retrieved May 13, 2005, from http://www.nccmt.ca/en/modules/cais/
Preskill, H. & Torres, R.T.(1999). Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.
Sackett, D., L., Straus, S., E., Richardson, W., S., Rosenberg, W., & Haynes, R., B. (2000). Evidence based medicine: How to practice and teach EBM. (2nd ed.). London: Churchill Livingstone.
The National Implementation Research Network (2013). Implementation Drivers. Retrieved June 19th, 2013 from http://nirn.fpg.unc.edu/learn-implementation/implementation-drivers
Learning Collaboration Leadership
www.excellenceforchildandyouth.ca Implementing evidence-informed practice: A practical toolkit 42
AcknowledgementsThis toolkit has been developed with the support and expertise of
many people. The contributions of staff from the Centre as well as the
valuable perspectives of the many reviewers from our networks are
greatly appreciated.
Ontario Centre of Excellence for Child and Youth Mental Health 2013 Implementation toolkit. Ottawa, Ontario. www.excellenceforchildandyouth.ca
© Copyright: This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. To view a copy of this license, visit http://creativecommons.org or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.
Centre implementation support team
Ann BarberLaura ConroySandra Huang Del FrariJana Kocourek Mark MacAulayVanessa Pleasance
Internal review team
Charles CarterEvangeline DansecoNatasha McBrearty
External review team
Amber McCartLorraine JeffreyNathalie JohnstonRenee ParsonsKatherine Sdao-Jarvie
Child and youth mental health organizations
Aisling Discoveries Children and Family CentreCentral Toronto Youth ServicesChild and Community ResourcesChildren’s Centre Thunder BayPathways for Children and YouthWoodview Children’s Mental HealthYouth Services Bureau
Suggested citationOntario Centre of Excellence for Child and Youth Mental Health (2013). Implementing evidence-informed practice: A practical toolkit. Ottawa, Ontario.