ATTC Network Evaluation National Evaluation of the ATTC Network: Findings and Recommendations...

ATTC NetworkEvaluation

National Evaluation of the ATTC Network:Findings and Recommendations

Presentation to ATTC Directors November 4, 2010

Richard Finkbiner, Ph.D.Project Director

Megan Cummings, B.A.

Margaret Gwaltney, M.B.A.Deputy Project Director

Cori Sheedy, M.A.

Roy M. Gabriel, Ph.D.Principal Investigator

Jeffrey R.W. Knudsen, M.A.


Overview of Today’s Presentation

Evaluation Purpose and Approach (review)

Evaluation Design and Data Collection (review)

Final Reporting Approach 7 Findings from National ATTC

Evaluation 5 Recommendations

2


Some History: 2005–2010

Planning the evaluation: 2005–2007 Concept and CSAT intent for national evaluation of ATTC Network

launched at Nov. 2005 ATTC Directors’ Meeting (Portland). Abt and RMC staff continued to engage ATTC Directors and

stakeholders in evaluation planning, logic model development, and instrument development through fall 2007.

OMB Instrument Package submitted fall 2007. MANILA, Abt, and RMC competed for and won CSAT award to

implement the evaluation, summer/fall 2007. Implementing the evaluation: 2007–2010

ATTC site visits, key informant interviews, Regional Advisory Board Surveys began winter 2008 and concluded spring 2009.

Customer Satisfaction and Benefit Survey began winter 2009 and concluded spring 2010.

Change in Practice Study began winter 2008 and concluded spring 2010.

Participation in every ATTC Directors’ meeting during this period.

3


CSAT Goals for Evaluation

Identify the successes of technology transfer efforts and build upon them in the future.

Share lessons learned for the enhancement of all regions’ activities.

Distinguish between region-specific and more cross-regional processes and outcomes

4


Review of Evaluation Design

8 evaluation questions 3 interconnected studies

Planning, Partnering and Service Delivery Study

Customer Satisfaction and Benefit Study Change in Practice (CIP) Studies

12 data collection activities 3 secondary data (GPRA, existing reports,

etc.) 4 tied to CIP studies

5

Logic Model for the National Evaluation of the ATTC Network

Resources

Funder Goals & Mandates

Regional/ State Needs

Inputs

State/ Regional

Multi-Region

National

ScopeTraining

Technical Assistance

Meetings/ Conferences

Product Development

Research Dissemination

Partnership Development

Academic Programming

Activities

Mediating Factors include: geographic expanse/density of service population; culture/ diversity of target population; longevity of ATTC contractor; longevity of SSA directors; state/regional addiction treatment infrastructure; organizational readiness

Awareness Raising

Skill Building

Change in Practice

Objectives

Program Outputs

Immediate Outcomes

Intermediate Outcomes

Long-Term Outcomes

Outcomes

Me

dia

ting

F

ac

tors


6


Planning, Partnering & Service Delivery Study: Data Collection Activities (Review)

Site visits to each regional ATTC and the ATTC National Office, consisting of interviews and focus groups with ATTC staff and consultants

ATTC Regional Advisory Board Survey Key informant telephone interviews with ATTC

stakeholders within each region Analyses of data from ATTC Event and Activity

Reporting Database

7


Customer Satisfaction and Benefit Study: Data Collection Activities (Review)

Primary data collection activity: Customer Satisfaction and Benefit Survey (CSBS)

Secondary data: Integrating findings from ATTC regional site visits, key informant interviews, Regional Advisory Board Surveys, and GPRA.

8


Critical Action Surveys Clinical Supervision (CS) Motivational Interviewing (MI) Treatment Planning MATRS (Tx PM)

Success Case Interviews Audio Tape Reviews (MI only) Organizational Readiness for Change

Survey (Tx PM only)

Change in Practice Study: Data Collection Activities (Review)

9


Completed Data Collection Activities

15 site visits—all ATTCs and National Office 14 Regional Advisory Board Surveys (N = 198,

56% response rate) N = 162 Key Informant Interviews (86%)

Includes 49 SSA/rep interviews (96%) GPRA analysis

N = 102,000 participant records from 2005–2007 N = 140,000 participant records from 2008–2010

ATTC Event and Activity Reporting Database N = 4,787 event records, 2008–2010

10


Completed Data Collection Activities (cont.)

N = 1,954 Customer Satisfaction and Benefit Surveys

N = 388 Critical Action Surveys (76% response rate)

N = 67 Success Case Interviews (93% of target) N = 9 MI Audiotape Surveys (12% of target) N = 16 Organizational Readiness for Change

Surveys

11


Final Reporting Approach

Final report organized around 7 main evaluation findings (not presented by sub-study, evaluation question, or data collection method) and 5 recommendations Building upon the interconnected nature of the data

collections in each of the 3 studies A synthesis of results that increases confidence in the

convergent validity and robustness of these findings for program and policy development

3 sub-study reports to follow (Jan. 2011) Webinars for ATTCs on these more specific results

Presentations Journal articles

12


Today’s Presentation

Finding 1: Richard Finkbiner, MANILA

Findings 2 and 3: Cori Sheedy, Abt Findings 4 and 5: Roy Gabriel,

RMC Findings 6 and 7: Jeff Knudsen,

RMC Recommendations: Roy, RMC

13


Suggested Ground Rules for Questions and Discussion in Today’s Presentation

During presentation of each finding, restrict your questions to those of clarification of our data sources, interpretation, etc.

Reserve offering more reflective observations and implications until after all 7 findings have been presented.

Discussion of the 5 recommendations (during working lunch) can be more dynamic and spontaneous.

14


FINDING 1

The vast majority of ATTC customers and stakeholders perceive the benefits they receive from ATTC services to be extremely valuable. Respondents who most often participated in ATTC services reported the greatest benefits.

High levels of satisfaction and perceived benefits of ATTC products and services

CSBS sample heterogeneous and representative of the addictions treatment workforce

Frequency of participation in a variety of ATTC events strongly related to perceived benefits

Motivation to participate in ATTC events Participation in and satisfaction with specific types of

ATTC services varies by role

15


CSBS Survey Response Rates

Raw Response Rate CSBS 48.6 %

“Trimmed” Response Rate 54.6 %

Range Across Regions 33.6 %--67.7 %

Median Response Rate 58.3 %

Source: CSBS (n = 1,954)

Customer Satisfaction and Benefit Survey: Response Rates

16


Participation in Different Types of ATTC Activities

17


Beyond Satisfaction: Perceived Benefits of ATTC Technology Transfer

18


Perceived Benefits by Varying “Dosage”of Participation in ATTC Events

19


GroupPercent CSBS

SampleOverall ATTCSatisfaction

Overall Increase in Knowledge

Overall Improved My Job Skills

High external motivation 1.8% 74% 87% 87% Some external motivation 7.3% 88% 91% 87% Neutral 42.7% 89% 94% 87% Some internal motivation 19.3% 94% 95% 89% High internal motivation 29.0% 97% 98% 95% Source: CSBS (n = 1,954)

Perceived Benefits by Participants’ Motivation for Attending ATTC Events

20


FINDING 2

ATTCs succeed in balancing the needs, interests, and requirements of diverse stakeholders within available resources. The variety of stakeholders creates a strategic tension, which ATTCs address by delivering services that have a broad reach and are less resource-intensive.

Leveraging funding resources Relationship-building across the ATTC Network Mechanisms for identifying workforce priorities and

stakeholder needs and interests Assessing and evaluating needs to strategically plan

services Partnering in service delivery Bundling of services Consequences of ATTCs meeting the diverse array of needs

and requirements

21


Leveraging Funding Resources

The majority of ATTCs have actively leveraged SAMHSA/CSAT funding, and in some cases, this additional funding is considerable. One ATTC has nearly doubled their SAMHSA/CSAT funding,

with an operating budget of more than $1,000,000.—Site Visit

The ATTC [grant] is the springboard to receive these other funds.—ATTC Director

Additional funding includes supplemental SAMHSA/CSAT funding and funding from NIDA, States, local organizations, and private foundations.

ATTCs combine funding streams (ATTC funds, partner resources, program income, etc.) to provide services. A little less than one third (27%) of ATTC activities had 2 or

more funding sources. —E&A Database

22


Relationship-Building Across the Network

ATTCs work with and across a broad array of stakeholders, spending considerable time and effort identifying partners and building and maintaining relationships.

During interviews, stakeholders identified the ATTCs’ efforts to collaborate with them and to maintain partnerships as a key factor to their satisfaction with the ATTCs.

ATTCs also view the extensive effort they devote to building and maintaining relationships as crucial to determining the most appropriate services for the region.

The methods used to build relationships vary and depend on the partner, preexisting relationships or personal connections with that partner, the partner’s location, and the expected outcome of the partnership.

23


Mechanisms for Identifying Priorities and Assessing and Evaluating Needs

ATTC mechanisms for identifying priorities and needs: ATTCs continually seek input from customers about regional

priorities and needs via formal and informal ways. Mechanisms typically include Regional Advisory and other

boards, regularly scheduled and ad-hoc communications and meetings, and participation in external committees and boards.

Assessing and evaluating needs to strategically plan services: Processes include holding annual ATTC staff retreats to discuss

priorities and develop the ATTC’s work plan, conducting and analyzing needs assessments, working directly with the SSAs to prioritize the State’s needs, and utilizing decision-making tools.

In responding to different needs, often a single activity will try to meet the needs of several different stakeholder groups.

Notably absent in the planning of services is a discussion of the expected outcomes of the activities planned.

24


Partnering in Service Delivery

ATTC partnerships serve as a key platform for effective service delivery and are an effort to extend the ATTCs’ reach to more participants and to obtain additional support and funding for their activities.

ATTCs partner with regional, State, and local organizations and agencies; national and Federal organizations; and other ATTCs, among others, to deliver services. Partners bring different resources and expertise to the relationship. 84% of events between January 1, 2008, and June 30, 2010,

(4,031 events) were delivered in partnership with other organizations and 23% of events had multiple partners. Most frequently cited partners included SSAs (41% of events), another ATTC (24%), State/regional organizations (23%), and community treatment programs (10%). —E&A Database

25


Bundling of Services

“Bundling” of services occurs when a series of ATTC activities is coordinated at regional and State levels or when services are delivered as a series on a specific topic. More than one third (38%) of ATTC events were delivered as part of a

series and in collaboration with other services. —E&A Database

Bundling of services has been shown to facilitate successful implementation of EBPs and organizational change (Fixsen & Blase, 2009)1.

During interviews stakeholders said there is a need for the ATTC to provide services aimed at changing practice and to increase follow-up activities, such as technical

assistance provided over a period of time.

1Fixsen, D.L. & Blasé, K.A. (2009, January). Implementation: The missing link between research and practice. NIRN Implementation

Brief No. 1. Chapel Hill: The University of North Carolina, FPG, NIRN.

26


Consequences of Meeting Diverse Array of Needs and Requirements

ATTCs are successful at balancing the needs of their stakeholders and providing training and technology transfer activities to meet identified needs within limited funding.

However, ATTCs encounter challenges in their efforts, e.g., Having to build and maintain relationships with many

stakeholders Delivering services to address diverse workforce needs Coping and operating with limited resources to meet

identified needs and other Federal requirements The result is that a disproportionate emphasis is placed

on services and activities that fall within the earlier phases of the technology transfer continuum with fewer activities aimed at changing treatment practice.

27


FINDING 3

The ATTCs have adopted a State-specific approach to service delivery, built upon the strong relationships they have established with SSAs. Additional demands on this relationship will likely arise from recent national policy developments and challenging economic conditions.

Relationship-building with SSAs Longevity, expertise, and shared values of ATTC and SSA

staff Policy and economic conditions affecting states and the

addictions treatment field

28


Relationship-Building With SSAs

ATTC directors and staff invest significant time and employ multiple strategies to initiate and nurture their relationships with SSAs. These strategies include: Including SSA directors on ATTC Regional Advisory Boards.

Nearly three fourths (73.3%) of SSAs or their representatives agree or strongly agree that their input is sufficiently considered in this process, virtually the same as that of all Regional Advisory Board members (73.4%)—Regional Advisory Board Survey

Scheduling regular calls/meetings between ATTC staff and SSA directors.

Participating on outside committees or organizations, where SSA directors, staff, and other leaders in the addictions field also are represented (e.g., State provider associations or SSA committees).

Having ad hoc meetings with SSA directors at conferences and other regional or national meetings.

29


Longevity, Expertise, and Shared Values of ATTC and SSA Staff

ATTCs have unique expertise that is an asset to the SSAs, and this expertise has helped build, solidify, and sustain the relationship between SSA directors and ATTCs.

A core function of the SSAs is providing training and assistance to professionals in the field, and SSAs are therefore motivated to form close relationships with their Regional ATTCs.

Although SSA Administrators interviewed have only been in their positions an average of 3.5 years, they and ATTC Directors have similar longevity in the field (ATTC Directors: average of 26.5 years; SSAs: average of 22.3 years), which facilitates establishing and maintaining relationships.

30


Policy and Economic Conditions Affecting States and the Addictions Treatment Field

Contextual factors also affect the relationship between SSAs and ATTCs and suggest their intensity and importance may increase in the near future. These include: Current economic recession, budget constraints, furloughs,

travel freezes, program cuts.Partnering with the ATTC is even more important in leaner times

when the State has fewer resources. We are such a small State and have a declining budget, so that partnering with the ATTC has been invaluable. We wouldn’t have been able to do training on returning veterans or MI without the ATTC.—SSA Director

Changing profile of individuals with substance use disorders and environment in which individuals are being treated.

Continuing high turnover in frontline staff and supervisors. Current health care reform and new requirements for parity,

which will lead to increasing numbers of insured individuals eligible for health coverage and behavioral health treatment.

31


FINDING 4

Many stakeholders see a need for the ATTCs to establish a national identity with regard to strengthening the workforce within the national behavioral health infrastructure.

Shifting federal direction and mandates to the ATTC Network

Increased collaboration among the Regional ATTCs Dissemination and coordination by the ATTC National

Office ATTCs moving toward a leading role in the national

behavioral health system

32


Some History . . .

ATTCs initially funded in 1993 N = 11 “independent Addiction Training Centers” U.S. not fully covered and no National Office

Expanded in 1998 N = 13 regional ATTCs and a National Office (no clear

role) Collaboration: The Change Book, TAP 21 Topically-focused inter-regional committees

A few years later ATTCs told to move away from national collaboration and

stick to meeting regional needs GPRA “targets” NIH/SAMHSA role demarcation

33


More Recently . . .

Increased collaboration 2007 RFA and direction from GPO Restoration of inter-regional work groups Reduce duplication and capitalize on regional “lessons

learned” TAP 21-A, ATTC Technology Transfer Model, ROSC

Expanded, proactive role of National Office Consolidated ATTC Website, Event and Activity Reporting

Database, Leadership Institutes, Workforce Survey, use of technology, etc.

National point of contact w/other professional groups (e.g., NADAAC, NASADAD)

Funding opportunities for the Network

34


The Changing Landscape

Behavioral Health is an Essential Part of Health —SAMHSA

ATTCs’ proven proficiency in establishing partnerships across wide variety of constituencies.

Data from this evaluation documenting benefits of ATTC work (beyond satisfaction) from a variety of perspectives .

Establishing partnerships with mental health and primary medical care professionals and organizations: National Association of State Mental Health Program Directors National Council for Community Behavioral Health Care Federally Qualified Health Centers, Community Health Centers

A consistent presence and voice in SAMHSA internal and inter-agency work groups.

Providing skill-based technology transfer to medical professionals (e.g., SBIRT).

Continuing to educate other professionals, disciplines.

35


Information Source

Not a universal feeling, but among those adding this to their “wish list” for ATTCs are stakeholders from: CSAT NASADAD Research Society on Alcoholism NADAAC Advocates for Human Potential SSAs Addiction Educators State Providers’ Associations

36


FINDING 5

Following their participation in ATTC initiatives designed to change clinical practice, practitioners report high levels of implementation of evidence based practices (EBPs) in addiction treatment settings. Implementation efforts were accompanied by improvements in associated clinical skills and improvement in supervisor/clinician/client relationships.

High levels of implementation of critical actions Improved implementation and EBP-related skills Key ingredients to successful implementation Improved clinical relationships Assurance of fidelity of implementation requires more

in-depth study

37


Methodology and Data Sources (Review)

“Success Case” method Critical Action Surveys

N = 100 MI participants (70% response rate1) N = 150 CS participants (79%) N = 138 Tx Planning MATRS (79%)

Success Case Interviews N = 24 MI (100% of target sample size) N = 21 CS (88%) N = 22 Tx Planning MATRS (92%)

1 Of eligible and consenting participants

38


“Critical Actions” of EBPs

To assess changes in practice, needed to get specific—How would we know that a clinician was putting into practice what they learned via ATTC technology transfer activities? Through review of research literature and ATTC curricula,

identified “critical actions” associated with each of the 3 topics 5 for MI (e.g., Seeking to understand your client’s frame of reference via

reflective listening) 6 for CS (e.g., Conducting regular supervisory interviews with supervisees) 7 for Tx PM (e.g., Producing individualized treatment plans that include

goals reflecting what the client wants to achieve)

These critical actions were the centerpiece of our Critical Action Surveys (“Since participating in the ATTC . . . to what extent have you been able to implement the following . . . ?)

Formed a composite of the 5, 6, or 7 critical action implementation responses for a “total score” of implementation of a given EBP

39


High Prevalence of Implementation of Critical Actions after ATTC Technology Transfer Event(s)

40

Legend (a) No plan to try (c) In process (d) Tried, with unclear results

(b) Not yet, but intend to (e) Tried, with positive results

(c) 24%

(b) 6%

(e)30%

(d)40%

Clinical Supervision

(d)19%

(e)81%

Motivational Interviewing

(d) 24%

(c) 11%

(b) 10%

(a) 2%

(e)53%

Tx Planning MATRS


Other Improvements Reported after ATTC Technology Transfer Event(s)

Significant improvement in proficiency with these critical actions (all 18 at p < .001)

Improvements in related clinical skills or agency operations MI: 5 skills, 97%–99% reporting improvement CS: 4 skills, 91%–98% reporting improvement Tx PM: 4 skills, 61%–83% reporting improvement

Perception of improved client behaviors such as client engagement, retention in treatment MI: 62%–83% Tx PM: 50%–60%

Reported by many interviewees: Improved clinical relationships MI and Tx PM: Clinician and Client CS: Clinical Supervisor and Clinician

41


Key Ingredients to Successful Implementation

Individual Clinician characteristics Belief that the practice was important to quality care Prior experience/familiarity with the critical actions

Treatment Agency/Organizational characteristics Formal support from clinical supervisor(s); informal

support and recognition from management Work environment supportive of client-centered

approaches to care ATTC TT Event(s) characteristics

Expert trainer, highly interactive format, peer-to-peer exchange

Evidence-based content, concrete, user-friendly materials (checklists, charts, forms)

42


FINDING 6

While the ATTCs deliver high quality services to meet the needs of diverse stakeholders and customers, there is less emphasis on matching participants to services and providing posttraining support to improve technology transfer outcomes.

ATTC services match needs and requirements of stakeholders and are highly rated

Recruiting participants to match the services provided and outcomes desired

Postevent support enhances technology transfer outcomes

43


Service Delivery Continuum

Individual, organizational, and event characteristics all play a role in supporting implementation.

In considering these data in the context of a full service delivery continuum, multiple lessons emerge.

44


Clearly Identified Strengths of ATTC Service Delivery

ATTCs successfully interact with a broad array of customers and stakeholders, resulting in a series of service delivery topics that are consistent with the needs and interests of ATTC stakeholders and customers and the requirements of their funders. (c.f., Finding 2)

Customers report a high level of benefit and satisfaction from the activities, events, and products provided by the ATTCs. (c.f., Finding 1)

Customers described specific characteristics of ATTC events that aided their learning and eventual implementation of critical actions associated with EBPs (c.f., Finding 5):

45

Event Format Content/Materials Trainer

Accommodating multiple learning styles

Highly interactive with trainer Opportunities for peer-to-peer

exchange Sessions spaced over time to allow

for “back at work practice” and “group problem-solving”

Face-to-face followup (and ongoing “refreshers”)

Evidence-based; efficacy supported by research literature

Easily referenced and used (checklists, reference documents, etc.)

Experienced with topic (expertise and comfort in presenting)

Experience in addictions field Belief in efficacy and utility of

topic


Importance of Recruitment in Achieving Technology Transfer Outcomes

Critical Action Surveys and subsequent success case interviews offer compelling evidence that participants’ success in postevent implementation is based more on the right match of service and participant than the length or intensity of the event(s). A participant’s prior experience with the practices and postevent

support provided were the strongest determinants of postevent implementation success (regardless of self-reported proficiency).

Many ATTC participants reported not being in an appropriate role to implement various critical actions, jeopardizing many potential technology transfer outcomes.

Differential recruitment processes have varying levels of control over who attends ATTC events—hugely important especially for events tagged with “change in practice” objective. ATTC-controlled versus shared or externally controlled recruitment

strategies Targeted versus open recruitment

46


Importance of Post Service Follow-Up & Support in Achieving Technology Transfer Outcomes

Data clearly indicate that participants need and want both formal and informal posttraining support to have meaningful, lasting implementation success.

Specifically, they need further support to assist with (a) making the new practice a priority, (b) getting better at the practice they are implementing, and (c) avoiding the tendency to “fall back into their old ways” of doing things.

Participants cited multiple organizational- and ATTC-based supports that aided their successful implementation.

47

Organizational Support ATTC Technology Transfer-Based Support

1. Formal support from clinical supervisor 1. Sessions spaced over time to facilitate “back at

work practice” and “group problem-solving”

2. Informal organizational support (recognition) 2. Repeat opportunities for booster and refresher

trainings (face to face) 3. Collegial support/shared implementation

experience 3. Evidence-based, easily accessible materials


FINDING 7

ATTC services produce improvements in the knowledge, skills, and practices of the addictions treatment workforce. However, limited resources appear to be devoted to the specification and measurement of the intended outcomes of these services.

ATTCs’ technology transfer efforts bring about a variety of positive changes

Defining technology transfer success remains a challenge The ATTC Network technology transfer model: boundaries

and definitions Lack of clarity in intended outcomes of ATTC technology

transfer activities The benefits of improved clarity in the intent and goals of

ATTC technology transfer

48


The Work of ATTCs Results in Positive Changes in Their Participants

As reported previously (c.f., Findings 1 and 5), multiple data sources in this evaluation indicate that ATTC service participants report positive changes in their knowledge, skills, and practices following ATTC events. Specifically: Statistical analysis of pretest, posttest, and 30-day follow up GPRA

over the first 3 years of the current funding period demonstrates participants perceive a statistically significant increase in their knowledge, skills, and effectiveness from pre- to postevent and from post- to 30-day follow-up of the event (both p < .01).

This perceived improvement is also supported by findings from the CSBS, with 95% of those respondents expressing agreement that they increased their knowledge, 91% that they increased their skills, and 90% that they were able to do their job better as a result of services received from the ATTCs. (c.f., Finding 1).

Critical Action Surveys and follow-up interview results further confirm the ATTCs’ impact on clinical practice as training participants reported high levels of implementation, increased proficiency, and significant impacts on their clinical practice (c.f., Finding 5).

49


Defining Technology Transfer Success is a Challenge

Need for common terms and definitions: Several models or frameworks for technology transfer Corresponding variety and inconsistency in their terminology related to

important concepts such as “success” Although the literature leaves little doubt that the ultimate goal when

dealing with evidence-based practices is full implementation with fidelity and/or rational adaptation, smaller improvements to practice can be worthwhile.

Changes may occur in multiple, incremental stages of implementation, distinguishing between installation, initial implementation, and full implementation (Fixsen et al., 20051) of practices.

While it is clear that ATTC technology transfer efforts can elicit incremental changes and the adoption of discrete behaviors associated with EBPs, it is difficult to assess their success without knowing their ultimate goal or intention of service delivery.

1Fixsen, D., Naoom, S.F., Blasé, K.A., Friedman, R.M. & Wallace, F. (2005) Implementation research: A synthesis of the literature. FMHI Publication No. 231 Univ. of South Florida, Louis de la Parte

Florida Mental Health Institute, The National Implementation Research Network

50


An Example . . .

Treatment Planning MATRS participants clearly changed their practices with regard to making the client’s voice a central part of the treatment planning process, which resulted in significant outcomes related to client-clinician relationships and overall client engagement in the ultimately prescribed treatment program. However, a significant piece of the Treatment Planning MATRS curriculum (using results from standardized assessment tools for the purpose of guiding the treatment planning process) was implemented far less frequently.

51


Current Framework Offers Promise

The ATTC Technology Transfer Model, developed in the current year by the ATTC Technology Transfer Workgroup, makes great strides in (a) defining the basic tenets of the technology transfer process, and (b) establishing boundaries as to where the ATTC Network’s responsibilities start and end. However, terms such as “promoting adoption”, “encouraging

success”, and “early implementation” have not been accompanied by any form of measurement or success metric and, at the service delivery level, the issues of identifying explicit goals, defining success, and implementing an assessment strategy remain.

The ATTC Network most clearly attempts to define the technology transfer intentions of discrete service delivery via 3 technology transfer objectives: awareness-raising, skill-building, and change in practice. These objectives are intended to represent important distinctions

in the specific intent of technology transfer activities, but they appear to be unclear or inconsistently applied across the Network.

52


An Example . . .

Of the 32 technology transfer events included in the Change in Practice study in this evaluation, none were specified by its sponsoring ATTC to have change in practice as its objective. Further, 15% of these events specified awareness-raising, an objective at the opposite end of the continuum from an intent to change clinical practice.

53


Benefits to Seeking Further Clarity

Added clarity, accompanied by consistent reporting, not only will help ATTCs define success but also will help: Better define expectations for all involved

(partners and participants). Establish the level of effort needed in planning,

delivery, and follow-up of services. Identify optimal strategies and resources needed

to achieve intended outcomes. Increase the evaluability of ATTC efforts,

rendering success easier to define and lessons learned easier to interpret.

54


Next Steps for Evaluation Findings. . .

Completion of more detailed reports of the methodology and findings of the 3 Evaluation Studies (Jan. 2011) Planning, Partnering and Service Delivery — Abt Associates Customer Satisfaction and Benefits — MANILA Consulting Change in Practice Study — RMC Research

Webinars for ATTC Directors and staff on more detailed findings of these studies (Jan.-Feb. 2011)

Dissemination of reports Executive Summary Full Report Region-Specific Results (CSBS, E&A Reporting Database)

55


Next Steps in Today’s Presentation

Lunch

Presentation and Discussion of Evaluation’s Recommendations

56


Recommendations

Requested of us by CSAT The collective sentiments from the

National Evaluation Team (NET) Not necessarily endorsed by CSAT or all

stakeholders Implications for both the ATTC Network

and its funders

57


Recommendation #1

SAMHSA/CSAT and the ATTCs should devote increased attention and resources to the Implementation phase of the Network’s Technology Transfer Model

Demonstrated expertise in planning, partnering, and service delivery

Requests from stakeholders Evidence from CIP studies Current health reform expands these

opportunities Obvious resource implications for

SAMHSA/CSAT, NIDA

58


ATTC Network Technology Transfer Model (2010)

59


Evidence From Change in Practice (CIP) Study

ATTCs have shown they can do this. CIP study event participants reported: Very high prevalence of (self-reported) implementation of

specific critical actions associated with all 3 CIP study topics Statistically significant (self-reported) improvement in

proficiency in critical action implementation Very high prevalence of (self-reported) improvement in

clinical skills and agency operations associated with the CIP study topics

Stakeholders requesting it Another need not being met is follow-up after dissemination: are people actually

doing what they were trained to do or did they just go back to their programs and do what they were doing before they came to the trainings? – Addiction Educator

60


New Opportunities

Behavioral Health is an Essential Part of Health

—SAMHSA

61


Recommendation #2

ATTCs, in collaboration with SSAs, other partners, and evaluation experts should invest more in identifying, clarifying, and measuring the intended outcomes of their services.

Benefits to ATTCs and their partners Provides opportunities to better demonstrate

successes and lessons learned SSA: “We don’t pay for activities, we pay for

results.”

62


Clarifying Intended Outcomes of Specific Activities: Benefits to All

To ATTCs Guiding the technology transfer approach, format, intensity, and

resources allocated to services Further clarifying lessons learned, standardizing approaches

across regions Committing to outcomes, avoiding “scattergun”

accomplishments unlikely to be replicated

To their Partners and Customers Clarifying their expectations, potentially leading to better

participant/service match Keeping planning and implementation activities on the same

page

To SAMHSA/CSAT Identifying best practices in technology transfer Advancing the reputation of ATTCs and the Network

63


Recommendation #3

ATTCs should not allow their strength in tailoring their services to specific needs of their SSAs and region to preclude further establishing a national, Network-wide identity and responding to opportunities to address national issues. A Network-wide strategy New opportunities with national health reform Working with new disciplines will require more

formal needs assessment processes (a continuation and acceleration of an ATTC trend)

SAMHSA/CSAT—not another “unfunded mandate”

64


Recommendation #4

To ensure the sustainability of the Network, SAMHSA/CSAT and the ATTCs must plan for “what’s next” in evaluation for this highly visible, long-standing program. First since 1993 GPRA ≠ Evaluation, and data used only minimally Evaluation implementation challenges, extending

beyond ATTCs to the treatment community Need capacity building in evaluation in some

regions

65


Challenges in Implementing This Evaluation

Excellent cooperation from ATTC regions and the addictions treatment community, in general. However, serious instances of Inattention to evaluation planning considerations Casual implementation of evaluation activities/procedures

(e.g., CSBS sampling, CIP event specification, ORC Survey, MI audiotape administration)

Inconsistent translation of data collection instructions (e.g., consent processes in CIP events)

GPRA system: A poor act to follow Lack of meaningful interpretation and use of data at federal

level Meaningless procedures and ad hoc standards in data

collection (e.g., resulting in follow-up response rates > 100%)

66


What’s Next in Evaluation for ATTC Network?

Needs Evaluation investment and capacity building across

Network Evaluation as a voice in Network activity planning

and implementation

Opportunities Secondary analysis of GPRA data Use the ATTC Event and Activity Reporting

Database Target evaluation resources to important, specific

Network initiatives (e.g., Leadership Institutes, skill-building and change in practice efforts with primary care physicians)

67


Recommendation #5

SAMHSA/CSAT and the ATTCs should identify and implement mechanisms to expand the resources available to support ATTC events, activities, and initiatives Increasing need and opportunities for ATTCs Very difficult for SAMHSA/CSAT to increase base

funding of ATTCs Other existing discretionary grant programs offer

opportunities Better documentation, marketing of ATTC

successes

68


Existing Discretionary Grant Opportunities

SAMHSA/CSAT and other Federal Agencies: When federal agency identifies ATTCs as a resource in grant

announcements, suggest/designate a portion of grant budget be allocated to ATTC as partner or subcontractor (TA, workforce development), as often done for evaluation

Encourage states to dedicate a portion of their SAPT Block Grant to ATTC, for workforce development and training

Include ATTCs in inter-agency strategic planning or work groups, e.g., implementing Affordable Care Act or 2008 MH Parity and Addictions Equity Act

Other TA providers supported by SAMHSA (e.g., State Systems Development, TA to States programs) can act as referral agents to ATTCs

ATTCs and National Office: Monitor discretionary grant opportunities and approach

states with partnering ideas and services

69


Next Steps for Recommendations

Submitted to CSAT with Final Report, Sept. 30

Distributed to CSAT leadership and staff associated with ATTC Network

We’re done

70


And, Finally, From the National Evaluation Team . . .

Heartfelt thanks to all who supported and contributed to this evaluation To the ATTCs and the National Office for participation in the

initial conceptual planning of the evaluation; and for assistance in the implementation of the many data collection activities in this evaluation.

To CSAT and, specifically, Deepa Avula, our project officer, for her belief in our evaluation

design and wise counsel in our implementation of the evaluation activities and interpretation of results

Cathy Nugent, ATTC project officer for most of the evaluation, for her unflagging support and encouragement of our efforts

Donna Doolin, current ATTC project officer, for picking up where Cathy left off and contributing her insights to the evaluation

To the many ATTC customers, stakeholders, and partners who contributed their opinions, perceptions and experiences to this comprehensive evaluation effort.

71

Date post:	18-Jan-2016
Category:	Documents
Upload:	silvester-bridges
View:	218 times
Download:	0 times

ATTC Network Evaluation National Evaluation of the ATTC Network: Findings and Recommendations...

Documents