None of the interventions are

evidence-based but think positive – you

get a new and untested

intervention

IMS APPROACH TO GOOD PRACTICE

Knut SundellJenny RehnmanMari Forslund

www.evidens.nu

Development & Evaluation

(assessment tools and interventions)

20 employees 30 projects lasting

3-4 years

Systematic reviews(assessment tools and

interventions)

10 employees 15 projects lasting 1-

2 yearsAdditional external

researchers

Support & Dissemination

(administrative support & dissemination of all

that is produced within IMS)

10 employees 15 projects lasting

less than 1 year

IMS director Board of IMS

(12 stake-holders)

Director-general NBHW

• Sweden is the third largest country in Western Europe with nine million inhabitants

• Social care service delivery involve agencies of the 290 municipalities, and 20 regional county councils. Median size of municipalities = 15.250 inhabitants

• National agencies and research councils, deals with research and development, supervision of service quality and safety, and the delivery of institutional care.

• The voluntary sector is considerable

• Social welfare states (e.g., low rate of unemployment, poverty, drug use, violence)

• General trust among Swedes in collective solutions and in government authorities

Sweden

Research on interventions for womenwith experience of partner violence(Anttila et al, 2007)

5 926 articles

Research on interventions for womenwith experience of partner violence(Anttila et al, 2007)

5 926 articles

Effect studies 99

Research on interventions for womenwith experience of partner violence(Anttila et al, 2007)

5 926 articles

Good evidence 6

0,1 per cent

Swedish outcome studies on social work

0

5

10

15

20

25

30

35

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008

RCT Quasiexp

Two articles in Sweden’s largest morning paper by the Director-General of the Swedish National Board of Health and Welfare claiming that social work agency managers did not have any idea whether social care services made any difference to clients’ lives.

Important dates

National action plan for preventing alcohol related harm & National action plan on drugs (lasting to 2007). 100 million SEK on researchTraining of local prevention workers

IMS is inagurated

General election and a new government

The first national guidelines on social work (substance abuse treatment)

Official reports of the Swedish government – “Evidence-based social work practice – favouring clients”

Agreement between the government and the Swedish Association of Local Authorities and regions on implementing the national guidelines on substance abuse treatment

An inquiry initiated at the NBHW on how to support EBP byNational guidelines, research reviewsNational and regional knowledge transferDeveloping criteria for local-follow up and open comparisons

Initiative to clarify the concept of EBP among governmental authorities

Addiction Severity Index is introduced in Sweden

Looking for an evidence-based practice

Random sample of local authorities (2/3)

Managers responsible for budget & staff

Autumn 2007

Electronic questionnaire

Response rate 88% (n = 834)

INTEREST IN EVIDENCE-BASED METHODS (N = 834)

Definitely

Partly

No

PREVALENCE OF EVIDENCE-BASED METHODS (SOMETIMES)

0

20

40

60

80

100

Children Youth Substanceabuse

Disabled Socialassistance

Elderly

%

24 per cent

PREVALENCE OF EVIDENCE-BASED METHODS (MOSTLY)

0

20

40

60

80

100

Children Youth Substanceabuse

Disabled Socialassistance

Elderly

%

7 per cent

PREVALENCE OF EVIDENCE-BASED METHODS (SOMETIMES)

0

20

40

60

80

100

%

PREVALENCE OF STANDARDISED ASSESSMENT TOOLS (SOMETIMES)

0

20

40

60

80

100

Children Youth Substanceabuse

Disabled Socialassistance

Elderly

%

63 per cent

0

20

40

60

80

100

Children Youth Substanceabuse

Disabled Socialassistance

Elderly

%

14 per cent

PREVALENCE OF STANDARDISED ASSESSMENT TOOLS (REGUAL)

Sweden: What are the political issues that need to be addressed?

Keywords: Ethics – Evidence – Transparency

not the technical issues concerning internal / external validity

Knowledge transfer / dissemination of innovations need extensive support at a regional level

Bring forward good examples

Stockholm evidence-based Clearinghouse for social work

• ”Evidence-based medicine is the conscientious, explicit and judicious

use of current best evidence in making decisions about the care of

individual patients” (Sackett et al., 1996)

• The general aim of “the Stockholm evidence-based Clearinghouse for social work” is to build a bridge between professionals and research on “what works”

• It was launched

on 1st September, 2008

18

Stockholm evidence-based Clearinghouse for social work

• Web-based service (at IMS’ homepage): http://www.socialstyrelsen.se/Amnesord/socialt_arbete/IMS/Metodguiden_index.htm

• Provides information on interventions, assessment tools and general knowledge (e.g., mechanisms)

• Target group: Professionals, politicians and policy makers

• Not recommendations

• Arranged in a simple, straightforward format reducing the need to conduct literature searches

• Methodology of the systematic review (e.g., Higgins & Green, 2008)

Topics Currently Available on the Website

Target groups

• Child/family

• Social assistance (for welfare recipients)

• Disability

• Addiction

• Elderly

Subheadings

• Interventions (18 descriptions/5 final evaluations)

• Assessment tools (46 descriptions/13 final evaluations)

• General knowledge

• IMS projects

20

The evaluation process

• Selection of interventions to evaluate – starts with discussion in IMS research council: Which interventions should we evaluate? Which outcomes are important? What databases should be searched?

• All relevant studies with a RCT or QE (including data at baseline) are reviewed

• Use of a guide/protocol to assist the review process• Studies are evaluated for to their internal validity (i.e., can we trust

the result?)• Two independent reviewers evaluate each study. Any

disagreements are settled based on consensus with help from a coordinator

22

23

Judging of internal validity

• Selection bias e.g., Are there any important differences between the groups?

• Performance biase.g., Are researchers, participants and data collectors ”blind”?

• Attrition biase.g., Any differences in size and type of attrition between the groups?

• Detection biase.g., Are the outcome measures measured in the same way in all groups?

Potential bias is evaluated within and across each domain of bias• Low risk: plausible bias unlikely seriously alter the results• Unclear risk: plausible bias that raises some doubt about the results• High risk: plausible bias that seriously weakens confidence in the results

(from Higgins & Green, 2008)

Grading of evidence

• The scientific raiting scale is a modified version of a scale developed at California Evidence-Based Clearinghouse for Child Welfare (http://www.cachildwelfareclearinghouse.org/scientific-rating/scale)

• Based on statements in Flay et al., 2005 (Society for Prevention Research, SPR)

• The scale is devided into five grades - A lower score indicates effectiveness and a higher level of research support

• Grading of evidence is based on:– Number of studies with a certain degree of internal validity (i.e., risk of

bias)

– If the practice is evaluated in usual care

– If the effect are sustained over time

– If the practice may cause harm

– If the practice is replicable

24

1. Effective practice with well-supported research evidence

• At least two studies with low risk of bias, in different usual care or practice settings, have found the practice to be superior to treatment as usual (TAU).

• If multiple effectiveness studies have been conducted, the overall weight of the evidence supports the benefit of the practice.

• In at least one study with low risk of bias, the practice has shown to have a sustained effect at least one year beyond the end of treatment.

• There is no theoretical or empirical basis indicating that the practice constitutes a substantial risk of harm to those receiving it.

• The practice has a book, manual, and/or other available writings that specify components of the service and describes how to administer it.

2. Effective practice supported by research evidence

• At least one study with low risk of bias has found the practice to be superior to treatment as usual (TAU).

• If multiple outcome studies (at least with medium risk of bias) have been conducted, the overall weight of evidence supports the benefit of the practice.

• In at least one study with low risk of bias, the practice has shown to have a sustained effect of at least six months beyond the end of treatment.

• There is no theoretical or empirical basis indicating that the practice constitutes a substantial risk of harm to those receiving it.

• The practice has a book, manual, and/or other available writings that specifies the components of the practice protocol and describes how to administer it.

3. Practice with promising research evidence

• At least one study with medium risk of bias has established the practice's benefit over no intervention (or placebo or waiting list) or is found to be comparable to or better than treatment as usual (TAU).

• If multiple effectiveness studies with at least medium risk of bias have been conducted, the overall weight of evidence supports the benefit of the practice.

• There is no theoretical or empirical basis indicating that the practice constitutes a substantial risk of harm to those receiving it.

• The practice has a book, manual, and/or other available writings that specify the components of the practice protocol and describe how to administer it.

4. Practice where the evidence fails to demonstrate effect

• At least two studies with low risk of bias have found that the practice has not resulted in improved outcomes compared to no intervention (e.g., placebo or waiting list), or that the practice is shown to be less effective when compared to treatment as usual.

• If multiple effectiveness studies have been conducted, the overall weight of evidence does not support the benefit of the practice.

• There is no theoretical or empirical basis indicating that the practice constitutes a substantial risk of harm to those receiving it.

• The practice has a book, manual, and/or other available writings that specify the components of the practice protocol and describe how to administer it.

5. Concerning practice

• At least one study of low or medium risk of bias shows that the intervention can cause serious harm, and/or there is a reasonable theoretical basis suggesting that the practice constitutes a risk of harm to those receiving it.

• The practice has a book, manual, and/or other available writings that specify the components of the practice protocol and describe how to administer it.

Beyond the five grades the scientific scale includes an additional category

(not rated - no number is given).

Practice with unknown effect • There is lack of studies with a medium or low risk of bias. • There is no theoretical or empirical basis indicating that the practice

constitutes a substantial risk of harm to those receiving it.• The practice has a book, manual, and/or other available writings that

specifies the components of the practice protocol and describes how to administer it.

Conclusions and summary - Is there a sufficiently robust evidence base to identify good practice?

• What are the strength and weaknesses?

• Are there significant gaps that should be addressed?

• Is there an agreed approach to deciding what counts as evidence?

• Is there an agreed approach for judging the quality of the evidence base?

• Is there a scope for working internationally to strengthen the evidence-base?

IMS practical delivery mechanisms to promote the adoption of good practice

IMS want to reach:

• Social workers• Politicians and decision makers• Authorities• Universities and students • Researchers• Clients and others who are interested

Target groups

• Social work agency managers• Government officials

Parts of our communication strategy

• Value of direct, face-to-face communication

• Web-based services

• Tailored products

• Short courses for social work agency managers about EBP

• Start kit for municipalities for working with EBP

IMS STRATEGI

ETICS

INFORMATION

TOOLS FOR CHANGE

NOT COACHING SPECIFIC METHODS

Target groups

Social workers

Politicians and decision makers

Governments

- - - - - - - -

Universitys

Goals

Increase the interest for EBP

Provide support for decision-making

Implementation knowledge

Support to managers

Long-term partnership

Activities

Lectures

Clearinghouse

Readiness for change

Handbook in leading EBP

Pilot-project

Train the trainers

Ask a researcher

Start-up-package

IMS network for EBP

Organizational Organizational Readiness for Readiness for

ChangeChange

Dwayne SimpsonDwayne Simpson

Anonymous survey to co-workers, managers and clients

Pilot-project Södermalm - Trying different mechanisms to promote EBP

• A steering group and 24 IMS-coodinatiors with special drive• Seminars about EBP and implementation• ORC-survey• Focusing on the managers with seminars about leading EBP

– the importance of the managers– management research– tools from the handbook

• Local seminars about how to find research and assessment tools

Training trainers in ASIAddiction Severity Index

• Big difference when local support in the new method is given and when the managers get’s support and owns the implementation

• 2 persons at IMS, 25 local trainers, 12 county administrative boards and 194 municipalities'

• Combination of IMS knowledge in implementation research and the method ASI and the local knowledge about the practice and support to the managers

• Success factors:– National support– Local competence– Managers in focus– Networking

Start-up- package

Content: EBP – concepts and implications  Evidence, ethics and affectivity Risk- and protective factors Assessment tools Research reviews Dissemination EBP and cost effectiveness  

Including: DVD-films The book Evidence based

practice in social work The book To change social work PowerPoint material Glossary Information about IMS website

Focused on Managers

Study material as an introduction to EBP

Publications

Collaboration with a publishing house (Gothia förlag)

Adapted to our target group

Reading guidances

IMS-nytt

New layout + more pages

4 numbers annually

Special feature issues

Research and interviews with professionals and clients

Challenges

Challenges

1. Lack of interest among the Swedish social work academia to support an EBP, and train the future generations of social workers accordingly

2. Lack of (inter)national agreement on how to grade evidence

3. Questions on the transportability of evidence-based interventions between countries

4. Lack of reliable and valid measures of quality of services at a local level (in order to motivate change)

5. Lack of an infrastructure for diffusion of innovations, and knowledge transfer

AIDA – acronym used in marketing

A Attention

I Interest

D Desire

A Action