DOCUMENT RESUME

ED 449 390 CG 029 836

AUTHOR Devine, PatriciaTITLE Using Logic Models in Substance Abuse Treatment Evaluations.

Integrated Evaluation Methods.INSTITUTION Caliber Associates, Fairfax, VA.; National Evaluation Data

and Technical Assistance Center, Rockville, MD.SPONS AGENCY Substance Abuse and Mental Health Services Administration

(DHHS/PHS), Rockville, MD. Center for Substance AbuseTreatment.

PUB DATE 1999-07-00NOTE 65p.; This document is being made available through

Caliber/NEDS Contract No. 270-97-7016.CONTRACT 270-94-0001PUB TYPE Guides Non-Classroom (055) Reports Descriptive (141)EDRS PRICE MF01/PC03 Plus Postage.DESCRIPTORS Activities; Data Analysis; Data Collection; *Evaluation

Methods; *Logic; Measures (Individuals); *Models; *Outcomesof Treatment; Program Development; *Substance Abuse

ABSTRACTBased on .a decade of evaluation experience, the Center for

Substance Abuse Treatment (CSAT) has developed the Integrated EvaluationMethods (IEM) Package, a series of conceptual and methodological applicationsto enhance CSAT-funded evaluation activities. Products in the IEM Package areorganized within an evaluation framework constructed on the basis ofaccumulated experiences among evaluation professionals. The framework isbased upon evaluation strategies, structures, and approaches appropriate forsubstance abuse treatment evaluators and providers. This document isspecifically aimed at supporting the design stage of the evaluation process.It provides a definition of logic models and discusses their use in treatmentservices planning and evaluation. It is intended to assist substance abusetreatment professionals to plan and conduct statistically valid, and,therefore, meaningful evaluation activities. An appendix entitled "IntegratedEvaluation Methods Package: A Guide for Substance Abuse TreatmentKnowledge-Generating Activities--Executive Summary" is included. (Contains 4figures, 3 tables, and 10 references.) (MKA)

Reproductions supplied by EDRS are the best that can be madefrom the original document.

co

a

INTEGRATED EVALUATION METHODS

USING LOGIC MODELS IN SUBSTANCE ABUSETREATMENT EVALUATIONS

BEST COPY AVAILABLE

NEDTAC

July 1999

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

This document has been reproduced asreceived from the person or organizationoriginating it.

Minor changes have been made toimprove reproduction quality.

Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position or policy.

CSAT CALIBERCenter for Substance

Abuse TreatmentSAMHSA

0

ASSOCIATBS

INTEGRATED EVALUATION METHODS

USING LOGIC MODELS IN SUBSTANCE ABUSETREATMENT EVALUATIONS

Prepared by:

Patricia Devine

Caliber Associates10530 Rosehaven Street, Suite 400

Fairfax, VA 22030

July 1999

This document was produced by the Center for Substance AbuseTreatment, Department of Health and Human Services,Caliber/NEDTAC Contract No. 270-94-0001 and is being madeavailable through Caliber/NEDS Contract No. 270-97-7016. J

CSATCenter for Substance

Abuse TreatmentSAMHSA

TABLE OF CONTENTS

Page

FOREWORD

ACKNOWLEDGMENTS ii

I. INTRODUCTION 1

1. CONTEXT FOR THE LOGIC MODEL DOCUMENT 1

2. HOW THIS PAPER IS ORGANIZED 2

II. DESCRIPTION OF THE LOGIC MODEL 3

1. DEFINITION 3

2. LOGIC MODEL COMPONENTS 3

3. USES OF LOGIC MODELS 4

III. USING LOGIC MODELS TO PLAN KNOWLEDGE-GENERATINGTREATMENT SERVICES 7

1. CONDITIONS AND CONTEXT FOR KNOWLEDGE-GENERATINGTREATMENT SERVICES 7

2. KNOWLEDGE-GENERATING TREATMENT SERVICES ACTIVITIES . . . 9

3. KNOWLEDGE-GENERATING TREATMENT SERVICES OUTCOMES . . . 10

IV. USING LOGIC MODELS FOR EVALUATING KNOWLEDGE-GENERATING ACTIVITIES 11

1. CONDITIONS AND CONTEXT FOR THE KNOWLEDGE-GENERATINGEVALUATION 11

2. KNOWLEDGE-GENERATING SUBSTANCE ABUSE TREATMENTSERVICES 13

3. KNOWLEDGE-GENERATING SUBSTANCE ABUSE TREATMENTOUTCOMES 14

4

TABLE OF CONTENTS (CONTINUED)

Page

V. LOGIC MODELS TO SPECIFY KNOWLEDGE-GENERATINGEVALUATION MEASURES 16

1. DATA MAP STRUCTURE AND UTILITY 16

2. DATA MAP DEVELOPMENT 16

2.1 Identifying Over-Arching Evaluation Questions 17

2.2 Defining Knowledge-Generating Secondary Evaluation Questions . . . 17

2.3 Identifying Evaluation Measures 18

2.4 Identifying Data Sources/Instruments 18

2.5 Determining Data Collection Time Points 19

VI. SUMMARY 20

REFERENCES 34

APPENDIX: INTEGRATED EVALUATION METHODS PACKAGE: A GUIDE FORSUBSTANCE ABUSE TREATMENT KNOWLEDGE-GENERATINGACTIVITIESEXECUTIVE SUMMARY

5

FOREWORD

Over the last 10 years the Center for Substance Abuse Treatment (CSAT) has

accumulated a great deal of experience in substance abuse treatment evaluation implemented

through coordinating centers, cross-site efforts, and national studies. The importance and value

of integrating ongoing evaluation activity into a system for treating substance abuse problems is

widely recognized by treatment providers and by CSAT. Also widely recognized, however, is

that current evaluation generated knowledge and practice are often under-utilized, due in part to

the lack of an integrated approach to capturing information with which to measure treatment

outcomes and improve the treatment process. CSAT recognizes that such an integrated

evaluation approach will more effectively support its knowledge generating activities.

Based on a decade of evaluation experience, CSAT has developed the Integrated

Evaluation Methods (IEM) Package, a series of conceptual and methodological applications,

including concept papers, technical assistance materials, and analytic tools, to enhance CSAT-

funded evaluation activities. Products in the IEM Package are organized within an evaluation

framework constructed on the basis of accumulated experiences among internationally known

treatment service evaluation professionals. Thus, the framework is based upon evaluation

strategies, structures and approaches appropriate for substance abuse treatment evaluators and

providers. The framework follows a standard set of evaluation activities: planning, selecting a

design, developing data requirements and collection instruments, collecting and analyzing the

data, and reporting the evaluation findings. (A summary description of the IEM Package iscontained in the Appendix to this document.)

This document, along with its two companion documents, A Guide to Process Evaluation

for Substance Abuse Treatment Services and A Guide to Selecting an Outcome Evaluation

Design for Substance Abuse Treatment Evaluations, is specifically aimed at supporting the

design stage of the evaluation process. This document provides a definition of logic models and

discusses their use in treatment services planning and evaluation. Taken together, these three

documents are intended to assist substance abuse treatment professionals to plan and conduct

scientifically valid, and therefore meaningful, evaluation activities.

Sharon BishopDirectorNational Evaluation Data and Technical Assistance Center

JACSAT\CTRTENMEM\CONCEPIALOGICMDL\DATA\LOGICMDL.WPD

6

NEDTAC, Page i

ACKNOWLEDGMENTS

This paper, together with the companion documents listed in the Appendix (the Integrated

Evaluation Methods Package), was developed for CSAT by the National Evaluation Data and

Technical Assistance Center (NEDTAC) under the guidance and direction of Ron Smith, Ph.D.,

Program Evaluation Branch, Office of Evaluation, Scientific Analysis, and Synthesis (OESAS).

Dr. Herman Diesenhaus, former Team Leader, Scientific Analysis Team, OESAS, contributed

many concepts that have been incorporated into the package. Charlene Lewis, Ph.D., former

Deputy Director, OESAS, supported this and other associated efforts, with the result that state-

of-the-art evaluation concepts were incorporated into many of CSAT's and SAMHSA's

evaluation initiatives. Jerry Jaffe, M.D., former Director, OESAS, also contributed his breadth of

experience in the substance abuse treatment and evaluation fields and his dedication to high

quality treatment services evaluation and provided the national level leadership necessary for

CSAT to promulgate these activities.

Caliber Associates was the prime contractor for NEDTAC in partnership with

Computech, Inc.; the Lewin Group; Capital Consulting Corporation; the Center for Substance

Abuse Research (CESAR), University of Maryland; the Alcohol Research Group (ARG), Public

Health Institute; the Drug Abuse Research Center (DARC), University of California, Los

Angeles; and the Urban Institute. Many people within the NEDTAC team contributed to this

effort. These staff include Patricia Devine, Jacquelyn Lowery, Harriet Perrine, Marsha Morahan,

Robin Walthour, and Donna Caudill. Contributions ranged from document development to

editing and production, and all of these efforts were equally invaluable and greatly appreciated.

JACSATACTRT_ENDVEM\CONCEPIALOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page ii

7

I. INTRODUCTION

The Center for Substance Abuse Treatment (CSAT) supports the integration of ongoing

evaluation within substance abuse treatment activities so as to demonstrate treatment service

effectiveness and to improve treatment services and their outcomes. To this end, CSAT

recommends the use of state-of-the-art evaluation methods and tools in planning, designing, and

implementing treatment services evaluations. This paper discusses one of these tools: the logic

model. The logic model provides the linkage among all of the evaluation activities and ensuresthe integration of process and outcome evaluation results.

1. CONTEXT FOR THE LOGIC MODEL DOCUMENT

CSAT's major evaluation goals are to: (1) increase knowledge about substance abuse

treatment services; (2) improve treatment services by applying knowledge gained through

knowledge development and application (KD &A) activities; (3) develop analytic methods and

approaches for use in knowledge-generating activities; and (4) develop substance abuse treatment

analysis databases. To meet these goals, CSAT has been sponsoring KD &A initiatives including

activities that focus on homelessness, marijuana use and treatment, managed care, women and

violence, and opioid treatment, as well as the replicability of exemplary treatment approaches

(e.g., methamphetamine treatment) and the evaluation of best practices for targeted populations(e.g., exemplary adolescent treatment).

CSAT's evaluation experiences have reinforced the fact that substance abuse treatment

evaluation involves a standard set of tasks that generally occur in the following order:

Planning the evaluation, which includes setting the evaluation goals and objectivesthat determine the overall parameters of the evaluation

Selecting the evaluation design, which sets forth the overall strategy for establishingthe evaluation questions, measurement approach, and generalizability of findings

Developing the data requirements, which flow from the evaluation questions andmeasures and include SDU, clinician, cost, and client data

Developing data collection instruments, which are based on the data requirementsand are developed or selected from a standard inventory of instrumentation

Collecting the data, which includes the development of data management processesand tools including quality control procedures, and collecting the data

JACSAT\CTRTENMEM\CONCEPTALOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 1

Introduction

Analyzing the data, which involves developing an analysis plan and conductingmultiple levels of comparison; the analysis process is governed by the analysis planand intended products and target audience(s)

Reporting the evaluation findings, which includes evaluation knowledgedissemination and application within field.

CSAT has directed the development of evaluation concepts, methods, and tools to support these

evaluation tasks. The evaluation tasks and corresponding evaluation methods products are

summarized in Exhibit I of the appendix to this document. As shown, the use of logic models in

CSAT evaluations is part of the second stage in the evaluation process: selecting the evaluation

design. A full discussion of the CSAT evaluation analytic framework and the other evaluation

concepts and tools is presented in the concept paper: Integrated Evaluation Methods: A Guide

for Substance Abuse Treatment Knowledge Generating Activities. This document, taken together

with the other evaluation methods products in Exhibit I, is known as the Integrated Evaluation

Methods Package. The documents that comprise the Package are being made available on theCaliber Associates NEDS contract Web site at http://neds.calib.com.

2. HOW THIS PAPER IS ORGANIZED

The paper is divided into six sections. Following the introduction, the paper provides a

definition of logic models and discusses their use in treatment services planning and in treatment

services evaluation. The paper concludes with a description and examples of using logic modelsto develop data maps that specify evaluation questions, measures, and variables.

JACSAT\CTRT_ENMEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 2

II. DESCRIPTION OF THE LOGIC MODEL

The following paragraphs provide a definition of logic models and describe the logic

model components and uses.

1. DEFINITION

A logic model is a descriptive, graphic representation of substance abuse treatment

services and how they are supposed to work. A logic model includes a succinct, logical chain of

statements that link substance abuse problems, treatment service interventions, and expected

treatment outcomes. Logic models incorporate the theoretical relationships among the source or

cause of the substance abuse problem, the design of the treatment service intervention, and the

expected treatment results (i.e., outcomes). Logic models are conceptually straight-forward but,

for substance abuse treatment, can be extremely complex because there are numerous, sometimes

competing, theories as to the causes of substance abuse, the most effective treatment

interventions, and the multiple short-term and long-term outcomes (Kumfer et al., 1993). For

any given treatment approach and evaluation strategy, these issues can be identified and included

in the logic model for that specific effort.

2. LOGIC MODEL COMPONENTS

A logic model typically consists of four components:

Conditions and context in which the substance abuse treatment services operate,including the target population characteristics, the community resources, and theregulations and policies that govern the treatment services operations

Activities and services offered as part of the substance abuse treatment

Short-term outcomes which are immediately expected to result from the treatmentservices

Long-term outcomes (impacts) which are expected, and which should correspond tothe treatment service goals.

The linkages for the conditions, activities, short-term outcomes, and long-term outcomes are the

theoretical underpinnings which guide the treatment services design, the implementation, and the

evaluation (Conrad et al., 1998).

JACSAT\CTRT_ENDAEM\CONCEPTALOGICMDL\DATA\LOGICMDLWPD

10

NEDTAC, Page 3

Description of the Logic Model

Logic models are most effective when presented graphically because the graphic

presentation clearly establishes the interrelationships of the four components. Logic models are

typically a series of boxes in columns, one column each for the conditions, activities, short-term

outcomes, and long-term outcomes. Horizontal lines reflect the interrelationships among the

problem conditions, the treatment service activities, and the expected treatment outcomes. A

simplified hypothetical logic model for a treatment service for pregnant women is illustrated in

Exhibit II-1.

EXHIBIT II-1EXAMPLE OF SIMPLIFIED LOGIC MODEL

CONDITIONS ACTIVITIES OUTCOMES(Short-term)

OUTCOMES(Long-term)

Pregnant womenwho abusesubstances

Treatm entservices for

womenAbstinence Drug-free

lifestyle

Prenatal care Healthy delivery Familyreunification)i

3. USES OF LOGIC MODELS

There are numerous applications for logic model techniques; within the Federal

evaluation environment, four applications predominate. These include: (1) Federal/fundingagency grants management; (2) substance abuse treatment services design and management; (3)

substance abuse treatment services evaluation; and (4) knowledge generation about substance

abuse treatment effectiveness and identification of exemplary or best practices.

One of the assumptions underlying the IEM is that the use of logic models should be a

key component in reviewing grant and cooperative agreement applications, monitoring projects,

and designing and implementing evaluations of knowledge-generating initiatives. SAMHSA,

including CSAT, specifies in the Guidance for Applicants (GFAs) that grant and cooperative

agreement applicants include a logic model within the grant application. The logic model assists

the technical review of the application and is then used by project officers to assess project

implementation and the extent to which the project remains faithful to the project design. Project

JACSATTTRT_ENDUEM\CONCEPT\LOGICMDLOATA\LOGICMDLWPD NEDTAC, Page 4

11

Description of the Logic Model

logic models within a grant program also are used to determine the similarities and differences

across projects as part of a program-wide evaluation (Conrad et al., 1998).

Logic models also assist substance abuse treatment services design and management and

substance abuse treatment services evaluation. Traditionally, logic models are developed in

advance of the treatment service design and the evaluation design. Integration Evaluation

Methods (IEM) efforts expand on this traditional use by incorporating evaluation activities and

products within the logic model process. When developed at the outset, the logic model ensures

that the treatment services staff and the evaluation staff have a shared understanding of the

purpose, components, and expected results of the treatment services, since the logic model:

Clearly identifies treatment service goals, objectives, activities, and desired outcomes

Clarifies assumptions and relationships between treatment services efforts andexpected results

Helps to specify what to measure through evaluation, when, and why

Aids in determining how to link process evaluation measures and outcome evaluationmeasures

Guides the assessment of underlying assumptions and facilitates self-correction of thetreatment services (Kumpfer et al., 1993).

In addition, developing the treatment services logic model jointly by treatment services and

evaluation staff assists in building consensus and a common understanding of treatment service

provision. The value of the logic model is maximized since the design and development of

treatment services and treatment service evaluation are fully coordinated.

The next two sections of this paper describe the value of logic models for treatment

service planning and for treatment service evaluation planning. In fact, the development of logic

models for treatment services and their evaluations must be coordinated. The interrelationship of

the treatment services logic model and the evaluation logic model is diagrammed in Exhibit 11-2.

J:\CSAT\CTRT_END\IEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 5

12

Iden

tify

and

mea

sure

clie

nt s

tatu

s at

fol

low

up(6

or

12 m

onth

s af

ter

disc

harg

e) to

mea

sure

trea

tmen

t eff

ectiv

enes

s an

dim

pact

.A

naly

ze c

osts

in r

elat

ion

to c

lient

out

com

es.

C.

CO

ND

ITIO

NS

EX

HIB

IT 1

1-2

INT

ER

RE

LA

TIO

NSH

IP O

F L

OG

IC M

OD

EL

S FO

R P

LA

NN

ING

TR

EA

TM

EN

T S

ER

VIC

ES

AN

D E

VA

LU

AT

ION

AC

TIV

ITIE

SO

UT

CO

ME

S(S

hort

-ter

m)

OU

TC

OM

ES

(Lon

g-te

rm)

TR

EA

TM

EN

TP

LAN

NIN

GPr

ogra

m g

oals

and

obje

ctiv

esT

reat

men

t mod

els,

mod

aliti

es, s

yste

ms

Lin

kage

s w

ithco

mm

unity

pro

vide

rs

RE

SO

UR

CE

PLA

NN

ING

Fund

ing

- St

affi

ngFa

cilit

ies

EV

ALU

AT

ION

PLA

N

Iden

tify

eval

uatio

n go

als

and

obje

ctiv

esD

evel

op e

valu

atio

n qu

estio

nsan

d m

easu

res

whi

ch, i

n tu

rn,

dete

rmin

e pr

ojec

t dat

a co

llect

ion

and

repo

rtin

g re

quir

emen

ts

OU

TR

EA

CH

Iden

tifyi

ngta

rget

popu

latio

nsC

LIE

NT

INT

AK

E

Scre

enin

g, a

sses

smen

tR

efer

ral

Tre

atm

ent p

lann

ing

TR

EA

TM

EN

T/A

FT

ER

CA

RE

Prov

ider

Man

ager

s/St

aff

Mon

itor

proj

ect p

erfo

rman

cePr

ovid

e se

rvic

esD

ocum

ent p

roje

ct in

puts

, pro

cess

es,

outc

omes

Clie

nts

Rec

eive

ser

vice

sC

ompl

ete

trea

tmen

t/ter

min

ate

PR

OC

ES

S E

VA

LUA

TIO

N

Ass

ess

proj

ect i

mpl

emen

tatio

nPr

ovid

e pr

ojec

t mon

itori

ngC

ontr

ibut

e to

sel

f-ad

just

ing

trea

tmen

t pro

cess

Ana

lyze

uni

t cos

ts

TR

EA

TM

EN

T E

XIT

ST

AT

US

Tre

atm

ent c

ompl

etio

nSu

bsta

nce

use/

abus

eC

rim

inal

act

ivity

Soci

al a

nd f

amily

func

tioni

ngM

enta

l and

phy

sica

lhe

alth

Em

ploy

men

t/ec

onom

ic s

tatu

sH

ousi

ng/li

ving

cond

ition

s-

Clie

nt s

atis

fact

ion

LON

G-T

ER

MO

UT

CO

ME

S

Red

uce

reci

divi

smR

educ

e su

bsta

nce

abus

eR

educ

e cr

imin

al a

ctiv

ityIn

crea

se e

mpl

oyab

ility

econ

omic

sta

tus

Incr

ease

soc

ial/f

amily

func

tioni

ngIn

crea

se m

enta

l and

phys

ical

hea

lth

OU

TC

OM

E E

VA

LUA

TIO

N.

Inte

rim

Out

com

esL

ong-

term

Out

com

es

Iden

tify

and

mea

sure

clie

nt s

tatu

s at

disc

harg

e co

mpa

red

to s

tatu

s at

inta

ke

1314

Ill. USING LOGIC MODELS TO PLAN KNOWLEDGE-GENERATING

TREATMENT SERVICES

Logic models are vital to the design and development of substance abuse treatment

services and useful for services management and monitoring, and for knowledge-generating

activities specific to a knowledge development or knowledge application activity. Since

substance abusing behaviors are influenced by a variety of factors, treatment services typically

attempt to address multiple factors, simultaneously. By assessing the conditions (including the

target population), framing the problem statement(s), designing the treatment services, and

specifying the expected outcomes, up front, the treatment services will be designed coherently.

The logic model then supports the management of the treatment services by: (1) providing a tool

to assess implementation so as to ensure that the services are implemented as planned;

(2) maintaining the integrity of the treatment services to the design, overtime; and (3) monitoring

treatment service costs.

A framework for developing a treatment services logic model using treatment services for

pregnant women as an example is presented in Exhibit III-1. The principles that guide the

development of the treatment services logic model are described, below.

1. CONDITIONS AND CONTEXT FOR KNOWLEDGE-GENERATINGTREATMENT SERVICES

In designing substance abuse treatment services, a first step is to identify the conditions

for the treatment services, define the "problem" to be addressed by the treatment interventions

and the knowledge-generating activity, and specify the assumptions on which the treatment

services design will be based.

Conditions which may influence the type of treatment services that should be offered

and/or that may affect the treatment services outcomes include demographic characteristics and

the substances being abused. For example, gender has been found to influence treatment service

access, entry, retention, and outcomes. It also influences the treatment services that are needed to

maximize treatment success. Therefore, treatment services for women should include targeted

outreach efforts and specific services that address women's unique medical and familial needs.

This also is true for other treatment populations such as adolescents, injection drug users, and

others. Also, different substance addictions respond differently to different treatments. For

example, some addictions respond better to pharmacological treatment while others respond

better to individual and group therapies. To ensure that treatment services are designed

JACSAT\CTRT_ENINEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 7

15

EX

HIB

IT I

II-1

SAM

PLE

LO

GIC

MO

DE

L F

OR

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T S

ER

VIC

ES

FOR

PR

EG

NA

NT

WO

ME

N

CO

ND

ITIO

NS

Soc

ial/e

nviro

nmen

tal

cont

ext

Lim

ited

trea

tmen

tav

aila

ble

Poo

r ac

cess

Tar

get p

opul

atio

n

Dem

ogra

phic

s

Sub

stan

ces

abus

ed

Ava

ilabl

e re

sour

ces

Gra

nts

Sta

te, l

ocal

fund

ing

Clie

nt fe

es

Thi

rd p

arty

pay

ers

16

AC

TIV

ITIE

S

Out

reac

h

Tre

atm

ent a

nd s

ervi

ces

coor

dina

tion

Inta

ke a

nd a

sses

smen

t

Tre

atm

ent p

lann

ing

and

man

agem

ent

Tre

atm

ent s

ervi

ces

tom

eet f

undi

ng a

ndta

rget

pop

ulat

ion

requ

irem

ents

OU

TC

OM

ES

(SH

OR

T-T

ER

M)

Incr

ease

d se

rvic

espr

ovis

ion

Impr

oved

trea

tmen

tne

twor

ks

Incr

ease

d co

stef

fect

iven

ess

Tar

gete

d tr

eatm

ent

plan

s

Incr

ease

d tr

eatm

ent

com

plet

ion

Incr

ease

d cl

inic

ian

effe

ctiv

enes

s

Incr

ease

d pe

riods

of

abst

inen

ce

Incr

ease

d st

abili

ty o

fhe

alth

/men

tal h

ealth

Incr

ease

d m

otiv

atio

nfo

r tr

aini

ng a

ndem

ploy

men

t

OU

TC

OM

ES

(LO

NG

-TE

RM

)

Impr

oved

bal

ance

of

trea

tmen

t nee

ds (

dem

ands

)an

d pr

ovis

ion

(sup

ply)

thro

ugh

mor

e ta

rget

ed/

inte

grat

ed s

ervi

ces

Dec

reas

ed s

ubst

ance

abus

e

Impr

oved

inde

pend

ent

livin

g

Fam

ily r

euni

ficat

ion

Sus

tain

abili

ty o

ftr

eatm

ent s

ervi

ces/

prov

ider

inde

pend

ent o

fgr

ant f

unds

Cos

t effe

ctiv

enes

s of

trea

tmen

t ser

vice

s 17

Using Logic Models to Plan Knowledge-Generating Treatment Services

appropriately, a literature review should inform the design based on tested theories or practices

that have proved successful for a specific group or substance of abuse.

The problem statement should include behaviors and conditions that can be affected by

the knowledge-generating treatment intervention. For example, lack of financial resources, or

poverty, may be a factor in an individual's substance abuse but treatment services cannot treat

poverty, per se. Rather, treatment services can provide education and job skills training which,

in turn, can lead to improved financial conditions.

Another important condition for substance abuse treatment services is the availability of

resources to fund the services and community resources to support the services. Federal, state,

and other resources are generally available but may be reserved for specific populations and/or

types of substances being abused. Also, existing community substance abuse treatment resources

may suggest strategies for inter-agency networking, coordination, and cooperation. In addition,

funding sources typically have associated regulations and policies which should be accounted for

by the substance abuse treatment services design.

A needs assessment is frequently used to identify the type and extent of existing problems

within the community, the services available, and the unmet needs. A needs assessment is a

process to determine the need, which can be defined as the gap between the problem and existing

resources to address the need (Linney et al., 1991).

2. KNOWLEDGE-GENERATING TREATMENT SERVICES ACTIVITIES

A substance abuse treatment services logic model requires the specification of two types

of activities: inputs to the treatment activities and service implementation. Inputs include thespecification of treatment service goals and objectives; identification of treatment models;

establishment of linkages with other community resources; and the treatment resources including

funding, staffing, facilities, and costs. Specifying the services implementation for the logic

model includes a listing of the specific services to be provided. Again, these services should

logically flow from the conditions (including target population needs), the project goals and

objectives, and the treatment model being adopted (Devine et al., revised 1999).

J:\ CSAT \CTRT_END \IEM \CONCEPT\LOGICMDL \DATA \LOGICMDL.WPD NEDTAC, Page 9

18

Using Logic Models to Plan Knowledge-Generating Treatment Services

3. KNOWLEDGE-GENERATING TREATMENT SERVICES OUTCOMES

The logic model should be carefully crafted so as to appropriately anticipate and

distinguish between short-term outcomes and long-term outcomes. The long term or ultimate

outcomes of the treatment services are sometimes shown in the logic model as the treatment

service goals. However they are stated, it is advantageous to identify stages of desired outcomes

and to differentiate between short-term and long-term treatment services outcomes.

For example, in residential treatment services designed to treat substance-abusing

pregnant women, the short-term treatment outcome effects may be an increased community

awareness, an increased number of mothers who receive prenatal care in the first trimester of the

pregnancy, an increased number of healthy births, a reduced number of low birth weight babies,

and a decreased number of women who use alcohol and drugs. The long-term outcomes or

ultimate treatment service goals may be to reduce infant morbidity and mortality, reduce mental

defects in newborns, maintain parental sobriety, and increase family reunification. In addition to

short-term and long-term client outcomes, the knowledge-generating treatment service may be

designed to have an impact on the community. For example, the location of the treatment service

may be determined by the community's greatest need for additional treatment services. An

expected short-term outcome may be a reduction in the number of people on treatment service

waitlists. The longer-term community or systemic outcomes may be reduced health care costs

since community residents are receiving substance abuse treatment services more promptly, and

the outcome of the treatment service is reduced need for primary health care.

It is important to state outcomes with as much specificity as possible so that they can be

measured. Outcomes should explain what problems the substance abuse treatment services are

attempting to eliminate, and, where possible, should include time frames and conditions under

which the outcomes are expected to occur. For example, a short-term outcome for the residential

treatment services for women may be to maintain 120 consecutive days in the treatment services.

A long-term outcome for these services may be to maintain 2 years of abstinence.

J:\ CSAT \CTRT_END \IEM\CONCEPT\LOGICMDL \DATA \LOGICMDL.WPD NEDTAC, Page 10

19

IV. USING LOGIC MODELS FOR EVALUATING

KNOWLEDGE-GENERATING ACTIVITIES

Logic models are essential to interpreting evaluation findings. The evaluator needs a

clear understanding of the treatment services goals, implementation sequences, and expected

links among the treatment population characteristics, services, and expected benefits so as to

accurately interpret the evaluation results (Wholey, 1979). An example of a logic model

framework for use in developing a substance abuse treatment services evaluation is presented in

Exhibit IV-1 and described below.

1. CONDITIONS AND CONTEXT FOR THE KNOWLEDGE-GENERATINGEVALUATION

Knowledge-generating substance abuse treatment services evaluations must understand

the environment in which the treatment services are operating and the new knowledge that can be

obtained. The conditions component of the logic modeling process assists in planning the

evaluation of the treatment population and the environment. First, the evaluation can assess the

target population in terms of prevalence and treatment needs. Services for pregnant women, for

example, would be under-utilized if offered in communities with few substance-abusing pregnant

women. Further, an evaluation designed to use random assignment must assess prevalence so as

to design the client flows for treatment and ensure that the client flows for the treatment

alternatives will yield sufficient sample sizes.

The evaluation also must be fully informed about the local treatment resources, funding

sources, and associated regulations and policies. Funding sources and/or alternative community

treatment resources may influence the implementation of the treatment services being evaluated

and the evaluation must take these confounding variables into account when interpreting the

evaluation results.

Logic models also are useful in assessing the evaluability of substance abuse treatment

services. Evaluability assessments are the pre-evaluation analyses that help to ensure that an

evaluation will be technically feasible and capable of answering the evaluation and research

questions important to decision makers. The evaluability assessment lays the groundwork for a

successful evaluation by:

J: \CSAT\CTRT_ END \IEM \CONCEPT\LOGICMDL \DATA \LOGICMDL.WPD NEDTAC, Page 11

20

EX

HIB

IT I

V-1

SAM

PLE

LO

GIC

MO

DE

L F

OR

EV

AL

UA

TIO

N O

F K

NO

WL

ED

GE

-GE

NE

RA

TIN

G T

RE

AT

ME

NT

SE

RV

ICE

S

-->

-

CO

ND

ITIO

NS

AC

TIV

ITIE

SO

UT

CO

ME

S(S

HO

RT

-TE

RM

)O

UT

CO

ME

S(L

ON

G-T

ER

M)

Nee

ds A

sses

smen

t(W

hat p

robl

ems

does

the

trea

tmen

t add

ress

?)

Proc

ess

Eva

luat

ion

(Wha

t did

the

trea

tmen

tpr

ovid

er d

o?)

Out

com

e E

valu

atio

n(D

id th

e tr

eatm

ent s

ervi

ces

achi

eve

obje

ctiv

es?)

Impa

ct E

valu

atio

n(D

id th

e tr

eatm

ent s

ervi

ces

achi

eve

long

-ter

m g

oals

?)

Tar

get p

opul

atio

n:

Ass

ess

prev

alen

ce

Det

erm

ine

feas

ibili

tyof

clie

nt f

low

Soci

al/e

nvir

onm

enta

lco

ntex

t:

Iden

tify

fund

ing

sour

ces;

ass

ocia

ted

regu

latio

ns a

nd p

olic

ies

Iden

tify

com

mun

itytr

eatm

ent r

esou

rces

and

rela

tions

hips

totr

eatm

ent b

eing

eval

uate

d

Eva

luab

ility

ass

essm

ent:

Exa

min

e tr

eatm

ent

serv

ice

cont

ext

Det

erm

ine

abili

ty o

ftr

eatm

ent s

ervi

ces

topa

rtic

ipat

e in

eva

luat

ion

Cla

rify

trea

tmen

tse

rvic

e go

als

Cre

ate

eval

uatio

n go

als

and

obje

ctiv

es

Con

duct

lite

ratu

rere

view

to e

stab

lish

theo

retic

al li

nkag

es

Dev

elop

eva

luat

ion

ques

tions

and

mea

sure

s to

:

Doc

umen

tim

plem

enta

tion

sequ

ence

s

Det

erm

ine

clin

icia

npr

actic

es

Cal

cula

te tr

eatm

ent

cost

s

Mea

sure

clie

ntou

tcom

es

Col

lect

and

ana

lyze

data

at t

reat

men

t int

ake,

in-t

reat

men

t, tr

eatm

ent

exit

SDU

des

crip

tion

Clin

icia

n ba

ckgr

ound

,ph

iloso

phy,

and

prac

tices

Tre

atm

ent c

osts

Clie

nt d

emog

raph

ics,

know

ledg

e, b

ehav

iors

,an

d at

titud

es

Col

lect

and

ana

lyze

clie

nt d

ata

at f

ollo

w-u

pto

det

erm

ine

last

ing

trea

tmen

t im

pact

s

Con

duct

com

mun

ityas

sess

men

ts to

dete

rmin

e la

stin

gim

pact

of

trea

tmen

tse

rvic

es o

n co

mm

unity

cond

ition

s (e

.g.,

subs

tanc

e ab

use,

crim

inal

act

ivity

, hea

lthan

d m

enta

l hea

lthco

nditi

ons)

Con

duct

cos

t ana

lyse

san

d co

st b

enef

itan

alys

es

2122

Using Logic Models for Evaluation Knowledge-Generating Activities

Establishing agreement with the treatment service provider and the evaluator on thehypothesized causal links between the treatment services design and the intendedintermediate and long term outcomes

Providing an organizing framework for measurement and data collection so as toensure that all data necessary to test the hypotheses and meet decision-maker's needswill be collected

Building consensus between the funder, the treatment director, the clinical director,and the evaluators on data collection procedures and ensuring that adequate datacollection systems are in place or will be established.

During the evaluability assessment, the evaluator examines the context for the substance abuse

treatment services and the evaluation to see if a rigorous, objective evaluation is possible and the

logic modeling process supports this activity (Wholey, 1994; Conrad et al., 1998).

2. KNOWLEDGE-GENERATING SUBSTANCE ABUSE TREATMENT SERVICES

The evaluation logic model supports the evaluation of the substance abuse treatment

services activities in five ways: (1) clarifying treatment service goals; (2) identifying the

underlying treatment theories; (3) providing a framework for organizing the process evaluation;

(4) providing a framework for integrating the process and outcome analyses; and (5) ensuring the

knowledge-generating goals and objectives are met.

Clarifying treatment service goals. Most substance abuse treatment services havemultiple treatment goals. Logic models assist the evaluation process by identifying the treatment

services goals that are most important to the evaluation and for which there are sound

measurement methods. For knowledge-generating activities, the new knowledge to be gained isthe primary treatment service goal.

Identifying underlying treatment theories. Many treatment services are developedbased on the experiences of the treatment providers. Implicit in these treatment services,

however, are underlying theories and hypotheses about the characteristics of the treatment

population, the substances that are abused, and effective methods for addressing specific

population and/or substance characteristics. An evaluation that is predicated on treatment theory

or hypotheses will support more targeted measurement and/or the interpretation of evaluation

findings within a theoretical construct (Orwin, 1998). The logic modeling process assists the

knowledge-generating activity in identifying the underlying treatment services theory by defining

JACSAT\CTRT_ENDVIEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 13

23

Using Logic Models for Evaluation Knowledge-Generating Activities

the assumptions and the linkages among the assumptions, treatment services activities, and the

treatment services outcomes.

Providing the process evaluation framework. The logic model provides theframework for the process evaluation by delineating all of the treatment service elements that

must be documented so as to fully understand the treatment. For knowledge-generating

activities, a fully developed logic model describes the treatment services in detail and identifies

the data that should and should not be collected as part of the process evaluation.

Providing the framework for integrating process and outcomes analyses. Similarly,the logic model provides the framework for integrating the evaluation analysis components. As

demonstrated, the process evaluation measures the knowledge-generating treatment service

activities while the outcome evaluation measures the treatment service short-term and long-term

outcomes associated with the knowledge-generating activities. The analyses, however, are not

conducted separately or in isolation of each other. Rather, the process evaluation analyses

provides the context for interpreting the outcome results as well as a conduit for identifying the

need to change the treatment design and/or the treatment activities. The outcome analysis

provides "red flags" when short-term or long-term outcomes are different than expected. In

addition, the process evaluation will provide critical variables and data that will need to be added

to the outcome database for analysis.

Ensuring that the knowledge-generating goals and objectives are met. The CSATKD&A program establishes the goals and objectives for the knowledge-generating treatment

services. The logic model is useful to C SAT, as well as the other stakeholders, in ensuring that

the overall CSAT goals and objectives are accomplished since the logic model is constructed on

the basis of a logical sequencing of treatment services so as to achieve the goals and objectives.

3. KNOWLEDGE-GENERATING SUBSTANCE ABUSE TREATMENTOUTCOMES

In addition to providing the framework for the process evaluation, the logic model guides

the outcomes evaluation. First, the logic model clarifies the treatment services goals that are

relevant to the treatment services outcomes, identifies measurable outcomes, and delineates the

target population and the treatment interventions.

JACSATTTRT_ENDVEM\CONCEPT\LOGICMDL\DATATOGICMDLWPD NEDTAC, Page 14

24

Using Logic Models for Evaluation Knowledge-Generating Activities

Second, if the control or comparison group to be used in the evaluation outcomes analysis

is receiving alternative treatment services, the logic model is a critical tool to identify the

similarities and differences between the treatment services being evaluated and the alternative

treatment services. These differences in treatment service provisions form the basis for the

expected differences between the treatment and comparison group differences (Conrad et al.,

1998) and will shape the process and outcome evaluation designs.

JACSAT\CTRT_ENDVEM\CONCEPIILOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 15

25

V. USING LOGIC MODELS TO SPECIFY KNOWLEDGE-GENERATING

EVALUATION MEASURES

The logic modeling process can be used to develop a framework for specifying the

evaluation questions, measures, data sources/instruments, and data collection time points.

Within the Integrated Evaluation Methods approach, this framework is called a data map, and its

purpose is to identify and clarify the evaluation data requirements.

1. DATA MAP STRUCTURE AND UTILITY

A data map very clearly lays out the specifications for evaluation data and the data

collection plans. The data map provides a bridge between the logic model and the data collection

activities and establishes the infrastructure for the data analysis plans. It is used to show

substance abuse treatment services staff the evaluation data requirements and the rationale for

these requirements. Data maps provide a structured format to answer the primary, secondary,

and tertiary questions that treatment providers most often ask the evaluation staff: "Why do we

have to collect these data?" and "How will you use the data once they are collected?"

2. DATA MAP DEVELOPMENT

The five steps in developing a data map for a knowledge-generating activity include:

(1) identifying the over-arching evaluation questions; (2) developing the secondary and tertiary

evaluation questions; (3) identifying the measures needed to answer the primary, secondary, and

tertiary evaluation questions; (4) identifying the data sources and instruments; and (5)

determining the data collection time points.

A sample data map that parallels an IEM initiative is presented in Exhibit VI-1 at the

conclusion of this paper. The sample data map in Exhibit VI-1 is designed to be used as a

template for creating the evaluation plan for a knowledge-generating activity using the IEM

package. Therefore, the following paragraphs describe the steps to creating a data map using the

sample data map as an example. The sample data map is illustrative of the data mapping

process; it is not intended to be comprehensive. In reality, the data map must be tailored to the

specific knowledge-generating activity with the evaluation objectives and questions reflective of

the specific activity.

JACSAT\CTRT_ENMEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 16

26

Using Logic Models to Sped Knowledge-Generating Evaluation Measures

2.1 Identifying Over-Arching Evaluation Questions

The over-arching evaluation questions are statements, in question form, that must

ultimately be answered by the evaluation. These questions should be tied directly to the

knowledge-generating activity goals and objectives and to the logical relationship (logic model

structure) between the conditions, activities, and outcomes. Exhibit VI-1 demonstrates how the

logic model structure guides the development of the over-arching questions:

Conditions for establishing the knowledge-generating activity are identified andassessed by the basic, over-arching question: What is the design for the knowledge-generating substance abuse treatment service?

Activities necessary for generating knowledge about the substance abuse treatmentservice are identified and assessed by the questions:

How was the knowledge-generating substance abuse treatment serviceimplemented?

How does the treatment service relate to the original design?

Who did the knowledge- generating substance abuse treatment service serve?

What were the resource requirements and costs of the knowledge-generatingsubstance abuse treatment service?

Outcomes, both short-term and long-term, are assessed by answering the question:What were the knowledge-generating substance abuse treatment serviceoutcomes? and long-term outcomes are further assessed by answering the question:What is the relationship between the costs and outcomes of the knowledge-generating substance abuse treatment service?

As shown in the example, the over-arching questions are presented at the top of each sub-section

in the matrix. Again, these basic, over-arching questions, while germane to all knowledge-

generating treatment services, should be substantively adapted to the specific knowledge-

generating goals, objectives, and study questions.

2.2 Defining Knowledge-Generating Secondary Evaluation Questions

Secondary evaluation questions are precise, measurable statements (in question format)

of what the evaluation intends to achieve and are based on the over-arching evaluation questions

J: CSATTTRT_ENDUEM \CONCEPT\ LOGICMDL DATATOGICMDLWPD NEDTAC, Page 17

27

Using Logic Models to Specify Knowledge-Generating Evaluation Measures

and on the treatment service objectives. Secondary evaluation questions provide a way to

incrementally measure achievement of the knowledge-generating goals and objectives and

provide a linkage to the overarching evaluation measures. Secondary evaluation questions

address the substance abuse treatment services effort, or process, and cover treatment services

operations, service delivery, and use of resources. Secondary evaluation questions also address

treatment effectiveness, or outcomes, and assess the treatment services' impact on clients and/or

service delivery network. Evaluation objectives and questions should be stated in terms that can

be measured. For example, to answer the question "What is the appropriate length of treatment

in terms of outcomes for different individuals and family units?" measures for treatment delivery

and treatment outcomes are needed. (Tertiary evaluations reflect a further refinement and

specification of the evaluation process. These same "rules" apply to the development of tertiaryevaluation questions.)

2.3 Identifying Evaluation Measures

Once the secondary (and tertiary) evaluation questions are defined, the measures needed

to answer the questions can be specified. Collectively, the evaluation measures provide

information on treatment service design, implementation, operations, service delivery, costs,

client behavior, client attitudes, and client experiences. These measures require information

about the service delivery unit; the clinician background and therapeutic approach; treatment

costs; and client characteristics, behaviors, and attitudes. Examples of measures needed to

answer the primary, secondary, and tertiary evaluation questions are specified in Exhibit VI-1.

This process supports the ultimate determination of whether the treatment services goals are met.

Each of the measures identified in the sample data map are further defined and operationalized

with data definitions and response categories from the IEM companion document: Minimum

Evaluation Data Set: Core Data Lists. (See the appendix to this document.)

2.4 Identifying Data Sources/Instruments

Substance abuse treatment services data are typically obtained from the service delivery

unit (including the provider director, clinician, and financial management staff) and from the

clients. Systems level data about the environment in which the treatment services operate are

obtained from linkage partners, collaborating agencies, and other community agencies or

community-based data sources. For each of the data sources, appropriate data collection

instruments are necessary and may include survey formats, interview formats, and protocols to

abstract records-based data and databases (criminal activity, employment, etc.). Examples of

JACSAT\CTRT_ENDVEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 18

28

Using Logic Models to Sped Knowledge-Generating Evaluation Measures

data sources and types of instruments are presented in Exhibit VI-1. The data

sources/instruments listed in the sample data map (Exhibit VI-1) are more fully described and

discussed in the IEM companion document: Guide to Process Evaluation for Substance Abuse

Treatment Services. (See the appendix to this document.)

2.5 Determining Data Collection Time Points

Generally, systems and SDU data are collected at baseline, quarterly, and annually; and

client-level data are collected at specified intervals. The IEM advocates the collection of client-

level data at four time points, including: intake (treatment entry), within treatment, treatment

exit, and at follow-up (3, 6, 12, and/or 18 months following treatment exit). It is recognized,

however, that data collection activities are the most resource intensive of all evaluation

components and the ability of the evaluation to conform to the IEM recommended approach is

directly linked to the evaluation budget and resources.

An important role for the evaluation planning process is the capability to adjust the

evaluation components, as needed, when faced with new information. Thus, after the data map

has been developed, the knowledge-generating evaluation questions will probably need to be re-

examined to clarify and finalize the evaluation plan.

JACSAT\CTRT_ENDMEM\CONCEPT\LOGICMDL\DATATOGICMDLWPD NEDTAC, Page 19

29

VI. SUMMARY

The list below provides a summary of recommended activities to aid in the development

of logic models:

Clarify the knowledge-developing substance abuse treatment services from theperspective of the KD or KA program, treatment services managers and staff,evaluators, and other key stakeholders

Explore treatment services "reality," including the plausibility and measurability oftreatment-site services goals and objectives in light of the knowledge-generatingtreatment goals and objectives

Involve intended users of evaluation information to determine the knowledge-generating evaluation priorities and intended uses of evaluation information onsubstance abuse treatment services performance

Apply logic model techniques to knowledge-generating substance abuse treatmentservice planning and implementation and to evaluation planning and implementation

Apply logic model techniques to data map development to ensure that thespecification of knowledge-generating evaluation questions, measures, and variablesare linked to the treatment goals and objectives and the knowledge-generating goalsand objectives.

The development of logic models is critical to ensuring that the knowledge-generating substance

abuse treatment services evaluation can be carried out in a way that will yield accurate and useful

information to document treatment effectiveness and improve treatment services and activities.

A review of the activities listed above shows the close relationship between the development of

the logic models, planning the treatment services, and planning the treatment services evaluation.

Applying the ideas presented in this paper will improve the process of substance abuse treatment

services evaluation that support knowledge generation and lead to the acquisition of new

knowledge so as to identify exemplary treatment services and to realize systemic and treatment

services improvements.

JACSATTTRT_ENDUEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 20

30

EX

HIB

IT V

I-1

SAM

PLE

DA

TA

MA

P FO

R K

NO

WL

ED

GE

-GE

NE

RA

TIN

G S

UB

STA

NC

E A

BU

SE T

RE

AT

ME

NT

EV

AL

UA

TIO

N

EV

AL

UA

TIO

N.

QU

EST

ION

SE

VA

LU

AT

ION

ME

ASU

RE

S/V

AR

IAB

LE

SD

AT

A S

OU

RC

E/I

NST

RU

ME

NT

,

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S.

Bas

elin

eQ

uart

erly

Ann

ually

Ql.

Wha

t is

the

desi

gn f

or th

e kn

owle

dge-

gene

ratin

g su

bsta

nce

abus

e tr

eatm

ent s

ervi

ce9

Wha

t is

the

cont

ext f

or th

eF

undi

ng s

ourc

esN

eeds

ass

essm

ents

subs

tanc

e ab

use

trea

tmen

tS

ervi

ce d

eliv

ery

area

Pro

cess

eva

luat

ion

inte

rvie

ws

serv

ice?

Com

mun

ity e

cono

mic

cha

ract

eris

tics

Rou

tine

man

agem

ent r

epor

tsS

peci

fic s

ubst

ance

abu

se p

robl

ems

Oth

er r

elat

ed s

ervi

ces

in th

e ar

ea

Wha

t wer

e th

e or

igin

s of

Des

crip

tion

of p

revi

ous,

sim

ilar

trea

tmen

t ser

vice

sN

eeds

ass

essm

ents

the

trea

tmen

t ser

vice

?E

vide

nce

of p

revi

ous

serv

ice

succ

ess

Pro

cess

eva

luat

ion

inte

rvie

ws

Sim

ilarit

ies,

diff

eren

ces

to p

revi

ous

serv

ice

Rou

tine

man

agem

ent r

epor

tsN

eeds

ass

essm

ent r

esul

ts

Wha

t is

the

trea

tmen

tP

rogr

am/tr

eatm

ent t

heor

yN

eeds

ass

essm

ents

serv

ice

ratio

nale

, goa

ls, a

ndT

reat

men

t ser

vice

goa

ls a

nd o

bjec

tives

Tre

atm

ent l

itera

ture

obje

ctiv

es?

Tar

get p

opul

atio

n ch

arac

teris

tics

Pro

cess

eva

luat

ion

inte

rvie

ws

Spe

cial

nee

ds o

f tar

get p

opul

atio

nsR

outin

e m

aint

enan

ce r

epor

tsS

DU

inst

rum

ent

Wha

t was

the

trea

tmen

tA

ctiv

ities

and

sta

ff in

volv

ed in

dev

elop

men

t pro

cess

Nee

ds a

sses

smen

tsse

rvic

e de

velo

pmen

tT

reat

men

t ser

vice

des

ign

logi

c m

odel

Pro

cess

eva

luat

ion

inte

rvie

ws

proc

ess?

Pro

blem

s ex

perie

nced

and

res

olut

ion

Rou

tine

man

agem

ent r

epor

tsK

ey d

ecis

ions

SD

U in

stru

men

tC

linic

ian

Bac

kgro

und

& P

ract

ice

31B

ES

T C

OP

Y A

VA

ILA

BLE

32

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

EV

AL

UA

TIO

N M

EA

SUR

ESN

AR

IAB

LE

SD

AT

A S

OU

RC

E/I

NST

RU

ME

NT

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Bas

elin

eQ

uart

erly

Ann

ually

Q2.

How

was

the

know

ledg

e-ge

nera

ting

subs

tanc

e ab

use

trea

tmen

t ser

vice

impl

emen

ted?

How

doe

s th

e tr

eatm

ent s

ervi

ce r

elat

e to

the

orig

inal

des

ign?

Wha

t was

the

Len

gth

of ti

me

trea

tmen

t ser

vice

in o

pera

tion

Proc

ess

eval

uatio

n in

terv

iew

sim

plem

enta

tion

proc

ess?

Impl

emen

tatio

n pr

oces

sR

outin

e m

anag

emen

t rep

orts

Cha

nges

to tr

eatm

ent s

ervi

ce d

esig

nSD

U in

stru

men

tFa

ctor

s th

at f

acili

tate

d ef

fect

ive

impl

emen

tatio

nC

linic

ian

Bac

kgro

und

& P

ract

ice

Fact

ors

that

impe

ded

effe

ctiv

e im

plem

enta

tion

How

do

clie

nts

gain

acc

ess

Proc

ess

that

fac

ilita

tes

clie

nt a

cces

sPr

oces

s ev

alua

tion

inte

rvie

ws

to th

e tr

eatm

ent s

ervi

ce?

Des

crip

tion

of C

entr

aliz

ed I

ntak

e U

nit (

if a

pplic

able

)R

outin

e m

anag

emen

t rep

orts

Mat

ch b

etw

een

trea

tmen

t ser

vice

s an

d cl

ient

nee

dsSD

U in

stru

men

tW

aitin

g lis

tsle

ngth

of

time

Clin

icia

n B

ackg

roun

d &

Pra

ctic

e

Wha

t are

the

linka

ges

with

Rel

atio

nshi

ps w

ith o

ther

age

ncie

sPr

oces

s ev

alua

tion

inte

rvie

ws

the

com

mun

ity a

nd o

ther

Impa

ct o

n tr

eatm

ent s

ervi

ces

of o

ther

age

ncy

Rou

tine

man

agem

ent r

epor

tsag

enci

es?

rela

tions

hips

SDU

inst

rum

ent

Clie

nt r

efer

rals

and

ref

erra

l sou

rces

Clin

icia

n B

ackg

roun

d &

Pra

ctic

e

Wha

t is

the

trea

tmen

tO

rgan

izat

iona

l str

uctu

rePr

oces

s ev

alua

tion

inte

rvie

ws

serv

ice

orga

niza

tion,

staf

fing

, and

man

agem

ent?

Stee

ring

com

mitt

ee c

hara

cter

istic

sT

reat

men

t sta

ff c

hara

cter

istic

sR

outin

e m

anag

emen

t rep

orts

SDU

inst

rum

ent

Staf

f de

velo

pmen

t eff

orts

Clin

icia

n B

ackg

roun

d &

Pra

ctic

eSt

aff

recr

uitm

ent,

turn

over

, vac

ancy

rat

esE

ffor

ts to

impr

ove

staf

f re

tent

ion

Staf

f m

oral

eIn

form

atio

n fl

ow w

ithin

and

out

side

the

orga

niza

tion

Nat

ure

of in

form

atio

n co

llect

ed a

nd u

ses

Mai

nten

ance

of

wai

t lis

tsT

reat

men

t qua

lity

assu

ranc

e pr

oced

ures

33B

ES

T C

OP

YA

VA

ILA

BLE

34

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

EV

AL

UA

TIO

N M

EA

SUR

ES/

VA

RIA

BL

ES

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Bas

elin

eQ

uart

erly

Ann

ually

Wha

t are

the

trea

tmen

tT

reat

men

t ser

vice

com

pone

nts

Pro

cess

eva

luat

ion

inte

rvie

ws

serv

ice

com

pone

nts?

Num

ber

clie

nts

rece

ivin

g sp

ecifi

c tr

eatm

ent c

ompo

nent

sR

outin

e m

anag

emen

t rep

orts

Num

ber

and

type

of u

nder

-util

ized

ser

vice

sS

DU

inst

rum

ent

Num

ber

and

type

of o

ver-

utili

zed

serv

ices

Clin

icia

n B

ackg

roun

d &

Pra

ctic

eR

elat

ions

hip

of tr

eatm

ent s

ervi

ces

to g

oals

, obj

ectiv

esT

reat

men

t ser

vice

s fo

r ch

ildre

n; c

olla

tera

lsC

hara

cter

istic

s of

trea

tmen

t ser

vice

com

pone

nts

Pro

cess

for

trea

tmen

t pla

nnin

g an

d ex

itN

umbe

r an

d ty

pe o

f anc

illar

y se

rvic

esP

lann

ed le

ngth

of s

tay;

act

ual l

engt

h of

sta

yE

xpec

ted

outc

ome

mea

sure

s; d

ata

colle

cted

Eva

luat

ion

activ

ities

35B

ES

T C

OP

YA

VA

ILA

BLE

36

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

EV

AL

UA

TIO

N M

EA

SUR

ESN

AR

IAB

LE

SD

AT

A S

OU

RC

E/I

NST

RU

ME

NT

DA

TA

_ C

OL

LE

CT

ION

TIM

E P

OIN

TS

Bas

elin

eQ

uart

erly

IA

nnua

lly

Q3.

Who

did

the

know

ledg

e-ge

nera

ting

subs

tanc

e ab

use

trea

tmen

t ser

vice

ser

ve?

Wha

t are

the

char

acte

ristic

sT

arge

t pop

ulat

ion

char

acte

ristic

sN

eeds

ass

essm

ents

of th

e ta

rget

pop

ulat

ion?

Dem

ogra

phic

sP

roce

ss e

valu

atio

n in

terv

iew

s,,/

Dru

g/al

coho

l use

Qua

rter

ly P

rogr

ess

Rep

orts

Fam

ily a

nd li

ving

con

ditio

nsS

DU

inst

rum

ent

Edu

catio

n, e

mpl

oym

ent,

inco

me

Clin

icia

n B

ackg

roun

d &

Pra

ctic

eC

rimin

al/ju

veni

le ju

stic

e sy

stem

invo

lvem

ent

Prio

r su

bsta

nce

abus

e tr

eatm

ent

Men

tal/P

hysi

cal h

ealth

and

oth

er s

ervi

ces

Abu

se h

isto

ryH

igh-

risk

beha

vior

s

Cha

ract

eris

tics

of p

eopl

e se

ekin

g tr

eatm

ent

(Sam

e as

abo

ve)

Wha

t are

the

char

acte

ristic

sC

lient

cha

ract

eris

tics

Gra

nt a

pplic

atio

nsof

the

trea

tmen

t ser

vice

Dem

ogra

phic

sP

roce

ss e

valu

atio

n in

terv

iew

scl

ient

s?D

rug/

alco

hol u

seQ

uart

erly

Pro

gres

s R

epor

tsF

amily

and

livi

ng c

ondi

tions

SD

U in

stru

men

tE

duca

tion,

em

ploy

men

t, in

com

eC

linic

ian

Bac

kgro

und

& P

ract

ice

Crim

inal

/juve

nile

just

ice

syst

em in

volv

emen

tC

lient

Inta

ke In

stru

men

tJ

Prio

r su

bsta

nce

abus

e tr

eatm

ent

Clie

nt In

-tre

atm

ent I

nstr

umen

tM

enta

l/Phy

sica

l hea

lth a

nd o

ther

ser

vice

sC

lient

Tre

atm

ent E

xit I

nstr

umen

tA

buse

his

tory

Hig

h-ris

k be

havi

ors

Num

bers

who

ent

er, l

eave

, com

plet

e tr

eatm

ent

How

do

the

trea

tmen

tC

ompa

rison

of t

arge

t and

trea

tmen

t pop

ulat

ions

Nee

ds a

sses

smen

tsse

rvic

e cl

ient

cha

ract

eris

tics

com

pare

with

the

targ

etpo

pula

tion?

SD

U r

ecor

ds

3'7

BE

ST

CO

PY

AV

AIL

AB

LE38

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

ST

ION

SE

VA

LU

AT

ION

ME

AS

UR

ES

/VA

RIA

BLE

SD

AT

A S

OU

RC

E/IN

ST

RU

ME

NT

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Bas

elin

eQ

uart

erly

IA

nnua

lly

Q4.

Wha

t wer

e th

e re

sour

ce r

equi

rem

ents

and

cos

ts o

f the

kno

wle

dge-

gene

ratin

g su

bsta

nce

abus

e tr

eatm

ent s

ervi

ce?

Wha

t wer

e th

e qu

arte

rlyre

venu

es a

nd c

osts

for

the

subs

tanc

e ab

use

trea

tmen

tse

rvic

e?

Ann

ual r

even

ues

by c

ateg

ory

Fed

eral

gra

nts

Sta

te g

rant

sLo

cal g

rant

sT

hird

par

ty p

aym

ents

Clie

nt fe

esO

ther

Qua

rter

ly e

xpen

ditu

res

by c

ateg

ory

Per

sonn

el, f

ringe

Equ

ipm

ent

Sup

plie

sF

ood

Insu

ranc

eT

rave

lC

ontr

actu

alex

tern

alE

valu

atio

next

erna

lA

ltera

tion/

reno

vatio

nR

ent

Indi

rect

Oth

er

Yea

r to

dat

e ex

pend

iture

s by

cat

egor

y(S

ame

cate

gorie

s)

Sub

stan

ce A

buse

Tre

atm

ent C

osts

Ana

lysi

s an

d A

lloca

tion

Tem

plat

e(S

AT

CA

AT

)

39B

ES

T C

OP

Y A

VA

ILA

BLE

40

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

EV

AL

UA

TIO

N M

EA

SUR

ES/

VA

RIA

BL

ES

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Bas

elin

eQ

uart

erly

Ann

ually

.

Wha

t wer

e th

e qu

arte

rlyal

loca

tions

of c

osts

tose

rvic

es?

Per

cent

of t

otal

per

sonn

el c

ost o

f:In

itial

ass

essm

ent/i

ntak

eM

edic

al e

xam

inat

ion

Psy

cho-

soci

al e

xam

inat

ion

Indi

vidu

al/g

roup

sub

stan

ce a

buse

cou

nsel

ing

Indi

vidu

al/g

roup

men

tal h

ealth

cou

nsel

ing

HIV

test

ing/

coun

selin

gM

edic

al/d

iagn

ostic

ser

vice

sH

ousi

ng/fo

odR

ecor

ds m

anag

emen

tC

linic

al c

ase

man

agem

ent

Net

wor

king

/out

reac

hC

hild

car

e se

rvic

esS

taff

educ

atio

nC

lient

edu

catio

nC

lient

tran

spor

tatio

nP

rogr

am e

valu

atio

nO

ther

Sub

stan

ce A

buse

Tre

atm

ent C

osts

Ana

lysi

s an

d A

lloca

tion

Tem

plat

e(S

AT

CA

AT

)

41B

ES

T C

OP

YA

VA

ILA

BLE

42

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

EV

AL

UA

TIO

N M

EA

SUR

ES/

VA

RIA

BL

ES

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Bas

elin

eQ

uart

erly

Ann

ually

Wha

t wer

e th

e an

nual

allo

catio

ns o

f cos

ts to

serv

ices

?

Per

cent

of t

otal

per

sonn

el c

ost o

f:M

edic

al e

xam

inat

ion

Psy

cho-

soci

al e

xam

inat

ion

Indi

vidu

al/g

roup

sub

stan

ce a

buse

cou

nsel

ing

Indi

vidu

al/g

roup

men

tal h

ealth

cou

nsel

ing

HIV

test

ing/

coun

selin

gM

edic

al/d

iagn

ostic

ser

vice

sH

ousi

ng/fo

odR

ecor

ds m

anag

emen

tC

linic

al c

ase

man

agem

ent

Net

wor

king

/out

reac

hC

hild

car

e se

rvic

esS

taff

educ

atio

nC

lient

edu

catio

nC

lient

tran

spor

tatio

nP

rogr

am e

valu

atio

nO

ther

Sub

stan

ce A

buse

Tre

atm

ent C

osts

Ana

lysi

s an

d A

lloca

tion

Tem

plat

e(S

AT

CA

AT

)

43B

ES

T C

OP

YA

VA

ILA

BLE

44

EX

HIB

IT V

IA (

CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

EV

AL

UA

TIO

N M

EA

SUR

ESN

AR

IAB

LE

S,

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Pre

Dur

ing

Exi

tPo

st

Q5.

Wha

t wer

e th

e kn

owle

dge-

gene

ratin

g su

bsta

nce

abus

e tr

eatm

ent s

ervi

ce o

utco

mes

?

Wha

t wer

e th

e re

tent

ion

rate

sN

umbe

r an

d pe

rcen

t of c

lient

s co

mpl

etin

g an

d no

tC

lient

Inta

ke In

stru

men

tfo

r th

e tr

eatm

ent s

ervi

ceco

mpl

etin

g tr

eatm

ent

Clie

nt In

-Tre

atm

ent I

nstr

umen

tcl

ient

s?C

hara

cter

istic

s of

thes

e cl

ient

sC

lient

Tre

atm

ent E

xit I

nstr

umen

t-

Ave

rage

leng

th o

f sta

y (L

OS

)C

lient

Fol

low

-up

Inst

rum

ent

Per

cent

age

of c

ompl

etio

ns (

actu

al v

s pl

anne

d LO

S)

Ado

lesc

ent I

ntak

e In

stru

men

tR

eten

tion

rate

s pe

r tr

eatm

ent p

opul

atio

nA

dole

scen

t In-

Tre

atm

ent I

nstr

umen

tch

arac

teris

tics

Ado

lesc

ent T

reat

men

t Exi

t Ins

trum

ent

./.R

eten

tion

rate

s pe

r tr

eatm

ent m

odal

ityA

dole

scen

t Fol

low

-up

Inst

rum

ent

Per

cent

age

of c

lient

s pa

rtic

ipat

ing

in a

fterc

are

Chi

ld In

take

Inst

rum

ent

Fac

tors

ass

ocia

ted

with

trea

tmen

t com

plet

ion

and

Chi

ld In

-Tre

atm

ent I

nstr

umen

t./.

non-

com

plet

ion

Chi

ld T

reat

men

t Exi

t Ins

trum

ent

Chi

ld F

ollo

w-u

p In

stru

men

t

Wha

t are

the

char

acte

ristic

s of

the

clie

nts'

sub

stan

ce u

se?

InU

se o

f spe

cific

dru

gs (

incl

udin

g ni

cotin

e, a

lcoh

ol),

ever

-C

lient

Inta

ke In

stru

men

tC

lient

In-T

reat

men

t Ins

trum

ent

wha

t way

s di

d th

e cl

ient

s'N

umbe

r of

day

s us

ed, l

ast 3

0 da

ysC

lient

Tre

atm

ent E

xit I

nstr

umen

tsu

bsta

nce

use

chan

ge?

Fre

quen

cy o

f use

, pas

t 30

days

Clie

nt F

ollo

w-u

p In

stru

men

tM

ost c

omm

on r

oute

of a

dmin

istr

atio

n, p

ast 3

0 da

ysA

dole

scen

t Int

ake

Inst

rum

ent

Com

bina

tion

of d

rugs

use

d, a

t sam

e tim

e, p

ast 3

0A

dole

scen

t In-

Tre

atm

ent I

nstr

umen

tda

ysA

dole

scen

t Tre

atm

ent E

xit I

nstr

umen

tA

ge fi

rst u

sed

Ado

lesc

ent F

ollo

w-u

p In

stru

men

tC

urre

nt d

iagn

osis

of a

lcoh

ol p

robl

em (

DS

M-I

V)

Chi

ld In

take

Inst

rum

ent

Cur

rent

dia

gnos

is o

f dru

g pr

oble

m (

DS

M-I

V)

Chi

ld In

-Tre

atm

ent I

nstr

umen

tC

hild

Tre

atm

ent E

xit I

nstr

umen

tC

hild

Fol

low

-up

Inst

rum

ent

45B

ES

T C

OP

YA

VA

ILA

BLE

46

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

N.

,I2U

EST

ION

SE

VA

LU

AT

ION

ME

ASU

RE

S/V

AR

IAB

LE

S

..

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

,

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

Pre

Dur

ing

'E

xit

Post

Wha

t are

the

clie

nts'

Gen

der

Clie

nt In

take

Inst

rum

ent

dem

ogra

phic

cha

ract

eris

tics?

Age

Clie

nt In

-Tre

atm

ent I

nstr

umen

tE

thni

city

Clie

nt T

reat

men

t Exi

t Ins

trum

ent

Rac

eC

lient

Fol

low

-up

Inst

rum

ent

Pre

gnan

cy s

tatu

sA

dole

scen

t Int

ake

Inst

rum

ent

Sex

ual o

rient

atio

nA

dole

scen

t In-

Tre

atm

ent I

nstr

umen

tV

eter

an s

tatu

sA

dole

scen

t Tre

atm

ent E

xit I

nstr

umen

tP

rimar

y la

ngua

ge s

poke

nA

dole

scen

t Fol

low

-up

Inst

rum

ent

Chi

ld In

take

Inst

rum

ent

Chi

ld In

-Tre

atm

ent I

nstr

umen

tC

hild

Tre

atm

ent E

xit I

nstr

umen

tC

hild

Fol

low

-up

Inst

rum

ent

Wha

t are

the

char

acte

ristic

s of

Cur

rent

mar

ital s

tatu

sC

lient

Inta

ke In

stru

men

tth

e cl

ient

s' s

ocia

l and

fam

ilyN

umbe

r of

chi

ldre

n, u

nder

age

18

Clie

nt In

-Tre

atm

ent I

nstr

umen

tre

latio

nshi

ps?

In w

hat w

ays

Num

ber

of p

erso

ns w

ith w

hom

clie

nt c

urre

ntly

live

sC

lient

Tre

atm

ent E

xit I

nstr

umen

tha

ve th

ese

rela

tions

hips

Cur

rent

res

iden

ce--

type

Clie

nt F

ollo

w-u

p In

stru

men

tch

ange

d?C

urre

ntly

livi

ng w

ith s

ubst

ance

abu

ser

Ado

lesc

ent I

ntak

e In

stru

men

t./

././

./P

robl

ems

expe

rienc

ed w

ith fa

mily

, fie

nds,

cow

orke

rsA

dole

scen

t In-

Tre

atm

ent I

nstr

umen

tA

dole

scen

t Tre

atm

ent E

xit I

nstr

umen

tA

dole

scen

t Fol

low

-up

Inst

rum

ent

Chi

ld In

take

Inst

rum

ent

Chi

ld In

-Tre

atm

ent I

nstr

umen

tC

hild

Tre

atm

ent E

xit I

nstr

umen

tC

hild

Fol

low

-up

Inst

rum

ent

47B

ES

T C

OP

YA

VA

ILA

BLE

48

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

"E

VA

LU

AT

ION

ME

ASU

RE

SNA

RIA

BL

ES

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

DA

TA

CO

LLE

CT

ION

TIM

E P

OIN

TS

Pre

Dur

ing

Exi

tPo

st

Wha

t are

the

clie

nts'

edu

catio

n,em

ploy

men

t, in

com

e st

atus

?C

urre

ntly

atte

ndin

g sc

hool

/trai

ning

Hig

hest

leve

l of e

duca

tion

com

plet

edC

lient

Inta

ke In

stru

men

tC

lient

In-T

reat

men

t Ins

trum

ent

In w

hat w

ays

have

thes

eH

igh

scho

ol d

iplo

ma

or G

ED

Clie

nt T

reat

men

t Exi

t Ins

trum

ent

stat

uses

cha

nged

?C

urre

nt e

mpl

oym

ent s

tatu

sC

lient

Fol

low

-up

Inst

rum

ent

Wee

ks u

nem

ploy

ed-

Ado

lesc

ent I

ntak

e In

stru

men

t

Wor

k tr

aini

ng p

rogr

am p

artic

ipat

ion

Ado

lesc

ent I

n-T

reat

men

t Ins

trum

ent

Larg

est h

ouse

hold

inco

me

sour

ceA

dole

scen

t Tre

atm

ent E

xit I

nstr

umen

tP

re-t

ax le

gal a

nnua

l hou

seho

ld in

com

eA

dole

scen

t Fol

low

-up

Inst

rum

ent

Pre

-tax

indi

vidu

al in

com

e, p

ast 3

mon

ths

Chi

ld In

take

Inst

rum

ent

Day

s th

at s

ubst

ance

use

cau

sed

mis

sed

wor

kC

hild

In-T

reat

men

t Ins

trum

ent

Chi

ld T

reat

men

t Exi

t Ins

trum

ent

Chi

ld F

ollo

w-u

p In

stru

men

t

Wha

t are

the

clie

nts'

cur

rent

Num

ber

of ti

mes

clie

nts

arre

sted

, eve

r; la

st 3

mon

ths

Clie

nt In

take

Inst

rum

ent

crim

inal

/ juv

enile

just

ice

Rea

sons

for

arre

sts

Clie

nt In

-Tre

atm

ent I

nstr

umen

tst

atus

? In

wha

t way

s ha

veN

umbe

r of

tim

es c

lient

s co

mm

itted

crim

eC

lient

Tre

atm

ent E

xit I

nstr

umen

tth

ese

stat

uses

cha

nged

?C

rime

com

mitt

edC

lient

Fol

low

-up

Inst

rum

ent

Num

ber

of c

onvi

ctio

ns, e

ver;

last

3 m

onth

sA

dole

scen

t Int

ake

Inst

rum

ent

Cur

rent

ly s

ervi

ng s

ente

nce

Ado

lesc

ent I

n-T

reat

men

t Ins

trum

ent

Cur

rent

ly a

wai

ting

char

ges,

sen

tenc

e, d

iver

sion

Ado

lesc

ent T

reat

men

t Exi

t Ins

trum

ent

Num

ber

of ti

mes

inca

rcer

ated

/det

aine

dA

dole

scen

t Fol

low

-up

Inst

rum

ent

Num

ber

of d

ays

inca

rcer

ated

/det

aine

d, la

st 3

mon

ths

Chi

ld In

take

Inst

rum

ent

Num

ber

of o

ffens

es c

omm

itted

, las

t 3 m

onth

sC

hild

In-T

reat

men

t Ins

trum

ent

Mem

ber

of g

ang,

eve

r; la

st 3

mon

ths

Chi

ld T

reat

men

t Exi

t Ins

trum

ent

Mos

t rec

ent a

ctiv

ity a

s ga

ng m

embe

rC

hild

Fol

low

-up

Inst

rum

ent

49B

ES

T C

OP

YA

VA

ILA

BLE

50

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

EG

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

,.

EV

AL

UA

TIO

N,- ,Q

UE

ST

ION

S.

',

.

EV

AL

UA

TIO

N M

EA

SUR

ES/

VA

RIA

BL

ES

" D

AT

A S

OU

RC

E/I

NST

RU

ME

NT

DA

TA

CO

LL

EC

TIO

NT

IME

PO

INT

S

POD

urin

gE

xit

Post

Wha

t phy

sica

l/men

tal h

ealth

Sig

nific

ant p

hysi

cal/m

enta

l hea

lth s

ympt

oms,

eve

r;C

lient

Inta

ke In

stru

men

tpr

oble

ms

have

the

clie

nts

last

3 m

onth

sC

lient

In-T

reat

men

t Ins

trum

ent

expe

rienc

ed?

Wha

t ser

vice

sP

hysi

cal/m

enta

l hea

lth a

sses

smen

t, la

st 3

mon

ths

Clie

nt T

reat

men

t Exi

t Ins

trum

ent

have

thes

e cl

ient

s re

ceiv

ed?

Num

ber,

type

phy

sica

l/men

tal h

ealth

ser

vice

sC

lient

Fol

low

-up

Inst

rum

ent

Wha

t hav

e be

en th

e ch

ange

s in

rece

ived

, las

t 3 m

onth

sA

dole

scen

t Int

ake

Inst

rum

ent

the

clie

nts'

phy

sica

l/men

tal

Phy

sica

l/men

tal h

ealth

pro

blem

s cu

rren

tly b

eing

Ado

lesc

ent I

n-T

reat

men

t Ins

trum

ent

heal

th?

trea

ted

Ado

lesc

ent T

reat

men

t Exi

t Ins

trum

ent

Sub

stan

ce a

buse

trea

tmen

t his

tory

, Ado

lesc

ent F

ollo

w-u

p In

stru

men

tC

hild

Inta

ke In

stru

men

tC

hild

In-T

reat

men

t Ins

trum

ent

Chi

ld T

reat

men

t Exi

t Ins

trum

ent

Chi

ld F

ollo

w-u

p In

stru

men

t

Wha

t typ

es o

f abu

se h

ave

the

Phy

sica

l abu

se, e

ver;

last

30

days

Clie

nt In

take

Inst

rum

ent

clie

nts'

exp

erie

nced

? In

wha

tA

ge fi

rst p

hysi

cally

abu

sed

Clie

nt In

-Tre

atm

ent I

nstr

umen

tw

ays

have

thes

e ab

use

Sou

rce

of p

hysi

cal a

buse

Clie

nt T

reat

men

t Exi

t Ins

trum

ent

expe

rienc

es c

hang

ed?

Sex

ual a

buse

, eve

r; la

st 3

0 da

ysC

lient

Fol

low

-up

Inst

rum

ent

Age

firs

t sex

ually

abu

sed

Ado

lesc

ent I

ntak

e In

stru

men

tS

ourc

e of

sex

ual a

buse

Ado

lesc

ent I

n-T

reat

men

t Ins

trum

ent

---

Em

otio

nal a

buse

, eve

r; la

st 3

0 da

ysA

dole

scen

t Tre

atm

ent E

xit I

nstr

umen

t

Age

firs

t em

otio

nally

abu

sed

Ado

lesc

ent F

ollo

w-u

p In

stru

men

tS

ourc

e of

em

otio

nal a

buse

Chi

ld In

take

Inst

rum

ent

Chi

ld In

-Tre

atm

ent I

nstr

umen

tC

hild

Tre

atm

ent E

xit I

nstr

umen

tC

hild

Fol

low

-up

Inst

rum

ent

51

3ES

T C

OP

Y A

VA

ILA

BLE

52

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

NQ

UE

STIO

NS

ME

ASU

RE

S/V

AR

IAB

LE

S,

,

'EV

AL

6AT

ION

ME

ASU

RE

S/V

AR

IAB

LE

SD

AT

A S

OU

RC

E /I

NST

RU

ME

NT

DA

TA

,

.TIM

E P

OIN

TS

Pre

Dur

ing

'I E

xit

Post

Wha

t hig

h-ris

k be

havi

ors

have

Inje

ctin

g dr

ugs

Clie

nt In

take

Inst

rum

ent

the

clie

nts

exhi

bite

d? In

wha

tE

xcha

nged

sex

for

drug

s or

mon

eyC

lient

In-T

reat

men

t Ins

trum

ent

way

s ha

ve th

e hi

gh-r

isk

Sha

red

need

les,

coo

kers

, oth

er w

orks

Clie

nt T

reat

men

t Exi

t Ins

trum

ent

beha

vior

s ch

ange

d?H

ad u

npro

tect

ed/u

nsaf

e se

xC

lient

Fol

low

-up

Inst

rum

ent

Tes

ted

for

HIV

at l

east

6 m

onth

s fo

llow

ing

Ado

lesc

ent I

ntak

e In

stru

men

tun

prot

ecte

d se

xA

dole

scen

t In-

Tre

atm

ent I

nstr

umen

tA

dole

scen

t Tre

atm

ent E

xit I

nstr

umen

tA

dole

scen

t Fol

low

-up

Inst

rum

ent

Chi

ld In

take

Inst

rum

ent

Chi

ld In

-Tre

atm

ent I

nstr

umen

tC

hild

Tre

atm

ent E

xit I

nstr

umen

tC

hild

Fol

low

-up

Inst

rum

ent

53B

ES

T C

OP

YA

VA

ILA

BLE

54

EX

HIB

IT V

I-1

(CO

NT

INU

ED

)SA

MPL

E D

AT

A M

AP

FOR

KN

OW

LE

DG

E-G

EN

ER

AT

ING

SU

BST

AN

CE

AB

USE

TR

EA

TM

EN

T E

VA

LU

AT

ION

EV

AL

UA

TIO

N`Q

UE

STIO

NS:

EV

AL

UA

TIO

N. M

EA

SUR

ES/

VA

RIA

BL

ES

DA

TA

SO

UR

CE

/IN

STR

UM

EN

T

..

DA

TA

,CO

LL

EC

TIO

N,

TIM

E P

OIN

TS

Bas

elin

e.

Qua

rter

lyI

Ann

ually

Q6.

Wha

t is

the

rela

tions

hip

betw

een

the

cost

s an

d ou

tcom

es o

f th

e kn

owle

dge-

gene

ratin

g su

bsta

nce

abus

e tr

eatm

ents

ervi

ce?

Wha

t are

the

cost

s, b

enef

its,

and

the

ratio

bet

wee

n co

sts

and

bene

fits

of th

ekn

owle

dge-

gene

ratin

gsu

bsta

nce

abus

e tr

eatm

ent

serv

ice?

Mon

etar

y va

lues

of t

he tr

eatm

ent s

ervi

ce o

utco

mes

Rel

atio

nshi

ps b

etw

een

cost

s an

d tr

eatm

ent o

utco

mes

Ben

efit-

cost

rat

io fo

r su

bsta

nce

abus

e tr

eatm

ent

Rel

ativ

e co

sts

and

bene

fits

com

pare

d to

oth

er s

ervi

ces

Non

-mon

etar

y be

nefit

s th

at s

houl

d be

con

side

red

Cos

t offs

ets

of o

utco

mes

Der

ived

from

dat

a co

llect

ed to

addr

ess

Que

stio

ns 1

- 5

Wha

t are

the

cost

s,ef

fect

iven

ess

valu

es, a

ndth

e ra

tio o

f the

cos

ts a

ndef

fect

iven

ess

of th

ekn

owle

dge-

gene

ratin

gsu

bsta

nce

abus

e tr

eatm

ent

serv

ice?

Val

ue o

f the

trea

tmen

t ser

vice

effe

ctiv

enes

sR

elat

ions

hips

bet

wee

n co

sts

and

trea

tmen

t effe

ctiv

enes

sE

ffect

ive-

cost

rat

io fo

r su

bsta

nce

abus

e tr

eatm

ent

Rel

ativ

e co

sts

and

effe

ctiv

enes

s co

mpa

red

to o

ther

serv

ices

Non

-mon

etar

y be

nefit

s th

at s

houl

d be

con

side

red

Cos

t offs

ets

of o

utco

mes

Der

ived

from

dat

a co

llect

ed to

addr

ess

Que

stio

ns 1

- 5

BE

ST

CO

PY

AV

AIL

AB

LE56

REFERENCES

57

REFERENCES

Conrad, K.J., Randolph, F.L., Kirby, Jr., M.W., & Bebout, R.R. (In press). Creating and usinglogic models: Four perspectives. Alcoholism Treatment Quarterly.

Devine, P. (revised 1999). Integrated evaluation methods: A guide for substance abusetreatment knowledge-generating activities. Fairfax, VA: National .Evaluation Data andTechnical Assistance Center, Caliber Associates.

Devine, P. (1999). A guide to process evaluation for substance abuse treatment services.Fairfax, VA: National Evaluation Data and Technical Assistance Center, CaliberAssociates.

Devine, P., Christopherson, E., Bishop, S., Lowery, J., & Moore, M. (revised 1999). The self-adjusting treatment evaluation model. Fairfax, VA: National Evaluation Data andTechnical Assistance Center, Caliber Associates.

Kumpfer, K. Shur, G. Bunnell, K., Librett, J., & Millward, A. (1993). Measurements inprevention: A manual on selecting and using instruments to evaluate preventionPrograms. Rockville, MD: U.S. Department of Health and Human Services.

Linney, J.A., & Wandersman, A. (1991). Prevention plus III: Assessing alcohol and other drugprevention programs at the school and community level. A four-step guide to usefulprogram assessment. Rockville, MD: Office for Substance Abuse Prevention.

National Evaluation Data and Technical Assistance Center. (revised 1999). Minimumevaluation data set: Core data lists. Fairfax, VA: Caliber Associates.

Orwin, R.G. (1998). Evaluation designs: Interpretability and the need for random assignment.CSAT Training Presentation. Fairfax, VA: National Evaluation Data and TechnicalAssistance Center, Caliber Associates.

Wholey, J.S. (1979). Evaluation: Promises and performance. Washington, D.C.: The UrbanInstitute.

Wholey, S.J., Hatry, H.P., & Newcomer, K.E. (Eds.). (1994). The handbook of practicalprogram evaluation. San Francisco: Jossey-Bass.

JACSAT\CTRT_ENINEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 34

58

APPENDIX:INTEGRATED EVALUATION METHODS PACKAGE:A GUIDE FOR SUBSTANCE ABUSE TREATMENT

KNOWLEDGE-GENERATING ACTIVITIES-EXECUTIVE SUMMARY

59

APPENDIX:

INTEGRATED EVALUATION METHODS PACKAGE:

A GUIDE FOR SUBSTANCE ABUSE TREATMENTKNOWLEDGE-GENERATING ACTIVITIESEXECUTIVE SUMMARY

Since its inception, the Center for Substance Abuse Treatment (CSAT) has provided

Federal leadership to improve substance abuse treatment accessibility, effectiveness, and

efficiency. CSAT's mission and activities have evolved from directly supporting treatment

services to supporting knowledge-generating activities. This evolution is evident in the current

Substance Abuse and Mental Health Services Administration policy on evaluation as described

in Evaluation Policy, SAMHSA, 1995.

The need for an integrated model of evaluation and planning at SAMHSA is presented in

"Evaluation in the Substance Abuse and Mental Health Services Administration," Evaluation

and the Health Professions, by Marsh, Jansen, Lewis, & Straw, 1996. CSAT also supports site-

specific, cross-site, and national evaluations that have provided experience with a wide array of

evaluation design and implementation methods. These experiences further supported the need

for an integrated evaluation strategy and led to the development of a comprehensive set of

evaluation products, including concept papers, technical assistance (TA) materials, and analytic

tools. Collectively, these products are referred to as the Integrated Evaluation Methods (IEM)

Package. The IEM Package organizes these products within an evaluation framework that is

designed to support CSAT knowledge development and application goals. The evaluation

framework itself was constructed on the basis of accumulated experiences among internationally

known treatment service evaluation professionals. The IEM Package reflects and incorporates

evaluation experiences gained over the past decade.

Evaluation Framework and the Integrated Evaluation Methods Package

National evaluation experiences have reinforced the fact that substance abuse treatment

evaluation involves a standard set of tasks that generally occur in the following order:

Planning the evaluation/knowledge-generating activities, which includes selectingthe substance abuse treatment issue, identifying the theoretical foundation for theintervention, determining knowledge development program goals and implementationapproach, and setting the evaluation goals and objectives that determine the overallparameters of the evaluation

60J: \CSAT\CTRT_ END \IEM \CONCEPT\LOGICMDL \DATA \LOGICMDL.WPD NEDTAC, Page 1

Executive Summary

Selecting the evaluation design, which sets forth the overall strategy for establishingthe process and outcome evaluation questions, measurement approach, andgeneralizability of findings

Developing the data requirements, which flow from the evaluation questions andmeasures and include: SDU, clinician, cost, and client data

Developing data collection instruments, which are based on the data requirementsand are developed or selected from an integrated inventory of instrumentation

Collecting the data, which includes developing data management processes and tools(including quality control procedures) and conducting the data collection activities

Analyzing the data, which involves multiple levels of comparison and is governedby an analysis plan

Reporting the evaluation findings, which includes evaluation knowledgedissemination and application within the field.

The evaluation process outlined above provided a framework for the development of products

related to these evaluation concepts and methods. Taken together, those products comprise theIEM Package.

Integrated Evaluation Methods Products

CSAT requested the development of a series of evaluation concept papers, TA materials,

and tools to support and operationalize each phase in the evaluation of substance abuse treatment

knowledge-generating activities. These items are included in the IEM Package. The concept

papers are based on theoretical evaluation research constructs that have been adapted to

substance abuse treatment services evaluation and knowledge-generating activities. The concept

papers primarily support the evaluation planning phase and address such topics as the self-

adjusting treatment evaluation model, cost analyses, and performance measurement. The TA

materials and tools include specific evaluation methods that have direct applicability to substance

abuse treatment knowledge-generating activities. The concept papers and TA materials that

constitute the IEM Package are listed and briefly described in Exhibit I.

61

JACSAT\CTRT_ENDVEM\CONCEPT\LOGICMDMATA\LOGICMDLWPD NEDTAC, Page 2

Executive Summary

EXHIBIT IEVALUATION FRAMEWORK AND INTEGRATED

EVALUATION METHODS PACKAGE

EVALUATIONFRAMEWORK. INTEGRATED EVALUATION METHODS PRODUCTS

1. Planning the Integrated Evaluation Methods: A Guide for Substance Abuse Treatmentevaluation/ Knowledge Generating Activities: Concept paper that describes the development of anknowledge- evaluation framework, evaluation concepts, and TA materials to support the framework.generatingactivities Self-Adjusting Treatment Evaluation Model: Concept paper that describes an

approach for integrating evaluation findings within treatment operations so as to adjustand improve service delivery.

Building Team Capability to Fully Implement and Utilize the Self-AdjustingTreatment Evaluation Model: Concept paper to assist treatment providers in buildingcapabilities to integrate the self-adjusting treatment model within day-to-day operationsand service delivery.

Adding "Value" to CSAT Demonstrations: The What, How and Why of CostAnalysis: Concept paper on the need for and types of cost analyses for CSATdemonstrations and knowledge-generating activities. (The Lewin Group)

Performance Measurement for Substance Abuse Treatment Services: Conceptpaper about the increasing importance of provider performance measurement andanalyses and an explanation of the case-mix adjustment methodology.

Client Levels of Functioning as a Component of Substance Abuse TreatmentServices Evaluation: Description of the rationale and methods for assessing client levelof functioning and recommended core LOF data elements that could help to measure theeffectiveness of treatment services received.

Substance Abuse Treatment Evaluation Policy Notebook: These materials are aimedat facilitating understanding of the SAMHSA policy for evaluation and federalregulations on client confidentiality and assisting evaluators to meet CSAT evaluationrequirements.

Substance Abuse Treatment Evaluation Resource Notebook: The notebook containsevaluation bibliographies and listings of organizations, hot lines, on-line data bases, andcontact information for obtaining assistance in evaluating treatment services.

2. Selecting the A Guide to Process Evaluation for Substance Abuse Treatment Services: TA toolevaluation design presenting purposes of process evaluation and the application of process evaluation

methods to single site and multi-site treatment services.

Using Logic Models in'Substance Abuse Treatment Evaluations: TA tool describinglogic model purposes and techniques for designing and planning the evaluation oftreatment services.

A Guide to Selecting an Outcome Evaluation Design for Substance AbuseTreatment Evaluations: TA tool describing overall strategies for developingevaluation questions, measurement, controls, validity/reliability, sampling, designeffects, and generalizability of findings. (Battelle)

JACSATTTRTENMEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD

62NEDTAC, Page 3

Executive Summary

EXHIBIT I (CONTINUED)EVALUATION FRAMEWORK AND INTEGRATED

EVALUATION METHODS PACKAGE

EVALUATION,FRAMEWORK INTEGRATED EVALUATION METHODS PACKAGE

3. Developing datarequirements

Minimum Evaluation Data Set (MEDS): Core Data Lists: TA tool for developing auniform set of variables and response categories for the service delivery unit (SDU),clinician, cost, and client evaluation measures.

Substance Abuse Treatment Cost Allocation and Analysis Template (SATCAAT):User manual to analyze treatment costs by unit of service for an SDU. (CapitalConsulting Corporation)

4. Developing datacollectioninstruments

,

Substance Abuse Treatment Services Evaluation Data Collection Instruments: Datacollection instruments that fully incorporate the MEDS and that have been field testedfor validity and reliability, as follows: Service Delivery Unit (SDU) Description;Clinician Background and Practice Survey; protocols to collect Adult, Adolescent andChild (in treatment with parent) Client Data at Intake, During Treatment, at TreatmentDischarge and Post Treatment; Adult and Adolescent Record Extraction forms; and asection on protection of human subjects and informed consent.

5. Collecting thedata

Staying In Touch: A Fieldwork Manual of Tracking Procedures for LocatingSubstance Abusers for Follow-up Studies (UCLA): User manual to establish andimplement client follow-up data collection systems and procedures.

Strategies for Follow-up Tracking of Juvenile, Homeless, and Criminal JusticeSystem-Involved Substance Abusers: Overview and Bibliographies, 1990-1998:Description of tracking techniques used to increase response rates for follow-upinterviews with homeless and juvenile/criminal justice involved substance abusers.

6. Analyzing thedata

A Guide to Substance Abuse Treatment Evaluation Data Analysis: Recommendedmethods and procedures for analyzing process, SDU, clinician, cost, and clientevaluation data.

7. Reporting theevaluationfindings

Substance Abuse Treatment Evaluation Product Outlines Notebook: Compendiumof outlines for evaluation products including evaluation plans, interim evaluation reports,final evaluation reports, replication studies, case studies, and ethnographies.

3EST COPY AVAILABLE

JACSAT\CTRTEND\IEM\CONCEPT\LOGICMDL\DATATOGICMDLWPD

63NEDTAC, Page 4

Executive Summary

CSAT Evaluation "Stakeholders"

Evaluation "stakeholders" are individuals, groups, or organizations that have a significant

interest in how well a program or activity functions. (See P.H. Rossi, H.E. Freeman, & M.W.

Lipsey, Evaluation: A Systematic Approach, 6th Edition, 1999.) Within the context of the IEM

Package, CSAT evaluation stakeholders include CSAT senior managers, CSAT project officers,

and CSAT grantees and contractors including treatment service providers, coordinating centers,

study sites, site-specific evaluators, and national evaluators.

Utility of the IEM Package for CSAT Evaluation Stakeholders

While the conceptual and TA materials were developed from the perspective of the site-

specific and multi-site evaluator, the concepts and TA tools have important utility for CSAT

managers, project officers, and treatment service providers. The stakeholder's position

determines the perspective and utility of the IEM Package concepts and tools. For example, a

CSAT senior manager can use the IEM Package to acquire a comprehensive evaluation context

for planning and funding the knowledge-generating activities, the project officer can use the IEM

Package to ensure that GFA/RFP applications are complete and include a full complement of

design, execution, and product components, and the site-specific and multi-site evaluators can

use the IEM Package to ensure that evaluation designs, data collection plans, data analyses, and

product development have a consistent evaluation framework and compatible data across

program areas. The suggested utility of the IEM Package for CSAT evaluation stakeholders is

summarized in Exhibit II.

64

JACSAT\CTRT_ENDVEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD NEDTAC, Page 5

Executive Summary

EXHIBIT IIUTILITY OF IEM PACKAGE FOR CSAT EVALUATION STAKEHOLDERS

STAKEHOLDERS ROLES AND RESPONSIBILITIES IEM PACKAGE UTILITY

SENIORMANAGERS

Policy developmentIssue identification for KD&AsGrant/contract funding decisionsOverall program managementSustainabilityDisseminationLong-term strategic planningProgram designsKA activities

Comprehensive evaluation frameworkComprehensive evaluation componentsRoles and responsibilities for local/nationalevaluators as well as CSAT/grantee staffsGuidance for evaluation designs andproductsStandardized evaluation measuresLogic models for program and evaluationdesign

PROJECTOFFICERS

GFA/SOW developmentGrant/contract application reviewGrant/contract monitoringKnowledge-generating productsIdentification and replication ofpromising practicesTechnical assistance assessment

Guidelines for high-quality evaluationdesigns (process and outcome)Logic models for program and evaluationdesignsList of evaluation measures withinstrumentationGuidelines for evaluation products

GRANTEES:STUDY SITES

Grant applicationsProject development, implementationLocal evaluation managementLocal evaluation coordinationKnowledge-generating productdevelopment

Evaluation plan outlineProcess and outcomes evaluation designsSDU, clinician, cost, and client measuresRoles and responsibilities for granteeprovider/evaluator staffGuidelines for evaluation products

GRANTEES:MULTI-SITEEVALUATORS

Grant applicationsComprehensive evaluation designsEvaluation implementation:

Data collectionData analysisReporting evaluation findings

Evaluation product development

Evaluation conceptsLogic modelsEvaluation designsEvaluation data requirementsData collection instrumentationData collection process and proceduresData analysisProduct development

NATIONALEVALUATORS/SERVICESRESEARCHERS

Contract applicationsComprehensive evaltiation designsEvaluation implementation:

Data collectionData analysisReporting evaluation findings

Evaluation product development

Evaluation conceptsLogic modelsEvaluation designsEvaluation data requirementsData collection instrumentationData collection process and proceduresData analysisProduct development

IEM products and other evaluation materials may be obtained from:http://neds.calib.com

JACSATTIRT_ENDUEM\CONCEPT\LOGICMDL\DATA\LOGICMDLWPD

6 5

NEDTAC, Page 6

U.S. Department of EducationOffice of Educational Research and Improvement (OERI)

National Library of Education (NLE)

Educational Resources Information Center (ERIC)

NOTICE

Reproduction Basis

ERIC

This document is covered by a signed "Reproduction Release(Blanket)" form (on file within the ERIC system), encompassing allor classes of documents from its source organization and, therefore,does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission toreproduce, or is otherwise in the public domain and, therefore, maybe reproduced by ERIC without a signed Reproduction Release form(either "Specific Document" or "Blanket").

EFF-089 (3/2000)