AN EVALUATION NEWSLETTER SPRING/SUMMER 2003, SPECIAL … · AN EVALUATION NEWSLETTER SPRING/SUMMER...

AN EVALUATION NEWSLETTER SPRING/SUMMER 2003, SPECIAL ISSUE #6

Larke Nahme Huang, Ph.D. Kathy S. Hepburn, M.S., Rachele C. Espiritu, Ph.D.National Technical Assistance Center for Children’s Mental HealthGeorgetown University

This is the question confrontingproviders, families, payers, policy

makers, researchers, and advocates inthe field of children’s mental health.Evidence-based practice is an emergingconcept and reflects a nationwide effortto build quality and accountability inhealth and behavioral health careservice delivery. Underlying this conceptis (1) the fundamental belief thatchildren with emotional and behavioraldisorders should be able to count onreceiving care that meets their needsand is based on the best scientificknowledge available, and (2) thefundamental concern that for many ofthese children, the care that is deliveredis not effective care. Some haveidentified this movement to evidence-based practice as the new “revolution”in health care that focuses onassessment and accountability (Kiesler,2000). While there is much reason foroptimism and hope in this movementtowards evidence-based practice inchildren’s mental health, there is alsoreason for much concern and caution.Some of these concerns andchallenges are presented below.

Moving from Science to ServiceAll too frequently, children and

their families receive care that is based

on outdated practices and narrowlydefined outcomes as opposed to carethat is based on increasing evidence ofeffectiveness and a wider spectrum ofdesired functional and quality of lifeoutcomes. The field continues to relyon practices that have little supportingevidence or, at worst, have pooroutcomes. The care that is oftenprovided is based on “that’s what we’vealways done” rather than on anemerging evidence-base for “whatworks.” Research on the effectiveness ofclinical treatments, service modalitiesand preventive interventions continuesto grow at a rapid rate. This researchhas spurred new excitement and hopefor making a difference in the lives ofthese children. However, therecontinues to be a significant gapbetween what we know works andwhat is practiced in the field.

Changing practice is a formidabletask that occurs at a painstakinglyslow pace, often requiring not onlychanges in practice behaviors, butrestructuring programs and allocatingan infusion of upfront resources. Inaddition, implementation of newpractices can be especially difficult inan environment of shrinking state andlocal budgets and competing priorities.Implementation often involvessignificant organizational change,provider re-training and changes inpublic and private reimbursement.Clearly, a challenge is to promote theeffective dissemination and

Data Matters is a free publication of the

National Technical Assistance Center for Children’s Mental HealthGeorgetown University Center for Child and Human Development

Funded by theChild, Adolescent and Family BranchCenter for Mental Health ServicesSubstance Abuse and Mental Health Services AdministrationU.S. Department of Health and Human Services

To Be or Not To Be…Evidence-Based?

continued on page 2

Inside this issue…Moving Science 4-9to Service

Establishing Criteria 10-16and Cataloguing Evidence-Based Practices

Involving Diverse 17-23Consumers: Families,Communities, and Providers

Highlighting the Fit 24-32with Systems of Care: State and Community Level Efforts

evidence, based on the rigor of theresearch design (for example, numberof controlled studies, randomization ofparticipants in studies, number ofsingle-case studies, etc.) have been putforth by various researchorganizations and public policyprograms. These range from“evidence-based” practice grounded insystematic randomized clinical trials,to “evidence-informed” practice basedon meta-analyses of existing researchstudies, to “evidence-suggested”practice based on consensus groups

Data Matters

2 Special Issue 2003

implementation of proveninterventions, the task often describedas moving “science to services.”

Practice-based Evidence or FromService to Science

Evidence-based practices are notavailable for all problems and needsand, even when available, do notnecessarily work uniformly across allfamilies and communities. Manycommunities and providerorganizations have developedinnovative strategies and “promisingpractices” that lack a systematicallydeveloped evidence base. Inparticular, services targetingethnic and racial minoritycommunities have oftendeveloped culturally-drivenpractices or haveincorporated culturaladaptations to existingevidence-based practices tobetter serve their children andfamilies, however, they maylack the capacity andresources for research andevaluation. The evidence baseneeds to be developed forthese services and theircommunities. If we limit thebuilding of the evidence base to aone-way “science to services”approach, we risk stifling innovationand recognition of potential practice-based evidence.

What Constitutes Evidence?In the field of children’s mental

health, “evidence base” refers toscientifically obtained knowledgeabout the prevalence, incidence orrisks for mental disorders or about theimpact of treatments or services onthese problems (Burns and Hoagwood,2002). It denotes quality, robustnessand accountability. But establishingthe criteria for what constitutes anevidence-based practice varies amongdifferent child-serving systems andprovider groups. Different levels of

we must continue to proceed with thebest existing knowledge, expertconsensus, and experience.

And Whose Evidence Is It?Concerns have been raised that

much of the research on practice andservice in children’s mental health hasoccurred in academic laboratory-typesettings with children who display asingle, well-circumscribed disorder.The intent of designing researchstudies in this manner is to prevent theintrusion of “confounding” variables.However, these variables often reflectreal-life situations and need to beincorporated into the examination ofeffective practice. Failure to attend tothese variables, which affect theconditions of practice, may diminishthe relevance of this research. Thesepractices may work only in acontrolled research setting, not in real-

life clinic settings. Additionally,children often present withcomplex disorders that do noteasily fall into a singlediagnostic category. Forexample, co-occurringdisorders, whether acombination of emotionaldisorders or emotional andsubstance abuse disorders,

are becoming increasingly prevalent.Children with these disorders areparticularly challenging to a researchendeavor that traditionally isolates acondition in order to determinediagnostic-specific treatments. Andfinally, much of this research does notinclude racially and ethnically diversepopulations, so the generalizability ofthese evidence-based practices remainsto be determined.

Family ChoiceIn the last decade, a strong family

movement has highlighted the positiveimpact of family involvement andfamily choice in the treatmentplanning and decision making for theirchildren with serious emotional

To Be or Not To Be… continued from page 1

and expert opinion (Evans, 2003).Similarly, many national efforts andprovider systems have constructedtheir own criteria and cataloguing ofevidence-based practices. Theserepresent critical efforts to identifyservices that produce positiveoutcomes for youth and warrant theexpenditure of shrinking fiscal andhuman resources. However, themultiple efforts and criteria foridentification of evidence-basedpractices raise potential confusion anddilemmas for practitioners, policymakers, families and consumers. Moreclarity is needed to ensure informeddecision-making. And, as we await thefindings from practice-based research,

Special Issue 2003 3

disorders. Major advocacy andprovider groups endorse families aspartners in planning. The field is nolonger concerned with whether toinvolve families, but how best to dothis. With the movement to evidence-based practice, how does this affect therole of families in decisions regardingtreatment and intervention? How willthe strength of science-based practicesbe integrated with family choice?Families often present first-handevidence of what works for their childin the context of their family andcommunity and concerns have beenraised that this “evidence” will beminimized in favor of “scientificevidence.” At the least, families shouldbe informed of evidence-basedpractices. But beyond this, familiesneed to also have a role in activelyshaping and evaluating practice.While families want to know whatworks and what practices are effective,they also need to have a voice indetermining what practices, servicesand supports address their needs andshould be the focus of researchendeavors. For example, whileresearchers may focus on theeffectiveness of psychotherapy, familiesmay prioritize building the evidencebase for effective respite care services.

The Fit with Systems of CareQuestions have been raised about

the compatibility of evidence-basedpractice and systems of care. Someconcerns have been expressed that themovement to evidence-based practicewill supersede or displace the systemsof care approach. A system of careapproach and evidence-based practiceare not competing efforts butcomplementary. Systems of care focuson improving access, developing abroad array of services and ensuringcoordination; it provides the contextfor evidence-based practices. Thesystem of care provides the servicedelivery vehicle for clinical treatmentand support services and neither thesystem nor the practice alone is likely

to yield positive results for children andtheir families (English, 2002). It isthese two concepts working in tandemthat provide the hope for improvedaccess and quality of care. Thus, themovement toward evidence-basedpractice converges well with a systemof care approach.

Will Funding Follow the Evidence Base?In 1998, approximately $11.75 billionwas spent for mental health servicesfor children in the specialty mentalhealth and general health sectorsalone. This represents a three-foldincrease since 1986 (Sturm, et al.,2001). It also raises the question ofhow these dollars are being spent.Given a continued reliance ontraditional services that lack a strongevidence base, are we utilizingresources for effective practices?Historically, large amounts of federaland state dollars were spent to pay formore restrictive and less effectiveservices. As the evidence increases toidentify “what works”, policy mustaddress both the selection and fundingof the most effective services. Animportant caveat in the fundingpicture is that we still need to learnmore about the generalizability of thecurrent evidence to children withcomplex disorders and children fromdiverse communities. Conversely, weneed to be careful not to de-fund orunder fund services and supports thatare promising but lack the evidencebase or to fund only a singlecomponent of an evidence-basedmultimodal service. For example,where pharmacological interventionsare adjunctive to psychosocialtherapies, some payment models fund only the medication componentof the treatment, reducing costs but ignoring fidelity to the evidence-based intervention.

Overview of Data Matters #6With the increasing momentum of

evidence-based practices, a broad

array of stakeholders is contributing tothe discussions and debates on thistopic. We are excited about this issue ofData Matters which provides a forumfor the perspectives of these diversestakeholders. The articles in this issuediscuss issues being addressed byleading researchers in the field,cataloguing of practices by differentchild-serving systems, family accoladesand cautions regarding evidence-basedpractices, challenges in working withproviders as the “consumers” of thesepractices, state and local communityefforts to implement evidence-basedpractices, and implementing anevidence-based practice in systems ofcare. While the voices are variouslysupportive or cautious, all share incommon the desire for high quality,effective services to improve the lives of children with emotional andbehavioral disorders. We are pleased to bring you this issue and hope youfind it informative. ◆

ReferencesBurns, B. J., & Hoagwood, K. (2002).Community treatment for youth: Evidence-basedinterventions for severe emotional and behavioraldisorders. NY: Oxford University Press.

English, M. (2002). Policy implications relevantto implementing evidence-based treatment. InB. J. Burns & K. Hoagwood (Eds.), Communitytreatment for youth: Evidence-based interventionsfor severe emotional and behavioral disorders. NewYork: Oxford University Press.

Evans, A. (2003). Addressing behavioral healthdisparities and improving cultural competencewithin a statewide system of care. Presentation atthe Santa Fe Symposium, American College ofMental Health Administration, Santa Fe, NM.

Kiesler, C. (2000). The new wave of change forpsychology and mental health services in thehealth care revolution. American Psychologist,55(5), 481-487.

National Institute for Mental Health. (2001).National estimates of mental health utilization andexpenditures for children in 1998 (Working PaperNo. 205). Washington, DC: Sturm, R., et al.

Georgetown University, National TA Center forChildren’s Mental Health. (2002). Systems ofcare: Issue brief. Washington, DC: Beth Stroul.

U.S. DHHS (1999). Mental health: A report of theSurgeon General. Rockville, MD.

Data Matters


on when the evidence base, howeverit is defined, is ready to be deployed,moved out, and used in communitysettings. Numerous and discrepantcriteria are being used byprofessional associations and by thescientific community to denoteevidence-based from non-evidencebased. The varying definitions makeit difficult for policy-makers orpractitioners to decide whichamongst the practices to adopt inany given circumstance. There are

attempts currently by foundationsand federal agencies to createagreed-upon criteria and to createan archive of research-basedpractices which can be updated andprovide assistance to fieldpractitioners and the scientificcommunity on the quality andstrength of the evidence aboutmental health care.

● Little Evidence-Based Help for Severe,Co-Occurring Mental Health ProblemsThe strength of the evidence inresearch-based knowledge largelycenters on discrete treatment fordiscrete disorders (Weisz et al.,1995a, b). Unfortunately, manychildren present with multiple,chronic, and severe problems. Thestrength of the evidence about

mental health care for these childrenwith serious emotional or behavioraldisturbances is not yet as strong.

● Uncoordinated and FragmentedServicesIn addition to this problem of co-occurring disorders, the evidencebase about how best to coordinateservices for children has laggedbehind knowledge about discretepractices. Mental health services foryouth are provided in thousands ofdifferent settings or locales: schools,clinics, health centers, juvenileprobation, etc. Each of these“systems” contains discrete rulesgoverning their administration ofmental health care—separatereimbursement policies andincentive structures, differenttraining requirements for providers,and diverse regulations governingentry in the care they provide for thepopulation they serve. The evidenceabout how best to coordinateevidence-based services and createparticipatory management teams foryouth that involve all key individuals(e.g., family members, providersacross all major systems and thechild) is almost non-existent.

New Directions for Research and New Models for InterventionDevelopment: Looking at aRevolution in InterventionDevelopment● To ensure ecological validity, the

research model should include theperspectives of stakeholders,families, and providersIf the goal is to enhance thegeneralizability and uptake of

Data Matters

In general, “evidence-basedpractice” refers to a body ofscientific knowledge about servicepractices, about the impact oftreatments on child or adolescentmental health conditions, or aboutthe impact of preventiveinterventions on the course ofchildren’s development.

Kimberly Hoagwood, Ph.D.New York State Office of Mental Healthand New York State Psychiatric InstituteColumbia University

The research base on the risks formental disorders or conditions, on

the efficacy of mental healthtreatments and preventive strategiesfor youth, and on the effectiveness ofservices has expanded enormously inthe past decade. There has been adoubling of research studies on childand adolescent mental health at theNational Institute of Mental Health(NIMH) and a tripling of funds forresearch on these issues over thisperiod of time (Blueprint for Change,NIMH, 2001). Yet this research-basedknowledge on the evidence about theimpact of mental health interventionshas been largely ignored. So what gives?

In this article, I describe some ofthe reasons why research knowledgeon evidence-based practice is notreaching its intended audience—i.e.,children and adolescents with mentalhealth needs, their families, andproviders of mental health care.Closing the gap requires acknowledgingthe existing evidence about specificmental health practices andredirecting research studies to linkpolicy initiatives about service deliveryand the science base.

Dissemination and ImplementationIssues: Some Reasons WhyResearch-Based Knowledge Is Not Used● No Consensus on What an Evidence-

Based Practice IsThere is currently no consensus onhow to define “evidence-based”, or

Evidence-Based Practice in Children’s Mental Health Services

WHAT DO WE KNOW? WHY AREN’T WE PUTTING IT TO USE?

MOVING SCIENCE TO SERVICE


Data Matters

research findings into practice, thenfrom the outset, research modelsshould incorporate the perspectivesof families, providers and otherstakeholders into the design of newtreatments, preventive strategies,and services. Only by doing so canissues relating to the relevance ofthe intervention for stakeholders, thecost effectiveness of the intervention,and the extent to which it reflectsthe values and traditions of familiesand community leaders beaddressed. These issues areultimately essential for the evidencebase to be of any practical utility.

● Create clinic and communityintervention development anddeployment modelsClinic and community interventiondevelopment and deployment modelswould attend to context variablessuch as characteristics of thepractice setting (e.g. practitionerbehaviors, organizational variables,community characteristics) andinvolvement of families andcommunity in the initial piloting andmanualization phase. These modelsare extremely challenging and canonly be accomplished with adequateresources and committedpartnership among scientists,families, providers and stakeholders.Such a model has been proposed byHoagwood, Burns & Weisz (2002) asa way to ensure strong scientifically-based practices and to accelerate thepace of the uptake of researchfindings into practice.

What is needed is a 180 degreeshift in how interventions aredeveloped. Such a shift will foregroundcontext variables (often considered tobe “nuisance” variables) instead ofputting them at the back end of a string of efficacy trials. Such a shift will focuson strengthening outcomes andaccountability by holding constant tothe goal of developing a scientifically-informed knowledge base on effective

Weisz, J. R., Donenberg, G. R., Han, S. S., &Weiss, B. (1995). Bridging the gap between laband clinic in child and adolescentpsychotherapy. Journal of Consulting and ClinicalPsychology, 63, 688-701(a).

Weisz, J. R., Weiss, B., Han, S., Granger, D. A., &Morton, T. (1995). Effects of psychotherapywith children and adolescents revisited: A meta-analysis of treatment outcome studies.Psychological Bulletin, 117, 450-468(b).

To review the complete journal article fromwhich this newsletter article was excerpted,please see Report on Emotional andBehavioral Disorders in Youth, Vol. 1:4,2001, pages 84-87, or for more informationplease contact: Kimberly Hoagwood,[email protected]

interventions for children, adolescentsand families. Only by doing so, canservices research address issuesessential to the uptake of evidence-based practices into diverse community settings. Without such a revolutionaryshift, the evidence base will sit unusedand unread on academic shelves. ◆

ReferencesHoagwood, K., Burns, B. J., & Weisz, J. R.(2002). A profitable conjunction: From scienceto service in children’s mental health. In B. J.Burns & K. Hoagwood (Eds.), Communitytreatment for youth: Evidence-based interventionsfor severe emotional and behavioral disorders (pp.327-338). New York: Oxford University Press.

Organizations, resources, and information cited in this issue related toevidence-based practices in behavioral health care

American Youth Policy Forumwww.aypf.org

California Institute for Mental Healthwww.cimh.org

Center for Substance Abuse Prevention (CSAP), Department of Health andHuman Services, National Registry of Effective Programs (NREP)http://modelprograms.samhsa.gov

Center for the Study and Prevention of Violence, Blueprints Initiativewww.colorado.edu/cspv/blueprints

Communities that Care, Developmental Research and Programs, Inc.www.preventionscience.com/ctc/CTC.html

CSAP’s Decision Support System (DSS)www.preventiondss.org

Department of Criminology and Criminal Justice, University of Marylandwww.preventingcrime.org (Direct link to article)www.ncjrs.org/works/wholedoc.htm (Direct link to lecture)

Department of Education, Safe and Drug-free Schoolswww.ed.gov(At this website, select Visit US Department of Education; use search option for “OSDFS”; selectOffice of Safe and Drug Free Schools—Publications; go to Publications—online publications—Exemplary and Promising Safe, Disciplined, and Drug-Free Schools Program 2001, Expert Panel)

Office of Juvenile Justice and Delinquency Prevention, National ProgramReview Committee, University of Utah, and CSAPwww.strengtheningfamilies.org

Prevention Research Center for the Promotion of Human Development,Pennsylvania State Universitywww.prevention.psu.edu/CMHS.html

U.S. Department of Health and Human Serviceswww.surgeongeneral.gov/library/youthviolence

WEB-BASED SOURCES


WEB-BASED SOURCES


Data Matters

Anne W. Riley, Ph.D.Johns Hopkins UniversityBloomberg School of Public Health

Disruptive behavior problems areamong the most common problems

of youth, are likely to impact children’ssuccessful transition to adulthood, andare fairly stable. Youth with behaviorproblems are more likely to receivemental health treatment than thosewith emotional problems, yet thetreatments they get often are those thatlack an evidence base. In particular,‘talk therapy’ does not work for youthwith significant behavior problemssuch as oppositional defiant andconduct disorder. Moreover, there is noevidence that residential treatmentcenters improve the outcomes of youth,and there is some suggestion that youthactually learn to become more deviantin these settings (Dishion, 1999;Bickman et al., 1995; Weisz, 1995;Weisz, Weiss, & Donenberg, 1992).

Evidence-Based Treatments forDisruptive Behavior Disorders

For conduct disorder and severeoppositional defiant disorder thefollowing types of treatments havebeen shown to work.● Parent Management Training (PMT)● Cognitive Problem Solving Skills

Training (PSST)● Combination of PSST and

PMT programs● Multisystemic Therapy (MST)

Parent Management Trainingseeks to change parent-childinteractions in the home, includingchild-rearing practices and coerciveinterchanges, by training parents to

manage their child’s behavior at homeand at school. The techniques arebased on social learning principles,and parents are helped to see howpositive and negative behaviors aredeveloped and maintained by theirconsequences. New skills are applied tosimple problems before trying to solvemore serious behavior problems.Duration of treatment varies withseverity. Programs for children withmild oppositional behavior typicallylast 6-8 weeks. However, typicaltreatments for clinically referred youthlast much longer, 12-25 weeks.

Examples of Parent ManagementTraining Programs:● Videotape Modeling Parent Training

(Webster-Stratton, 1994)● Parent-Child Interaction Therapy

(PCIT) (Hembree-Kigin, 1995)● Defiant Children: A Clinician’s Manual

for Parent Training (Barkley, 1987)● Helping the Noncompliant Child

(Forehand, 1981)

References for CaregiversForehand, R. & Long N. (1996).Parenting the Strong-Willed Child.Chicago: Contemporary Books. (312) 540-4500.

Cognitive Problem Solving SkillsTraining (PSST) is the program withthe best evidence for efficacy whenparents are not available or willing toparticipate in PMT sessions. Positiveresults have also been obtained whenPSST is used in conjunction with PMT.Problem solving skills training focuseson altering the cognitive processes thatunderlie social behavior. The treatmentfocuses on cognitive distortions and

impulse control problems common inaggressive youth who are helped tobuild skills that reduce the extent towhich they attribute hostile intent tothe actions of others and to developnon-aggressive responses to perceivedprovocations by peers.

Some programs are designed to beadministered in small group settings of3 to 5 children over a period of 18-22sessions. The therapist plays an activerole as a coach and for modeling theskills taught. The therapist leads role-playing of social situations so thatskills are practiced with the therapistproviding cues, feedback and praise.However, without the involvement ofparents and/or teachers, thegeneralization of the skills gained intreatment and the duration oftreatment effects is somewhat limited.

Examples of Problem Solving SkillsPrograms (see (Frick, 1998):● Self-Instructional Training (Kendall,

1991, 1985)● Anger Coping Program

(Lochman, 1996)● Promoting Alternative Thinking

Strategies (PATHS) Curriculum—The FAST Track Modification(Bierman, 1996)

Multisystemic Therapy (MST) is atreatment strategy that focuses onreducing antisocial behavior inadolescents by helping the various“systems” that influence thempromote acceptable behavior(Henggeler, 1998). It involvesimmediate and extended family, peers,schools and neighborhood, thusencompassing the context in which

Evidence-Based Treatments for Children’sDisruptive Behavior Problems

WHAT DO WE KNOW?



Data Matters

SummaryMost youth who receive an

empirically supported treatment getsignificantly better and do so morequickly than with other treatments orno treatment (Brestan, 1998;Chambliss, 2001; JCCP, 1998; Spirito,1999). This is important. We mustcontinue to move toward scientificallysupported treatment for all mentalhealth problems in children andadolescents, including those withdisruptive behaviors. Currently, thereis at least one scientifically supportedtreatment for each of the commonmental health problems in childrenand adolescents, Attention DeficitHyperactivity Disorder, conductdisorder, major depressive disorder, andanxiety disorders. This paperhighlights those shown to be effectivefor children with disruptive behaviors,and thus expands interventionoptions. This is the good news.However, the preparation of clinicians,fidelity to the parametersof the treatment protocol, and adaptingintervention for the individual familyand child can be difficult and remainsome of the greatest challenges oftransferring the evidence-basedtreatments into real-world settings. ◆

ReferencesBarkley, R. (1987). Defiant children: A clinician’smanual for parent training. New York: Guilford.

Bickman, L., Guthrie, P. R., Foster, E. M.,Lambert, E. W., Summerfield, W. T., Breda, C. S.,& Heflinger, C. A. (1995). Evaluating managedmental health services: The Fort Bragg experiment.New York: Plenum Press.

Bierman, K., Greenberg, M. T. (1996). Social skillstraining in the FAST Track program. In R. Peters& R. J. McMahon (Eds.), Preventing childhooddisorders, substance abuse, and delinquency (pp.65-89). Thousand Oaks, CA: Sage.

Brestan, E., & Eyberg, S. M. (1998). Effectivepsychosocial treatments of conduct-disorderedchildren and adolescents: 29 years, 82 studies,and 5,272 kids. Journal of Clinical ChildPsychology, 27, 180-189.

Burns, B. J., & Hoagwood, K. (2002). Community-based interventions for youth with severe emotionaldisturbances. New York: Oxford University Press.

the adolescent lives. Goals oftreatment are family driven, butoverarching goals include: (1) helpparents and caring adults shape theadolescent’s behaviors (2) overcomedifficulties, such as marital problems,that may get in the way of parenting,(3) reduce negative parent-childinteractions (4) develop cohesion andemotional warmth among familymembers. The focus of treatment hastypically been on seriously disorderedadolescents including juveniledelinquents, and parents are fullpartners in the treatment.

Treatment may employ differenttechniques such as: PMT, contingencymanagement, PSST, marital therapyand others. Treatment is oftenconducted in homes or at school. Thebehavior of therapists is governed by aset of 9 treatment principles.Adherence to these guidelinesoperationalizes fidelity to MST. Theadministration of MST is demanding.There is significant clinical decisionmaking and multiple interventionsneed to be implemented. Because ofthe varying demands of each case, andthe intensive interventions used, MSTtherapists must be capable of applyinga range of empirically-basedtherapeutic approaches (such asstructural family therapy, cognitivebehavior therapy) and tailoringinterventions to the unique needs ofthe family. MST is conducted by a“team” comprised of 2 to 4 MSTtherapists and their on-site supervisor.They work together for purposes ofgroup and peer supervision, and tosupport the 24-hour/7-day/week on-call needs of the team’s client families.

Information and Training Resources ● MST Services, Inc.

(843) 856-8226Fax: (843) 856-8227,[email protected] or go towww.mstservices.com,www.mstinstitute.org

Chambliss, D., & Ollendick, T.H. (2001).Empirically supported psychologicalinterventions: Controversies and evidence.Annual Review Psychology, 52, 685-716.

Dishion, T., & McCord, J. (1999). Wheninterventions harm—Peer groups and problembehavior. American Psychologist, 54(9), 755-764.

Forehand, R., & McMahon, R. J. (1981). Helpingthe noncompliant child: A clinician’s guide to parenttraining. New York: Guilford.

Frick, P. J. (1998). Conduct disorders and severeantisocial behavior. New York: Plenum.

Hembree-Kigin, T., & McNeil CB. (1995). Parent-child interaction therapy. New York: Plenum.

Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (1998). Multisytemic treatment of antisocial behavior inchildren and adolescents. New York: Guilford.

JCCP. (1998). Empirically supported psychosocial interventions for children. Journal of ClinicalChild Psychology, Special issue, 27, 138-226.

Kendall, P. (1991). Child and adolescent therapy: Cognitive-behavioral procedures. New York: Guilford.

Kendall, P., & Braswell, L. (1985). Cognitive-behavioral therapy for impulsive children. New York: Guilford.

Lochman, J., &Wells, K. C. (1996). A social-cognitive intervention with aggressive children:Prevention effects and contextualimplementation issues. In R. Peters & R. J.McMahon (Eds.), Preventing childhood disorders,substance abuse, and delinquency (pp. 111-143).Thousand Oaks, CA: Sage.

Spirito, A. E. (1999). Empirically supportedtreatments in pediatric psychology. Journal ofPediatric Psychology, Special issue, 24.

Webster-Stratton, C. (1994). Advancing videotape parent training: a comparison study. Journal ofConsulting and Clinical Psychology, 62(3), 583-593.

Weisz, J. R., Donenberg, G. R., Han, S. S., &Weiss, B. (1995). Bridging the gap betweenlaboratory and clinic in child and adolescenttherapy. Journal of Consulting and ClinicalPsychology, 63(5), 688-701.

Weisz, J. R., Weiss, B., & Donenberg, G. R.(1992). The lab versus the clinic: Effects of childand adolescent psychotherapy. AmericanPsychologist, 47(12), 1578-1585.

For more information on interventions for disruptive behavior go tohttp://www.strengtheningfamilies.org. Formore information on the evidence base forchildren and youth with mental disorderssee: Community Treatment for Youth:Evidence-based interventions for severeemotional and behavioral disorders (Burns,2002) or request a copy of a workbook onevidence-based treatments for youth fromDr. Anne Riley, ([email protected]).



Data Matters

children and families were better. Forexample, during a two-year follow-upperiod, the number of days delinquentyoungsters were incarcerated in thestate training school was lower forparticipants in MTFC than for acomparison group of youngstersplaced in group-care programs. In thestudy that examined outcomes forchildren and adolescents leaving theOregon State Hospital, we found thatyouth were moved off of waiting listsand placed in the community morequickly and that they had fewerbehavior problems in MTFC than incomparison placements.

These two early studies set thestage for a series of subsequent largerinvestigations of MTFC in both the

juvenile justice and in the child welfaresystems. In the first of these(Chamberlain & Reid, 1998), welooked at outcomes for 79 boysreferred because of chronic problemswith delinquency. Study boys had anaverage of 14 police offenses before

Patricia Chamberlain, Ph.D.Oregon Social Learning Center

Multidimensional Treatment FosterCare (MTFC) provides

community-based family care foryouth who are having severeemotional and behavioral challenges.MTFC has been identified as anevidence-based practice by theBlueprints for Violence Prevention, theCalifornia Institute for Mental HealthCaring for Foster Youth Initiative, andBurns & Hoagwood (2002). As analternative to group and residentialcare and to institutionalization andincarceration, the MTFC programrecruits, trains, supports andsupervises families in the communityto provide placements for youthparticipating in the program. Intensiveservices are provided to both the youthand to the members of their family(biological, adoptive, relative) so thatafter the youth completes the MTFCprogram, they can return homeand continue to be successful.This article briefly outlinesresearch conducted by the authorand collaborators and theevidence from pilot studies andlarger investigations that establishedthe utility of MTFC as a therapeuticprogram for youth with severeemotional and behavioral challenges.

The MTFC model was first tested intwo studies in the early 1990s todetermine the feasibility of using thismodel for adolescents referred fordelinquency and for youngstersleaving the state mental hospital.Results showed that MTFC was feasibleand that compared to alternativeresidential treatment models, it wascost effective and the outcomes for

placement and had spent an average of75 days during the previous year inlocked detention settings. Boys wererandomly assigned to placement inMTFC or Group Care and assessed oneyear after their placements ended.Compared to boys in Group Care,MTFC boys● Spent 60% fewer days incarcerated

in follow-up;● Had fewer than half the number of

subsequent arrests;● Ran away from programs 3 times

less often;● Returned to live with

parents/relatives twice as manydays; and

● Had significantly less hard drug usein follow-up.

In addition to studying outcomes,we were interested in identifying the“active” ingredients of MTFC. Whatabout the MTFC model makes itwork? To study this, we assessedboys after they had been in theirplacements for 3 months. Boysand their caretakers in MTFC orin Group Care (GC) were askedabout specific parenting practicesthat we hypothesized would

mediate outcomes. Thesemediators included key parenting

skills that have been noted in theliterature as relating to thedevelopment of delinquency andantisocial behavior. They included:consistent discipline, close supervision,and positive encouragement andengagement with adult caretakers. Wealso asked about the amount of timeyouth spent with delinquent peers.Significant differences were observedbetween MTFC and GC boys in several

A Glimpse at Establishing the EvidenceMULTIDIMENSIONAL TREATMENT FOSTER CARE



Data Matters

Finally, teens need to know what thelimits are through consistent disciplineand close supervision. In MTFC, fosterparents are trained and supported towork with youth in the context of thefamily and the community with aspecial focus on parenting skills shownto predict positive outcomes for boyswho have been in serious troublebecause of delinquency.

Through this process of researchand program evaluation, MTFC hasbeen established as an evidence-basedpractice having a deterrent effect ondelinquency and anti-social behavior.The evidence was established througha strong research design (includescontrol groups with randomassignment), results that showconsistent and sustained positiveoutcomes for youth beyond one year of

areas. Most notably, MTFC participantsspent more time with their adultcaretakers and less overall timewithout adult supervision; they weredisciplined more consistently for ruleviolations and misbehavior; they spentless time unsupervised with delinquentpeers; and reported less influence bydelinquent peers (Chamberlain, Ray, &Moore, 1996). The next step was to seeif the mediators related to outcomesfor boys in follow-up.

To examine this question, we useda data analysis method calledstructural equation modeling (SEM).In SEM you can look at multipleindicators simultaneously. First, wetested whether boys in MTFC hadreceived better supervision, discipline,adult mentoring, and had lessunsupervised contact with delinquentpeers than boys in GC. Next, we testedwhether boys who had thoseconditions had better long-termoutcomes, regardless of whether theywere placed in MTFC or GC. Theanswer to both questions wasessentially “yes.”

The results of this study highlightimportant components of atherapeutic environment for boys withserious delinquent or antisocialbehavior. First, since most juvenilecrime is committed by groups of youthand is a social event, parent skillstraining on supervision andmonitoring of peer relationships(although difficult) is crucial. Second,adult support and mentoring is just asimportant to teens as it is to youngerchildren. Even if teens act like they donot value adult attention— they do.Having a positive relationship with amentoring adult sets the stage forlearning new skills, for modelingappropriate social behavior, and fortaking the risks necessary to changeone’s way of acting in the communityand with peers. These types ofengagement opportunities areembedded in the MTFC approach.

A CHILD WELFARE RESOURCE REPORT

Evidence-Based Practices in Mental Health Services for Foster Youth(2002), a report written by Lynne Marsenich, LCSW was produced by

California’s Institute for Mental Health’s, Caring for Foster Youth initiative and funded by the Zellerbach Family Fund. Although focused on the promotionof mental health for foster children throughout California, the report holdsbroader interest and application for those in the mental health and foster care fields in any state.

Designed to initiate and inform interdisciplinary dialogue between mentalhealth and child welfare administrators, and practitioners, researchers, thejudiciary, families and foster youth, the report has four primary goals:

• To highlight the available social science evidence on mental health servicesfor foster children, from which service systems models can be developed

• To encourage the integration of known research into the planning,development, and delivery of mental health services to children in foster care

• To dispel some of the prevailing myths and misperceptions about the mentalhealth needs and best treatment options for children in foster care

• To outline some of the implications of this information and offerrecommendations designed for the delivery of mental health services to foster children

Ultimately, the report makes recommendations for training, research andprogram improvements for the delivery of mental health services to foster youth.

Copies of this complete report may be downloaded or ordered from the California’sInstitute for Mental Health website, www.cimh.org

treatment, and replication in differentchild systems. Additionally, we areexpanding the target population bycurrently conducting a parallel studyfor girls referred from juvenile justice. ◆

ReferencesBurns, B. J., & Hoagwood, K. (2002).Community treatment for youth: Evidence-basedinterventions for severe emotional and behavioraldisorders. New York: Oxford University Press.

Chamberlain, P., Ray, J., & Moore, K. M. (1996).Characteristics of residential care for adolescentoffenders: A comparison of assumptions andpractices in two models. Journal of Child andFamily Studies, 5, 259-271.

Chamberlain, P., & Reid, J. (1998). Comparisonof two community alternatives to incarcerationfor chronic juvenile offenders. Journal ofConsulting and Clinical Psychology, 6(4), 624-633.

For more information on MTFC andrelated topics, please contact PatriciaChamberlain at [email protected].


Psychosocial Treatments—American PsychologicalAssociation

In an effort to identify specificempirically-supported psychosocialinterventions for children, a specialtask force of the AmericanPsychological Association (APA)modified adult treatment criteriapreviously set by the APA Society forClinical Psychology (Chambless et al.,1996) for “well-established” and“probably efficacious” child treatments(Lonigan, Elbert, & Johnson, 1998).These criteria resulted in thepublication of a series of reviews in1998 that examined the efficacy of a

number of treatments forchildren. This series appeared inVolume 27 of the Journal of

Clinical Child Psychology.According to these criteria,

treatments are to be supported byeither group-design or single-subject

experiments. Such research studiesmust also clearly describe subject

characteristics. Unlike other evidence-based practice criteria, the APAstandards stress replication byindependent research teams and preferthat identified interventions havetreatment manuals. The APA taskforce defined two categories ofpsychosocial treatment by thesecriteria with examples as follows:● “Well-established” treatments are

required to have two or more studiesconducted by different researchteams that demonstrate theirsuperiority to medication, placebo, oran alternative treatment; equivalenceto an already established treatment;or 9 single-subject case studies.


Data Matters

ESTABLISHING CRITERIA & CATALOGUING EBPs

Heather Ringeisen, Ph.D.National Institute of Mental Health

This is a time of hopefulanticipation in children’smental health.

The past several years have seendramatic increases in our

understanding of successful strategiesfor the identification and diagnosis ofemotional or behavioral disorders inchildren, as well as strategies for theirtreatment and service provision.Recent reviews have identified avariety of efficacious treatments,including psychopharmacologic(Vitiello, Jensen, & Bhatara, 1999;Weisz & Jensen, 1999); psychosocial(Journal of Clinical ChildPsychology, 1998; Weisz & Jensen,1999); integrated community andprevention services (Burns,Hoagwood, & Mrazek, 1999;Greenberg, Domitrovich &Bumbarger, 2001); and school-based approaches (Rones &Hoagwood, 2000). We have learnedthat current treatments cansuccessfully reduce symptoms of childpsychopathology, improve adaptivefunctioning, and sometimes serve as abuffer to further long-termimpairment. This is not to imply thatthe knowledge base is complete, oreven sufficient. Informationsurrounding children continues to lagbehind the empirical evidence aboutadult mental illness and treatment.Nevertheless, important groundworkfor further research in child mentalhealth has been laid.

Consequently, child mental healthpractice is becoming part of a newemphasis in the development andimplementation of treatments that aresupported by positive researchfindings. Many national efforts haveestablished different sets of reviewcriteria for determining when aparticular type of intervention issupported by sufficiently positivescientific results. These efforts haveincreased attention to the quality ofparticular interventions and provide

Identifying Efficacious Interventions for Children’s Mental Health

WHAT ARE THE CRITERIA AND HOW CAN THEY BE USED?

criteria by which to understand,evaluate and select treatments forvarious mental health problems. Here,efforts by the American PsychologicalAssociation, InternationalPsychopharmacology AlgorithmProject, and prevention scientists willbe described to illustrate such criteria.

“Well-established” treatments wereidentified for attention-deficit/hyperactivity disorder, or ADHD,conduct problems and phobias.

● “Probably efficacious” interventionsare required to have two or morestudies that demonstrate theirsuperiority to wait-list control, oneexperiment meeting the criteria for a“well-established” treatment, orthree single-case studies. “Probablyefficacious” treatments wereidentified for the treatment ofdepression, anxiety disorders, ADHD,conduct problems, and phobias.

PsychopharmacologicalInterventions—InternationalPsychopharmacology Algorithm Project

Weisz and Jensen (1999) recentlyreviewed evidence on the efficacy ofchild pharmacotherapy utilizingcriteria established for the InternationalPsychopharmacology AlgorithmProject (Jobson & Potter, 1995). Bythese criteria, a drug is consideredefficacious if studied through randomassignment and control groupcomparison, and with replicated resultsin one or more similarly well-controlledstudies. Here, replication by otherinvestigators is not a criterion anddrugs can be considered efficaciousfollowing one randomized trial. Thereview identified several psychotropicmedications with empirical support forboth childhood externalizing andinternalizing disorders. In addition, theNational Institute of Mental Health(NIMH) commissioned six scientificreviews of published research on thesafety and efficacy of psychotropicmedications for children. Categories ofdrugs reviewed include: stimulantmedications, mood stabilizers, selectiveserotonin reuptake inhibitors, tricyclicantidepressants, and antipsychoticagents. These reviews can be found inVolume 38 of the Journal of theAmerican Academy of Child andAdolescent Psychiatry.

Preventive Interventions—Greenberg, Domitrovich,and Bumbarger

Recently, Greenberg, Domitrovichand Bumbarger (2001) published areview to identify universal, selectiveand indicated prevention programs thatreduce symptoms of both externalizingand internalizing childhood mentaldisorders. They focused specifically oninterventions for school-age children(5-18 years). In order to be included asefficacious programs, programevaluations required well-structuredstudy designs, clear specification ofstudy participants, a written manualthat specified intervention proceduresand outcome effects on measuresrelated to mental disorders. In theprevention review, a minimal numberof studies was not specified, but amanual was required. This reviewpublished in Volume 4 of Preventionand Treatment identified 34 programsthat met such criteria.

ConclusionsIt is important to stress what these

various criteria for efficacious childinterventions do and do not tell amental health consumer—whether an administrator, practitioner, orfamily member.

The various criteria for“efficacious” child mental healthinterventions focus on scientificvalidation, or data-based empiricalsupport. They set criteria to ensurethat scientific studies have adequatepower to detect meaningfuldifferences, sufficient researchmethods, and statistically significantfindings. So, the criteria set a scientificstandard of empirical support. Thesecriteria do not necessarily summarizean intervention’s readiness for broad-scale implementation or anintervention’s applicability for diversegroups (e.g., age, ethnicity, geographiclocation), and they do not takeindividual consumer preferences into consideration.

Lists of efficacious interventionscan be incredibly helpful in determininginterventions with the strongest scientific support; however, such lists must beindividually interpreted within a localframework. Knowledge of local needs,resources and target audiences are keytools in maximizing the usefulness ofefficacious treatment criteria. Whenexamining empirically-supportedpractices identified through various sources, administrators should still ask,“are these results applicable to my “local”population?” Practitioners shouldwonder, “are the results applicable tomy particular client?” Families shouldquestion, “are these practices right formy child?” Such consideration of bothscientific standards, local needs, andconsumer fit will enable bothresearchers and consumers to helpchild mental health practice do moreof the “right things” right. ◆

ReferencesBurns, B. J., Hoagwood, K., & Mrazek, P. (1999).Effective treatment for mental disorders inchildren and adolescents. Clinical Child andFamily Psychology Review, 2, 199-254.

Chambless, D. L., Sanderson, W. C., Shohman, V.,Bennett Johnson, S., Pope, K.S., Crits-Cristoph, P.,et al. (1996). An update on empirically validatedtherapies. The Clinical Psychologist, 49, 5-18.

Greenberg, M. T., Domitrovich, C., & Bumbarger,B. (2001). The prevention of mental disordersin school-aged children: Current state of thefield. Prevention and Treatment, 4, Article 1.

Jobson, K. O., & Potter, W. Z., (1995).International psychopharmacology algorithmproject report. Psychopharmacology Bulletin,31(3), 457-9, 491-500.

Lonigan, C. J., Elbert, J. C., & Johnson, S. B.(1998). Empirically supported psychosocialinterventions for children: An overview. Journalof Child Clinical Psychology, 27, 138-145.

Rones, M., & Hoagwood, K. (2000). School-based mental health services: A researchreview. Clinical Child and Family PsychologyReview, 3(4), 223-241.

Vitiello, B., Bhatara, V. S., & Jensen, P. S. (1999).Current knowledge and unmet needs inpediatric psychopharmacology. Journal ofAbnormal Child Psychology, 22, 560-568.

Weisz, J. R., & Jensen, P. S. (1999). Efficacy andeffectiveness of child and adolescentpsychotherapy and pharmacotherapy. MentalHealth Services Research, 1, 125-158.


Data Matters



Data Matters

Paul Brounstein, Ph.D., Deborah Stone, Ph.D., Ann Acosta, M.A,and Stephen Gardner, D.S.W.Division of Knowledge Application andSystems ImprovementCenter for Substance Abuse PreventionThe Substance Abuse and Mental HealthServices Administration

Steven P. Schinke, Ph.D.Columbia University School of Social Work

Within the Substance Abuse and Mental Health Services

Administration (SAMHSA), the Center for Substance AbusePrevention (CSAP) has made aconcerted effort to develop acomprehensive system to connect“science-based” substance abuse and mental health treatment andprevention programs with practice.The primary vehicle for this effort has been the National DisseminationSystem (NDS); the engine for thisvehicle is the National Registry ofEffective Programs (NREP—located on the World Wide Web at:www.modelprograms.samhsa.gov).The NREP helps move both theprevention field and governmentagencies, charged with bridging thegap between research and practice,towards greater accountability inpublic and private sector funding. Byoffering easily accessible informationon programs with proven evaluationresults, efforts have been made to helpprepare the prevention community forthe new performance results-orientedenvironment. The purpose of theNREP and the NDS includes:

● Making science-based programs thefoundation for national, state andcommunity efforts;

40 Promising programs. The 49 Modelsare currently being activelydisseminated, supported by print and web-based resources(www.modelprograms.samhsa.gov) as well as by training and technicalassistance made available for eachspecific model. Other Federal agencies,such as the Department of Education,are also beginning to incorporateCSAP’s Model Programs in their lists ofprograms for implementation. TheNREP has also broadened its scope byincluding programs to prevent or treatHIV/AIDS transmission, gambling,workplace substance use, post-traumatic stress disorder and violence,and in the coming months, will includeco-occurring substance abuse andmental health disorders programs. Thebroadening of NREP’s scope and theincreasing endorsement by other largerFederal agencies will have a far reachingimpact by significantly increasing thenumber of scientifically defensibleprevention programs implemented incommunities across the country.

Review ProcessPublished (e.g., peer reviewed

journal articles) and unpublishedprogram materials (e.g., grant finalreports, manuscripts underdevelopment) are submitted to NREPand distributed to teams of scientistsfor review. Team members workindependently to read, analyze, andscore each program according to 15criteria, summarized in the box.Review team members regularly meetto compare their assigned ratings,clarify any areas of disagreement, andundergo supervision for their programrating reliability.

● Recognizing that intervention effortsmust be comprehensive, yet tailoredto meet local population needs;

● Supporting the implementation ofscientifically defensible modelprograms across the country; and

● Creating a system of public andprivate partners working to develop capacity and theinfrastructure necessary to identify,implement and monitor effectiveprevention programs.

The NREP identifies three types ofscience-based programs through anexpert review process—Promisingprograms are those that have generallybeen well-implemented and evaluatedbut whose results are not consistentlypositive across domains ofmeasurement or replications; Effectiveprograms are well-implemented, well-evaluated and have demonstratedconsistent positive outcomes acrossdomains of measurement and/orreplications; Model programs share thecharacteristics of Effective programsbut also include the proviso thatprogram developers work with CSAPin the active dissemination of theprogram, providing materials, trainingand technical assistance therebyensuring localities replicating/adaptingthe program, when adapted, will notviolate the model of change used inthe program and that the program willbe implemented with strong fidelity.The agreement with developers toprovide both training opportunitiesand technical assistance is a crucialaspect of the NDS.

To date, the NREP has identified49 Model Programs, 38 Effective and

SAMHSA’s National Dissemination System for Model Prevention Programs

THE NATIONAL REGISTRY AND DISSEMINATION OF EFFECTIVE PROGRAMS



Data Matters

NREP reviewers include doctoral-level scientists, experts in preventionresearch methodology and programs,and they prepare for their taskthrough extensive training plusillustrative program reviews andcritiques. Currently, 27 scientistsconduct NREP reviews. Reviewerbackgrounds span such fields aspsychology, sociology, social work,education, public health, biostatistics,and public affairs. NREP reviewers arelargely employed in academia, but anumber are with private research anddevelopment firms, think tanks,consulting, health services, andprivate practice. Approximately one-half of all reviewers are women, and15 of the 27 reviewers are Black,Hispanic, or Asian.

Criteria and Selection ProcessThree trained reviewers

independently rate programs on aseries of criteria designed to reflectquality of implementation, user-friendliness (e.g., translations) andsolidity of the causal linkdemonstrated between interventionand outcomes. Criteria used may varya bit depending upon the topic area,but always reflect these three moregeneral items. Most often, ratings aremade on the 15 dimensions listed inthe box. If all raters score within onepoint and on the same side of themidpoint, averages are used, otherwiseconsensus conferences are held.

The final two criteria are used assubjectively scaled criteria and areused in making the determination ofprogram review status. To be identifiedas an Effective or potential Modelprogram, both utility and integrityscores must exceed 4; these scoresmust both exceed 3 to be identified as a promising program.

Other SupportsSAMHSA/CSAP has learned that

local planners and implementors needmore than written information. To

● Training and technical assistancethrough the program developers;

● Capacity building through CSAP’sState Incentive Grants and BlockGrants; and

● Prevention Program OutcomeMonitoring System (PPOMS)described below.

Prevention Program OutcomeMonitoring System (PPOMS)

To complete the system and helpto provide feedback to theidentification and disseminationefforts, CSAP will launch PPOMS inthe fall of 2003. PPOMS will help

respond to these needs, CSAP hascreated the Dissemination Systemwhich boasts the following supports:

● Achieving Outcomes (AO) trainingactivities in community-basedstrategic planning and CSAP’sDecision Support System (DSS—www.preventiondss.org)—togethercomprise a systematic approach thatguides the field to program selection;

● Program promotion usingcollaboration with national partners,web-based technology at:www.modelprograms.samhsa.gov,and print materials (e.g., Here’sProof: Prevention Works; CSAPAnnual Report);

C R I T E R I A

● Theory—the degree to which programs reflect clear, well-articulated principlesabout substance abuse behavior and how it can be changed.

● Intervention fidelity—how the program ensures consistent delivery.

● Process evaluation—whether program implementation was measured.

● Sampling strategy and implementation—how well the program selected itsparticipants and how well they received it.

● Attrition—whether the program retained participants during its evaluation.

● Outcome measures—the relevance and quality of evaluation measures.

● Missing data—how the developers addressed incomplete measurements.

● Data collection—the manner in which data were gathered.

● Analysis—the appropriateness and technical adequacy of data analyses.

● Other plausible threats to validity—the degree to which the evaluation considersother explanations for program effects.

● Replications—number of times the program has been used in the field.

● Dissemination capability—whether program materials are ready forimplementation by others in the field.

● Cultural- and age-appropriateness—the degree to which the program addressesdifferent ethnic-racial and age groups.

● Integrity—overall level of confidence of the scientific rigor of the evaluation.

● Utility—overall pattern of program findings to inform prevention theory and practice.

continued on page 14



measure the impact of thedissemination of prevention programsto the field, by attempting to quantifythe extent to which programs aredisseminated, how they are adapted forthe field, and what outcomes theprograms produce. Data generated byPPOMS will allow CSAP to quantifythe market penetration, processes, andeffectiveness of its science-basedprogram replications. The nationalPPOMS assessment will ask preventionpractitioners about their use of,modifications to, and satisfaction with science-based and otherprevention programs.

PPOMS information will allowCSAP to better direct its disseminationof NREP-identified programs andprovide access to targeted training andtechnical assistance for practitioners.Equally important, PPOMS findingswill shed new light from the field onthe core components of science-basedprograms and how fidelity andadaptation contribute, and are related,to programmatic outcomes.

Advancing Science InstituteThis past year, CSAP held its first

Advancing Science Institute in whichprograms, not meeting the criteria forPromising program status were invitedto review their intervention andevaluation designs with an eyetowards building their evidence base.This activity will be broadened to bringmore “home grown” programs intothe fold of effective, evidence-basedefforts. In accomplishing this, CSAPwill both provide information forservice to inform science as well asprovide communities with a broaderselection of readily implementable,effective programs. ◆

Data Matters

Sharon F. Mihalic, M.A.Center for the Study and Prevention of Violence

Blueprints for Violence Preventionbegan at the Center for the Study

and Prevention of Violence (CSPV), asan initiative of the State of Colorado,with initial funding from the ColoradoDivision of Criminal Justice, theCenters for Disease Control andPrevention, and the PennsylvaniaCommission on Crime andDelinquency. With later support fromthe Office of Juvenile Justice andDelinquency Prevention, Blueprintshas evolved into a large-scaleprevention initiative, in bothidentifying model programs andproviding technical support to helpsites choose and implement programswith a high degree of integrity.

The identification of effectiveprograms has been in theforefront of the nationalagenda on violenceprevention for the lastdecade. Today, afterreviewing over600 violencepreventionprograms, theBlueprints initiativehas identified 11 modelprograms and 21 promising programs.Taken together, these programs targetpopulations spanning thedevelopmental age range, from birth to19 years. In addition, these programsboth prevent violence and treat youthalready displaying problem behaviors.

Over the past decade, manyorganizations have produced lists ofprograms and practices thatdemonstrate at least some evidence of

effects on violence/aggression,delinquency, substance abuse, andtheir related risk and protectivefactors. Although these lists provide avaluable resource for the community,they can be confusing to the public.First, most differ in focus, with somelists being quite narrow; for example,limiting their descriptions to drugabuse, family strengthening, or school-based programs only. Secondly, andperhaps more importantly, the criteriafor program inclusion varytremendously from list to list, withsome agencies adopting a morerigorous set of criteria than others.

The Blueprints initiative likelyutilizes the most rigorous set of criteriain the field. However, this highstandard is necessary if programs areto be widely disseminated, for whenthis occurs, it will not always be

possible to conduct localevaluations to determine

if programs aredemonstrating the

intended results.Therefore, it is

important thatprograms

demonstrateeffectiveness,

based on a rigorous evaluation, priorto their widespread dissemination.

Blueprints Selection CriteriaBlueprints model programs meet

such a standard, and there iswidespread consensus that Blueprintsprograms are effective interventions.Although a program model can rarely,if ever, be proven superior to all others,a particular model elicits greaterconfidence after its theoreticalrationale, goals and objectives, and

Blueprints for Violence PreventionTHE IDENTIFICATION OF EFFECTIVE PROGRAMS

SAMHSA continued from page 13



Data Matters

outcome evaluation data have beencarefully reviewed. In turn, acommunity that implements such astrategy has a greater likelihood of asuccessful violence prevention effort.

Blueprints programs meet rigoroustests of effectiveness in the field byidentifying three important factorswhen reviewing program effectiveness:evidence of deterrent effect with astrong research design; demonstrationof a sustained effect; and multiple sitereplication. Programs meeting allthree of these criteria are classified as“model” programs, while programsmeeting at least the first criterion areconsidered “promising.” A summary ofthe criteria is provided below.

● Evidence of Deterrent Effect with aStrong Research DesignAll Blueprints programs mustdemonstrate evidence of a deterrenteffect on problem behavior—violence(including childhood aggression andconduct disorder), delinquency,and/or drug use. This evidence mustbe based on a strong researchdesign, as this is the most importantof the selection criteria, throughsufficient quantitative data todocument effectiveness in preventingor reducing these behaviors and theuse of experimental designs withrandom assignment or quasi-experimental designs with matchedcontrol groups.

Further, the programs must havethe following quality factors: 1)sample sizes large enough to providestatistical power to detect at leastmoderate sized effects, 2) lowattrition to ensure integrity of theoriginal randomization or matchingprocess to allow generalization offindings, and 3) consistent measuresand administration.

● Sustained EffectsMany scholarly reviews classify aprogram as effective if itdemonstrates success by the end of

the treatment phase. However, it isalso important that program effectsendure beyond treatment, and fromone developmental period to thenext. For these reasons, designationas a Blueprints program requires asustained effect at least one year beyondtreatment, with no subsequentevidence that the effect is lost.

● Multiple Site ReplicationReplication is an important elementin establishing program effectivenessand understanding what works best,in what situations, and for whom.Some programs are successfulbecause of unique characteristics inthe original site that may be difficultto duplicate in another location (e.g.,the presence of a charismatic leaderor extensive community support andinvolvement). Replication establishesthe strength of a program and itsprevention effects by demonstratingthat it can be successfullyimplemented in other sites.

Programs that have demonstratedsuccess in diverse settings (e.g., urban,suburban, and rural areas) and withdiverse populations (e.g., differentsocioeconomic, racial, and culturalgroups) create greater confidencethat such programs can betransferred to new settings.Becoming a Blueprints modelprogram requires at least onereplication with fidelitydemonstrating that the programcontinues to be effective.

SummaryThe Blueprints selection criteria

establish a high standard of programeffectiveness that has proved difficultfor most programs to meet, thusexplaining why only 11 modelprograms have been identified to date.Although rigorous, this standardreflects the level of confidencenecessary for recommending thatthese programs be widely disseminated

and to provide communities thatreplicate these programs withreasonable assurances that they willprevent violence when implementedwith fidelity. The Blueprints initiativewas never intended as a means ofcompiling a comprehensive list of allprograms that had some evidence ofeffectiveness. Instead, the modelprograms, in particular, were selectedto reflect programs with very strongresearch designs that demonstratedevidence of effectiveness indelinquency, violence, or substanceabuse prevention and reduction.

It is important to remember thatprograms not on this list are notnecessarily ineffective. In fact, it islikely that there are many goodprograms that have not yet undergonethe rigorous evaluations needed toqualify as a Blueprints program.Similarly, there are other programsthat have demonstrated effectivenessin outcomes not considered by theBlueprints. Nonetheless, our work hasrevealed that many prevention andintervention programs are ineffective,and a few are iatrogenic (i.e., harmful).Thus, it is critical that outcomeevaluations are performed and resultsmade available to the community.Without this information, we cannotdetermine what programs work, norcan we be confident that children arebenefiting from these efforts. CSPVcontinues to review new researchfindings, and we hope to continue toexpand our list of Blueprints programsto include other credible, effectiveinterventions that can be confidentlyimplemented by communities. CSPValso reviews on-going evaluations ofall the Blueprints programs, to refineour knowledge of their sustainedeffects, as well as their adaptability toother populations and settings. ◆

For further information, please visit ourwebsite www.colorado.edu/cspv/blueprints/or email [email protected]


• Model• Promising• Effective

CSAP focuses on the effectiveness and impact of substance abuse prevention efforts. Programs are scored on a 5 point scale based on15 criteria with 1 being the lowest and 5 being the highest score. Model programs are well implemented and evaluated according torigorous standards of research, scoring at least 4.0 on the 5-point scale. Promising programs have been implemented and evaluatedsufficiently and are considered to be scientifically defensible, but have not yet been shown to have sufficient rigor and/or consistentlypositive outcomes required for Model status. Promising programs must score at least 3.33 on the 5-point scale. Effective programsmeet all the criteria as the Model programs, but for a variety of reasons, these programs are not currently available to be widelydisseminated to the general public.

FOCUS AND CRITERIARATING CATEGORIES

CENTER FOR SUBSTANCE ABUSE PREVENTION (CSAP)DEPT. OF HEALTH & HUMAN SERVICES, NATIONAL REGISTRY OF EFFECTIVE PROGRAMS http://modelprograms.samhsa.gov

DEPARTMENT OF EDUCATION, SAFE AND DRUG-FREE SCHOOLS www.ed.gov (Visit U.S. Department of Education and search for OSDFS)

• Exemplary• Promising

Relevant outcomes are related to making schools safe, disciplined, and drug-free: reducing substance use, violence, and otherconduct problems and promoting positive changes in scientifically established risk and protective factors. Program criteria iscarefully and thoroughly described on the website, and includes (1) evidence of efficacy/effectiveness based on a methodologicallysound evaluation that adequately controls for threats to internal validity, including attrition; (2) the program’s goals with respect tochanging behavior and/or risk and protective factors are clear and appropriate for the intended population and setting; (3) the rationaleunderlying the program is clearly stated, and the program’s content and processes are aligned with its goals; (4) the program’s contenttakes into consideration the characteristics of the intended population and setting; (5) the program implementation process effectivelyengages the intended population; (6) the application describes how the program is integrated into schools’ educational missions; and(7) the program provides necessary information and guidance for replication in other appropriate settings.

STRENGTHENING AMERICA’S FAMILIES www.strengtheningfamilies.org

SURGEON GENERAL’S REPORT (2001) U.S. Department of Health and Human Services • www.surgeongeneral.gov/library/youthviolence

• Exemplary I• Exemplary II• Model• Promising

The National Program Review Committee, the University of Utah, and CSAP reviewed the programs that focused on family therapy,family skills training, in-home family support, and parenting programs. Each program was rated on theory, fidelity, sampling strategy,implementation, attrition, measures, data collection, missing data, analysis, replications, dissemination capability, cultural and ageappropriateness, integrity, and program utility and placed into the following categories:• Exemplary I: Program has experimental design with randomized sample and replication by an independent investigator. Outcome

data show clear evidence of program effectiveness.• Exemplary II: Program has experimental design with randomized sample. Outcome data show clear evidence of program effectiveness.• Model: Program has experimental or quasi-experimental design with few or no replications. Data may not be as strong in

demonstrating program effectiveness.• Promising: Program has limited research and/or employs non-experimental designs. Data appears promising but requires

confirmation using scientific techniques.

• Model• Promising:– Level 1: Violence

Prevention– Level 2: Risk

Factor Prevention

The primary focus of the report by the Surgeon General is violence prevention and intervention. The criteria the Surgeon General setwere appropriately rigorous methods of inquiry and sufficient data to support the conclusions. Model programs have rigorousexperimental design (experimental or quasi-experimental), significant effects on violence or serious delinquency (Level 1) or any riskfactor for violence with a large effect size of .30 or greater (Level 2), replication with demonstrated effects, and sustainability of effect.Promising programs meet the first two criteria (although risk factors of .10 or greater are acceptable), but programs may have eitherreplication or sustainability of effects (both not necessary).


Data Matters

AGENCY AND PRACTITIONER RATING CATEGORIES AND CRITERIA FOR EVIDENCE BASED PROGRAMS

Compiled by Sharon F. Mihalic, M.A.Center for the Study and Prevention of Violence

The following chart identifies a sample of federal and private agencies who have rated the effectiveness of preventionprograms designed to reduce or eliminate problem behaviors, such as delinquency, aggression, violence, substance use,

school behavioral problems, and risk factors identified as predictive of these problems. This chart describes the set ofcriteria that has been identified for program inclusion by each agency and also describes the focus of each work (i.e.,school-based programs, violence programs, substance abuse programs, etc.).

The actual Matrix of Programs, not presented in this document, is a table listing approximately 300 programs thathave been rated by each agency as effective. The Matrix of Programs can aid the practitioner by showing how variousprograms have been rated across different agencies. Look for the Matrix of Programs, developed by Sharon Mihalic atwww.colorado.edu/cspv/blueprints.



A. Elaine SlatonNational Federation of Families forChildren’s Mental Health

As a parent whose son receivedpsychotropic medications for which

there was no research base for use inchildren—and therapy for which therewas no evidence base to indicate a highlikelihood that it would help, not harm,I am indeed invested in the search forevidence of effectiveness in children’smental health. As a children’s mentalhealth advocate I am further investedin not wasting ever-shrinking funds forservices. I am, however, alarmedabout—even frightened of—thecurrent push to bring evidence-basedpractices (EBP) to scale.

Evidence-based practices, as Iunderstand them, are● service programs that have met strict

scientific standards of effectiveness;● programs that require intensive

training and supervision to ensurefidelity to the model;

● a fairly short list, includingMultisystemic Therapy (MST);Functional Family Therapy (FFT);and Multidimensional TherapeuticFoster Care (MTFC); and

● “proof ” of what works and is costeffective for decision makers.

“Strict scientific standards ofeffectiveness” are largely mainstreamacademic research criteria that lackthe depth of diverse “ways ofknowing.” I am concerned that theexperience, cultures, traditions, andknowledge of families of children andyouth with emotional and behavioraldisorders have not significantlyimpacted the designs of the practices,

the research, or the criteria by whichthese practices have been deemedevidence-based. Voices of youth, ofcommunity elders, of natural healersare missing in the “evidence” definedby strict scientific standards ofeffectiveness. My son and our family,like many others, found our mostsignificant healing throughceremonies in Indian Country. Wewere welcomed, despite our whiteness,to participate in healing ways thatcenturies of natural healers knewworked. There was no scientificresearch behind these ceremonies, butdeep indigenous knowledge of theireffectiveness. These are the kinds ofservices that will go unfunded andunrecognized if EBPs continue to bedefined as they are currently. Nativechildren and their families—as well aschildren and families of othercultures—will be further denied accessto their indigenous ways of healing.

“Fidelity to the model” means thatthe model practice must beimplemented in exactly the sameway—regardless of the race or cultureof the people, the geographic location,the presence of natural helpers andhealers, or the traditional ways of

healing indigenous to the families.The “short list” of practices

approved as evidence based to date,does not include the services andsupports families across this countryhave come to identify as critical to thehealing process for their children andfamilies. That is, the list does not includerespite, wraparound, traditional nativeceremonies, or equine therapy1. Asfunding for children’s mental healthservices tightens, what will happen to theservices we have defined as critical to ourhealing if they are not on the EBP list?

Questions we families—and thosewho support us—must ask:● Who developed the programs? (By

“who”, I mean their race, culture,relationship to children withemotional and behavioral disorders,and if their knowledge base isacademic or experiential or both?)

● Who selected the outcomes?● Who defined terms, such as

effectiveness and success?● Who was included in the research?

Who was not included?● Who defined the criteria by which a

practice is deemed Evidence-Based?● Whose money paid for the program

development and the research—and,who will benefit financially from thereplication of these practices?

As a trainer of the Federation ofFamilies for Children’s Mental HealthEvaluation Skills Training for Familiesand Youth, I have become keenlyaware that families and youth fear thepotential racist implications—evidentor not—in research. If families are leftout until a practice has been deemedevidence based, they/we will not trustit and will not readily advocate for it. Ifthe program development, theresearch, and the EBP criteria are notinclusive of all voices, they have not

Data Matters


1Animal assisted therapy where horses and humans interact through defined stages and activitiesthat can benefit individuals with both physical and emotional difficulties. This method has beenapplied to individuals with physical disabilities, alcohol dependency, drug abuse, and those involvedwith the juvenile justice system.

A Family Perspective on Evidence-Based PracticesINVOLVING DIVERSE CONSUMERS: FAMILIES, COMMUNITIES, & PROVIDERS


Data Matters

INVOLVING DIVERSE CONSUMERS: FAMILIES, COMMUNITIES, & PROVIDERS

Jan KamaradGrand Island, Nebraska

My son, Brent is a child withADHD. I was quick to learn about

the ADHD disorder; however, what waslearned was not completely applied. Ihad often found myself using phrases“how many times have I told you,” or“if I have told you once I have told youa thousand times.” We had beentreating his ADHD with both behaviormodification and medication withlimited success. Our family reached acrisis point when Brent becameinvolved in the Juvenile Justice systemand was about to be removed from ourhome. Brent’s probation officersuggested MST therapy as somethinghe had heard about and could only tellme that it was family-based therapy. Wewere desperate and would try anything.

When our therapist first met withme, I asked some basic questions like:“OK now tell me about this, what doesMST stand for? What are yourobjectives or goals for my son and myfamily? How do I know that thistherapy works?” He was veryknowledgeable about the therapymodel, and so positive. I don’tremember any research data orspecific evidence to convince me thatMST would work. What I do remembermost was his positive approach;wanting me to make my owndetermination—of what my goalswere for my son and for my family; andhis support for the fact that I know mychild best!

MST is a strengths focusedprogram. The strengths of theindividuals and family unit were

Through MST, I changed my styleof interacting with my children andproblem solving. Our therapist’sinstructions were “Do not engage inbattle.” When you find yourselfengaging, you have reacted to theproblem, not solved it. I know I amreacting when I use phrases like “Whydid you?” or “What were youthinking?” With this silly question theanswer was of course, “I don’t know.”My reaction would be stronger and thebattle would be on. “Do not engage” isthe hardest step. First, I have to decide ifthe battle is mine, and if it is, do I wantto take it on. Second, I have to learn toslow down, that if a problem isn’tsolved in an hour, it is okay. I think theword the therapist used here is patienceand I am continually working on that.Our therapist taught me to look for thecause of a problem rather than tosimply react. He would say “Gee, Jan,what do you think is causing this?” Hemade me examine what was taking

MANAGING SUCCESS TOGETHERThe Evidence Base for MST from

One Family’s Perspectivepointed out and they became our focusand we built on them. Duringstrengths discovery I was asked to listmy goals as a parent. Each goal startedwith “Brent needs to...” or “I wantBrent to...” With the strengthsdiscovered and the goals stated wewent into a 5-month intensive process.As we were preparing for release ourtherapist once again asked me to writemy goals for parenting. A few of mygoals are: listen well, talk openly,lighten up and don’t sweat the smallstuff. As our therapist and I discussedthese goals I realized how the focushad shifted from Brent to me. I nolonger rely on Brent to meet my goals!

Multi Systemic Therapy (MST)helped me to gain an understanding ofmy son. I now understand how towork with Brent usingstraightforward, clear directives andan immediate-gratification learningstyle. This helped bring ADHD from adiagnosis to the reality of life.

Editor’s Note: Ms. Kamarad has shared her personal family story to illustrate theimpact of one evidence based practice that helped her to reach her personal and familygoals. Her evidence or measures of success were “listening well, …talking openly,…problem solving, …decision making, …empowerment, …relaxed parenting, …and funwith my children.” Is this the language of evidence base? From this parent’sperspective, yes. In addition, her story raises many important questions related to thefamily perspective on evidence based practices: How do families find out about evidencebased practices? What do families want or need to know about evidence based practices?How can information about the practice and the “evidence” be made most meaningfulto families? What is “informed consent” to pursue a particular therapeutic approachwhen a decision based on evidence is desirable? How do families define their ownevidence and whether a particular practice will be or was helpful to them? These andother questions should be considered by both families and clinicians as they enter atherapeutic relationship and engage in intervention and strategies toward change.

critical to the development ofsuccessful program practices for allpeople seeking help, regardless of raceor ethnicity.

Why is it missing?There are several reasons for the

disparity of information on the mentalhealth needs and evidence-basedpractices (EBPs) for racial and ethnicminorities. Methodological challengessuch as identification and recruitmentof participants, and reluctance andresistance on the part of diversepopulations are often encounteredwhen studying the mental health ofminorities. For these reasons andothers, most of the efficacy studies ontreatment interventions included, ifany, very small numbers of racial andethnic minorities. Therefore, group-specific analyses to determine efficacywere not possible and results not

generalizable to these sub-populations. Growing

effort by federalfunding agenciesand the researchcommunity to

increase thenumber of racialand ethnicminoritiesincluded in

research willadvance our

knowledge for thisgrowing population.

Another reason forlimited information on EBPs for


Data Matters


place. I learned that I was quick toblame Brent as the problem ratherthan to find the cause of the problemand resolve it.

With that said how do you find thecause of the problem and solve it? I usea technique that I call the “decisionmaking process.” In the center of a circle I write the problem, I have feelers coming out of the circle where I put the triggers of the problem and under thetrigger I list possible solutions. Resolvethe trigger, you resolve the problem.With some problems you will have several triggers, remember patience andwork on one at a time! I ask my children to contribute their ideas and solutionsduring the process. This process helpsme enormously and I use it in both mywork and personal life. At times, thesolution is not immediate and you mayhave to try different strategies but theend result is always positive.

One of the family gains from MSTis the empowerment. As a parent,sitting across from any professionalcan be a very intimidating experience.MST’s strong statement that “youknow your child best” taught me that Ican advocate for my children in theschool, in the justice system and in thecommunity. If I don’t try to set mychildren up for success, who will?

What I enjoy most that MST hastaught me is relaxed parenting. I havefun with my children; I no longercontrol their lives but contribute totheir decision making. My children lookto me to guide them and offer support.After our release from MST, I wrote thedirector to thank him and told himMST should not be known as MultiSystematic Therapy but as ManagingSuccess Together. With Brent now ajunior in our public high school andhis sister, Shayla, in her first year atmiddle school, this is what our familydoes. This is not to say we do not haveour conflicts—we do—we just knowhow to manage them now! Our familyis evidence that MST is effective. ◆

Rachele C. Espiritu, Ph.D.National Technical Assistance Center for Children’s Mental HealthGeorgetown University

Where’s the evidence?The evidence base for racial and

ethnic minorities is alarminglyincomplete. According to a specialanalysis performed for the SurgeonGeneral’s Report on Mental Health:Culture, Race, and Ethnicity (2001),information on race or ethnicity wasnot available for nearly half of the10,000 participants included inclinical trials used to generatetreatment guidelines. Furthermore,very few minorities were included intrials reporting data on ethnicity andnot a single study analyzed the efficacyof the treatment by ethnicity or race.Unfortunately, very little is knownabout the effectiveness of treatmentsfor ethnic minorities (Bernal, 2001).As the strong movement towardsevidence-based practicescontinues, one thing isclear: more researchis needed tounderstand theimpact of culture onmental health andaccess to effectivemental services.Understanding the riskand protective factors thatcut across culturesand those factorsthat are uniqueto specificcultures is

What About Promotoras, Shamans, and Kru Khmers?

THE NEED TO EXPAND THE EVIDENCE BASEFOR DIVERSE COMMUNITIES



Data Matters


communities of color is related toexclusion. The standards of evidence-based practices often exclude the fewexisting efficacy studies on specificsub-groups due to their small samplesize. As Bernal & Scharron-del-Rio(2001) point out, the criteria ofefficacy research often emphasizesinternal validity (whether observedchanges can be attributed tointerventions) over external validity (generalizability).

What do we know?What do clinicians, agencies,

organizations, administrators do iffaced with a clinical problem for whichthere are no randomized, controlledtrials and no good evidence-basedopinions? What types of programsexist for communities of color?

While various federal agencies and research organizations haveestablished different levels of evidencebased on the evaluation design,outcomes, replication anddissemination, the criterion ofcultural appropriateness is oftenmissing. Fortunately, the Office ofJuvenile Justice and DelinquencyPrevention (OJJDP) and the SubstanceAbuse and Mental Health ServiceAdministration’s Center for SubstanceAbuse Prevention (CSAP) did includecultural and age appropriateness ascriteria in their process of identifyingeffective programs. They state thatcultural- and age-appropriateness area hallmark of programs that have beentested with diverse groups ofparticipants. For example, the “bestpractice” family strengtheningprograms identified by OJJDP incollaboration with CSAP includesFunctional Family Therapy (FFT), anempirically-grounded, family-basedintervention program for acting-outyouth. While FFT was originallydesigned to treat middle class families

with delinquent and pre-delinquentyouths, according to the website, theprogram has recently included poor,multi-ethnic, and multi-culturalpopulations. Further, almost all of themodel programs identified by CSAP’sNational Registry of EffectivePrograms indicate use with multipleethnic groups.

What about curanderismo, qi gong,or talking circles? Building from theground up

The movement towards evidence-based practices may leave behindtraditional therapies, such ascuranderismo (folk/medical beliefs,rituals practices that address thepsychological, social and spiritualneeds of Mexican and Mexicanpopulations), “talking circles” of theNative Indian community, and othertraditional remedies. Additionally, well before there were social workers,psychologists, and psychiatrists,traditional, faith-based healers such as the Cambodian Kru Khmer andshamans were around. Despite having many years of practice-basedevidence and experience to supportthese therapies, the lack of “crediblescientific evidence” often devalues the use of traditional treatments.Communities of color often don’t have the capacity to build thenecessary research.

While research is warranted onthe adaptability of evidence-basedpractices for other racial or ethnicgroups, Bernal and Scharron del Rio(2000) suggest a different focus.Rather than using a comparativeapproach, they suggest a focus on thetreatment of specific ethnic minoritygroups. In other words, rather thanmaking ethnic comparisons acrosstreatment outcomes, documenting why or what makes a treatment workis more important.

What is needed?Research is necessary to

understand factors that mightinfluence the efficacy of interventionswithin a specific racial or ethnic group,and the adaptability of interventionsto other racial or ethnic groups. Whilefidelity to the EBP intervention is oftenrequired, language and theacculturation levels of the populationserved often necessitate modificationsto the delivery of the service andtranslations of materials.Understanding the traditional valuesand beliefs about mental health ofracial and ethnic minorities can alsohelp the program developers andgroup leaders improve the program’seffectiveness for these populations.Additionally, workforce training forproviders and researchers, newtraining models, and the enhancementof consumer/family advocacy forcommunities of color can help improvethe capacity to provide culturallycompetent services.

SummaryKey questions regarding the

nature of evidence remain: in additionto evidence of effectiveness andefficiency, evidence related to thetransportability, implementation,dissemination of EBP to communitiesof color, are important to examine.The provision of the highest standardof mental health services that areculturally and linguisticallyappropriate and accessible for allindividuals regardless of race orethnicity should continue to challengethe EBP movement. ◆

ReferenceBernal, G., & Scharron-del-Rio, M. (2001). Are empirically supported treatments valid for ethnic minorities? Toward an alternative approach fortreatment research. Cultural Diversity and EthnicMinority Psychology, 7(4), 328-342.

Promotoras continued from page 19


Data Matters

Michael Hurlburt, Ph.D.Children’s Hospital, San Diego

Penny Knapp, M.D.California Department of Mental Health

There is a growing recognition ofthe importance of using evidence-

based practice in mental health.However, evidence-based practices(EBPs) are variously defined as: a)practices that major organizationshave endorsed, b) practices that arisefrom a strong foundation of basicresearch, c) practices that have somekind of outcome evidence to supporttheir use, and/or d) practices that meeta defined threshold of sufficientoutcome evidence to support their use.The phrase evidence-based practice isregularly used to mean one or more ofthese different things and it seemsunlikely that consensus will be reachedabout any single definition for the term.

Given a term with many potentialmeanings, it should not be surprisingthat there is no single answer to thefrequently asked question: “Why, giventhe presence of evidence, are evidence-based practices not employed asfrequently as might seemappropriate?” In fact, the very breadthof the term “evidence-based practice”is certain to result in different answersto this question and to differences ofopinion across individuals.

Our approach to answering thisquestion was to discuss specificexamples of evidence-based practiceswith potential consumers defined as

individuals or groups having asignificant stake in whether particularpractices are adopted, including at aminimum, administrators in mentalhealth service systems, programmanagers (i.e., supervisors in servicedelivery organizations or units), clinicalstaff members, and families of childrenreceiving services. In our researchactivity, we talked with these potentialconsumers about their evaluations ofthe evidence for specific practices andabout barriers to and facilitators oftheir use. This article describes ourresearch that took place in California.

In 2000-2002, in partnershipwith researchers from a consortium of

four mental health service researchcenters1, the California Department ofMental Health undertook a large studyof specific aspects of outpatient mentalhealth care quality for children andadolescents in California. Given thelikelihood that future improvements inservices could be driven by EBPs, onecomponent of the research led by thisconsortium focused on understandinghow potential consumers of suchpractices evaluate the relevant evidenceand what they perceive as barriers to andfacilitators of the use of specific EBPs.

Potential EBP consumers wereselected from three counties inCalifornia. The counties were chosen fortheir diversity along a number ofdifferent dimensions, such as theracial/ethnic composition of thepopulations they served and the totalcounty population size. Within eachcounty, a structured process was used toselect a representative from each of theconstituent groups mentioned above;members were not selected based ontheir opinions about the concept ofevidence-based practice. In fact, anexplicit goal was to hold discussions withindividuals who had not necessarilyadopted or considered adopting anyEBP. In total, about 15 peopleparticipated in a series of five, four-hourmeetings taking place over five months.The meetings focused on threepsychosocial interventions for youthswith disruptive behavior problems thathave a relatively solid evidence baseunder any of the criteria listed at theoutset, including Parent ChildInteraction Training (PCIT), TheIncredible Years Basic program, and

The New Consumers ofEvidence-Based Practices

REFLECTIONS OF PROVIDERS AND FAMILIES

1Research Center on Managed Care for Psychiatric Disorders, UCLA; Child and Adolescent Services Research Center, Children’s Hospital, San Diego; National Research Center on AsianAmerican Mental Health, UC Davis; Center for Mental Health Services Research, UC Berkeley, and UC San Francisco. continued on page 22



Cognitive Problem Solving SkillsTraining (see e.g., Kazdin, Siegel, &Bass, 1992; Schumann, Foote, Eyberg,Boggs, & Algina 1998; Webster-Stratton,Hollinsworth, & Kolpacoff, 1989).

Prior to and during meetings,research team members providedparticipants with a detailed summaryof one of these practices and theresearch evidence supporting its use.We felt that it was crucial forparticipants to discuss informationabout specific practices rather thanconsider the less well-defined conceptof evidence-based practice. The bulk oftime in each meeting was devoted todiscussing participants’ evaluations ofthe research evidence and theiropinions about things that wouldfacilitate or impede the use of eachintervention in their organizationand/or county public mental healthsystem. Research team membersfacilitated the meetings and, whennecessary, assumed the role ofclarifying information about the threepractices. They explicitly avoidedserving as advocates for any of thethree practices, spending most timefacilitating the discussion and probingparticipants’ comments. Each meetingwas guided by a small number ofprompts intended to facilitatediscussion of the essential questionsoutlined above.

As in any group discussion, fullconsensus did not occur on everyissue, but some important centralobservations emerged from the seriesof meetings.

Lack of familiarityAlthough participants reported

familiarity with the term “evidence-based practice,” most participantswere not aware of the interventionsdiscussed in the meetings. Those thatwere familiar had limited knowledge ofthe research and outcomes associated

included priorities such as: 1)improving the System of Care culture,2) human resources: improving accessfor non-English speakers and findingsufficient psychiatry time, 3) settingstandards, 4) increasing consumerinvolvement in service planning, 5)expanding access to services, and 6)decreasing use of residential treatment services.

Locus of responsibilityWhen asked how changes in

practices might come about,participants generally reported thatindividual service deliveryorganizations were responsible forinitiating efforts to utilize effectivepractices. In the face of competingpriorities, staff members and programmanagers indicated that it was difficultto initiate, sustain, and modify newpractices without broader supportfrom some level higher than theindividual service organization. Itappeared that there was not yet awidely recognized locus ofresponsibility, either within counties,or across counties, for encouragingand supporting the use of EBPs.

In addition to the participantresponses that contributed to thesefour central observations, otherimportant perspectives also emerged.Participants were able to suggestadditional data collection that wouldmake research more compelling topotential consumers, and they wereable to make suggestions formodifications of interventions thatthey felt would increase their potentialapplicability. For example, participantsfelt that the placement of EBPs in theirpriority list would increase if researchresponded better to the specificquestions of interest to them (e.g.,“Can PCIT be delivered effectively in asingle room or in someone’s home?”;“How would reports of satisfaction

with each practice. Despite efforts bythe research team members tosummarize results relevant to eachintervention in a consistent, digestiblemanner, it was clear that evaluatingthe evidence and its implications was adifficult and time- consuming activitythat participants did not regularlyhave time to undertake in their own schedules.

Results not convincingParticipants did not generally feel

that the studies summarized insupport of each intervention providedcompelling evidence that the practiceswould be effective with the childrenand families served in their settings orwould be worth the investmentrequired. In particular, participantswere concerned that the clinicalcomplexity of families served in theirown service populations would besignificantly greater than that offamilies served in research settings,and that interventions had not beenused with the array of racial/ethnicgroups served in California.Participants also felt that the specificprocedures employed in several of theinterventions were too costly and notbroadly applicable withoutmodification.

PrioritiesIn an exercise asking about

priorities for improving services,participants listed many of their ownimportant priorities for improvingcare. This exercise was conducted inthe initial meeting and at the end ofthe series of meetings. Participantsrarely mentioned incorporating EBPsas one of their top priorities, even afterreviewing data for these threeinterventions. Participants reportedhaving a number of other priorities towhich they devoted time. These variedsomewhat by individuals’ roles, but

Data Matters


The New Consumers continued from page 21

evaluation of services for childrenwith mental health issues and theirfamilies? The family movement,numerous private foundations, and theFederal government, have committedto finding more appropriate andaccessible services that are family-driven, individualized, and culturallycompetent in an effort to rectify thecrisis in the children’s mental healthsystem. We must not let go of thesegoals while focusing on EBPs. ◆

2Author of Democracy and Technology andfounder of the Loka Institute, board meeting, 2002.3Report of the Surgeon General’s Conference onChildren’s Mental Health, 2000.

ReferencesKazdin, A. E., Siegel, T. C., & Bass, D. (1992).Cognitive problem-solving skills training andparent management training in the treatmentof antisocial behavior in children. Journal ofConsulting and Clinical Psychology, 60, 733-747.

Schumann, E. M., Foote, R. C., Eyberg, S. M.,Boggs, S. R., & Algina, J. (1998). Efficacy ofParent-Child Interaction Therapy: Interimreport of a randomized trial with short-termmaintenance. Journal of Clinical ChildPsychology, 27, 34-45.

Webster-Stratton, C., Hollinsworth, T., &Kolpacoff, M. (1989). The long-termeffectiveness and clinical significance of threecost-effective training programs for familieswith conduct-problem children. Journal ofConsulting and Clinical Psychology, 57, 550-553.

Correspondence related to this article canbe directed to Michael Hurlburt [email protected] or at the Child andAdolescent Services Research Center;3020 Children’s Way, MC 5033: SanDiego, CA 92123


Data Matters

differ if an independent consumeradvocate interviewed individuals whohad received a specific kind ofservice?”), and if a clear andsustainable locus of responsibilityexisted for encouraging and supportingadoption and application of EBPs.

Many of the lessons enumeratedhere may not come as a surprise.However, to the degree that they areaccurate, there are importantimplications. First, closer and ongoingpartnering of researchers withpotential consumers of EBPs may becrucial to helping consumers to makedecisions in light of available evidence.Second, closer and ongoing partneringof researchers and consumers wouldlikely suggest avenues for research(both the type of data collected and the variations of practices developedand tested) that would yield findingsmore relevant to consumers in theirroles as advocates and decision


makers. Third, it may be very helpfulto have clear and sustainable local loci of responsibility for encouragingand supporting systematic use of EBPsin defined geographic areas. Obstaclesto self-initiation on the part ofindividual service organizations appear to be large.

In closing, it is worth noting thatthese discussions intentionally focusedon a small subsample of practices thatcould be called evidence-based. Thefindings are not likely to be applicableto all practices that could be labeledevidence-based. However, we believethat the most important results have todo with differences in questions andobjectives held by researchers andpotential consumers of EBPs thatcontribute to the gap between research and practice. Active models of ongoing collaboration may be acentral component to broader or morerapid experimentation with EBPs inservice systems. ◆

been examined with the racial lensesessential to ensuring safety for all.

Until we find a better way, a waythat honors all people, their culturaltraditions, their race, their ways ofknowing and of healing, we must stayvigilant. We must ask questions. Wemust learn the language of researchand evaluation in order to speak thelanguage of EBP. We must createpartnerships—across racial andcultural lines, across the boundariesdrawn between professionals andfamilies—to affect the systems changethat will help our children, ourfamilies, and our communities. Wemust advocate for research bases forthe services and supports that are

important to us. And we mustparticipate in that research.

Richard Sclove2 recently said thatresearch, science and technologyshould serve communities andhumanity and not exacerbatecontemporary crises. The SurgeonGeneral’s National Action Agenda refersto children’s mental health as a publiccrisis3. This contemporary public crisisreaches every community in thiscountry and I am not convincedresearch and science—as being appliedin EBP—will not make it worse. Willwe revert to expert-based decision-making and ignore the progress madetoward more democratizeddevelopment, implementation and

A Family Perspective continued from page 17

Copies of Data Matters may be distributed freely. If you have ideas for future issues or if you have suggestions/corrections for the mailing list, please send information to:

Larke N. Huang ● Fax: 202-687-1954 ● e-mail: [email protected] and current issues of data matters may be found on our Web site: <www.georgetown.edu/research/gucdc/eval.html>

E. Wayne Holden, Ph.D.ORC Macro

The effectiveness of psychosocialinterventions for children’s mental

health disorders has become a topic ofconsiderable debate. A number ofevidence-based treatments have beenshown to be efficacious within researchsettings for specific presenting problems.Less is known, however, about theeffectiveness of these evidence-basedtreatments in complex communitysettings. This has led to a concern thatevidence-based treatments supportedby the results of research are not beingimplemented well in “real world”settings. A clearer understanding ofthe factors that influence delivery ofthese services in the community willassist in understanding how toimprove effectiveness. Thedevelopment of treatment manualsand practice guidelines to guide thedelivery of services in the communityis one approach to improving theimplementation of evidence-basedtreatments. Evaluating these evidence-based, manualized approaches asthey are implemented in thecommunity will provideinformation on real-worldeffectiveness.

A treatment effectivenessstudy is currently underwayas part of the nationalevaluation of theComprehensiveCommunity Mental HealthServices for Children andTheir Families Program.This study examines theeffectiveness of an evidence-based treatment provided to a

child and family outcomes study forthe national evaluation. This involvesinitially identifying sites for the study,documenting procedures for thespecific intervention to be studied,assessing whether the interventionwas implemented as designed, andutilizing a methodology and datacollection strategy that builds upon theframework for the child and familyoutcomes study to follow cases acrosstime. The design allows for the testingof the effects of an evidence-basedintervention integrated into thesystem-of-care approach versussystem-of-care services as usual. Bothgroups participating in the study willcontinue to be eligible for othersystem-of-care services.

The two grantee communitiesinitially selected for participation inthis study are the Bridges Program ineastern Kentucky, and the ClackamasCounty Partnership in Portland,Oregon. Service providers in thecommunity will provide Parent-ChildInteraction Therapy (PCIT) to children

between the ages of 5 and 9 whoare referred for the treatment ofdisruptive behavior disorders.

The study will examine theeffectiveness of this

treatment.PCIT was designed for

young children withdisruptive behaviordisorders. This evidence-based treatment typicallyincludes 8 to 12 weekly

family therapy sessions andinvolves (a) initial assessment,

feedback, and joint developmentof therapy goals by the clinician

Data Matters

Evidence-Based Interventions within theComprehensive Community Mental Health

Services for Children and Their Families Program

HIGHLIGHTING THE FIT WITH SYSTEMS OF CARE: STATE & COMMUNITY LEVEL EFFORTS


selected group of children with specificdiagnoses served within CMHS-fundedsystems of care. The goal of the studyis to examine whether children whoreceive evidence-based treatmentdelivered in systems of care experiencebetter outcomes and maintain thoseoutcomes longer than children in thesame system who do not receive theevidence-based treatment. Within thisstudy, a treatment fidelity substudy isbeing conducted to assess whether theevidence-based treatments areimplemented as intended, and whethersystem-of-care principles were evidentin the care received by these childrenand their families. This will beaccomplished by administering theSystem-of-Care Practice Review, ameasure specifically developed for thenational evaluation that assessesservices experiences at the interfacebetween service providers and families.

The study reflects an integratedprocess that dovetails with the general


Data Matters

and caregiver(s); (b) behavioral playtherapy followed by a caregiverteaching session; (c) direct coaching inthe next several sessions to masterbehavioral play therapy goals; and (d)interactive discipline training forcaregivers followed by severalcoaching sessions. Direct consultationand coaching with the child’sclassroom teacher is also available as acomponent of the treatment. A posttreatment evaluation is thenconducted and changes frompretreatment are reviewed with thefamily to reinforce the improvementsmade. Finally, booster sessions areconducted with families over thesubsequent 12-month period tomaintain positive skills. PCIT has beenfound to be effective for significantlyreducing problems for up to 18months in the home and school, andpositively affecting outcomes foruntreated siblings.

PCIT shares some of the keysystem-of-care principles, most notablya recognition of the importance of thefamily’s role in serving children,sufficient flexibility to attend to theindividual needs of the child andfamily, and integration of assessmentdata to monitor progress and reinforcegains. PCIT also addresses disruptivebehavior disorders, the most prevalentproblem among children served inCMHS-funded systems of care.Additionally, PCIT is geared towardyounger children, allowingintervention at early stages of thedisorders and possibly avertingdeleterious long-term impact ofdisruptive behavior disorders.

Work on this study has alreadyyielded interesting information aboutthe implementation of evidence-basedinterventions within systems of care.The outcomes of this study over thenext several years will assist inunderstanding how the effects ofevidence-based interventions can bemaximized within systems of care. ◆

George P. Gintoli, M.S.South Carolina Department of Mental HealthJohn A. Morris, M.S.W.SC Center for Innovation in Public Mental Health

Context and the ConceptSouth Carolina is and has been an

interesting laboratory for change. As arelatively poor, largely rural state,mental health professionals,consumers and advocates have had tobe especially creative in designing andimplementing any change agenda.While no one would choose toexperience the kinds of budget cutsthat South Carolina—like many of itssister states—has faced and continuesto face, it does focus the attention ofleadership like a laser. There is simplyno excuse for spending a nickel onprograms that don’t have a highlikelihood of success.

In the past eighteen months, oursystem has adopted a new template forchange, published in a planningdocument entitled Making RecoveryReal 1. Adopted by the SC MentalHealth Commission, it lays out anambitious agenda for modifying thestate’s service mix for children andadolescents, their families, and adultswith serious mental illnesses. Itexplicitly drives the public servicedelivery system, one of the fewvertically integrated/state-operatedsystems, on the use of evidence-basedmodels. In instances where there arenot services that meet the gold

standard of evidenced-based practice(randomized, controlled studies in realpractice settings), then promising andemerging practices are highlighted.This quote captures the foundationprinciple of the operational plan:

“The content of this plan wasdeveloped within the context ofthree core themes: the realities ofresource constraints, includingmoney (given the state’s currentbudgetary crisis), humanresources (given the historicdifficulties in recruiting andretraining adequate numbers oftrained staff, especially in ruralcommunities), and time. Theseconstraints demand planning forservices that exemplify twocharacteristics: a high probabilityof success, which means evidence-based or promising/best practices,and consistency with a recoveryphilosophy...”

Some key questions in moving theplan forward are: How ready is thesystem? Are new resources available,or must we redirect existing ones? Howdo we sell best practices and buildconsensus? Who are our partners andwhat new relationships must beformed? How do we measure progress?How do we address sustainability?

Structural ResourcesAgainst a backdrop of decreasing

state mental health revenues, theDepartment of Mental Health isjoining with the state’s academicinstitutions to move the plan forward;building on new resources fromgrants; contracts and collaborative

Evidence-Based Practices:Essential Elements of Reform

Even in Tough Economic TimesS O U T H

C A R O L I N A


1South Carolina Department of Mental Health(2002). Making Recovery Real: A planningdocument for the SC Department of MentalHealth. Morris, JA (Ed.), 65 pp.



Data Matters

relationships; new Medicaid servicelines; reallocation of existing resources;and the essential support of keyadvocate and consumer groups (NAMI,FFCMH, MHA, P& A, SC SHARE)2.

Much of the work ofimplementing best practices, whetherin pilot projects or via statewidereplication, will be facilitated by the SCCenter for Innovation in Public MentalHealth, which is a partnershipbetween SCDMH and the Departmentof Neuropsychiatry and BehavioralSciences at the USC School ofMedicine. In addition, there are activepublic/academic relationships with theFamily Services Research Center at theMedical University of South Carolinain Charleston; the Institute for Familiesin Society at USC; and the Institute forFamily and Neighborhood Life atClemson University. Several projectsare building relationships with SC’straditionally black colleges anduniversities, as well. The final essentiallink for measuring progress is ourpartnership with South Carolina’sunique multi-agency Data Warehouse,which has the capacity to provideunduplicated data on clients across thehealth and human service spectrum.We can track real world outcomes(school performance, juvenile justiceinvolvement, health status) as weimplement best practice interventions.

Making Change HappenEffective linkages between the

practice world and the research worlddon’t happen automatically, so the roleof the Center for Innovation is tofacilitate the public system/academiclinkage process. Conscious, concertedeffort is required to make the bridgework. We use a variety of strategies tomake this happen, but the most criticalelement in our success is a

commitment to listening to what endusers (provider organizations,clinicians, and consumers) tell us theyneed. We help build task specificcoalitions, assist with grantsdevelopment, and bring our resourcesto bear to assist with program, budgetand policy elements of evidence-basedrollouts. We try to consciously use theresources of sister institutes andresearch groups as partners forexternal evaluations, for conceptualsupport and other ways.

The most challenging part of thiseffort has often been the simultaneoustranslation function and mediationbetween researchers and practitioners.Researchers and practitioners oftendon’t understand each other becauseof their different frames of referenceand the needs that drive what they doevery day. The Center for Innovationaccepts as a part of our role preparingeach group to work with the other,creating common ground fordiscussion, and ensuring that there issufficient common purpose tomaintain a practical coalition.

A variety of specific activities ortasks undergird the change process, alldesigned to make the change processless scary and burdensome. Someexamples include:

● Planning for Change: We useddistance education technology (ourclosed circuit system) so that wecould involve lots of local folks, limitexpense, and eliminate travel. Wesupported the work of a stakeholdersteering committee by setting areasonable and meaningful set oftasks (designing a new missionstatement for DMH, identifying corevalues, reviewing draft plans); bysetting a firm set of timelines thatwere honored (so that it didn’t

become the much-dreaded EndlessCommittee); and by committing to keep the plan small and to theextent possible, not written in “bureaucratese.”

● Resource Development: Changeprocesses sometimes need a boost,and resources are chronically scarce. Using the Center forInnovation and its partners, we have been aggressive in seekinggrant support, and are relentless in pushing for sustainability as partof any new initiative. South Carolinais fortunate to have an excellentworking relationship with the state Medicaid authority, and theyhave supported many of our bestpractice efforts.

● Training and Technical Support:Initial training in a best practice isan essential but not a sufficientcondition for system change. As wemove to scale on evidence-basedpractices, whether for adults orchildren and their families, we try to create peer support teams amongthe sites and offer on-going technicalassistance and problem solving. For example, when new proceduresor forms are required, we try to takethe onus of making these changesoff the backs of clinicians, helpingthe system adapt to new ways ofdoing business.

● Tracking Outcomes: There are fewthings that support enthusiasm fornew ways of providing services thanevidence that people’s lives areimproving. For each new practice,we work to build in outcomeindicators that can be used todetermine effectiveness. The Centerfor Innovation has designed theDMH system of “dashboardindicators,” which will beimplemented during 2003.

Essential Elements of Reform continued from page 25

2National Alliance for the Mentally Ill, Federation of Families for Children’s Mental Health, SCMental Health Association, SC Protection and Advocacy for Persons with Disabilities, SC Self HelpAssociation Regarding Emotions



Data Matters

Sustaining ChangeFrom the Department of Mental

Health’s perspective several otherplanning and implementation issuesare directly linked to system supportsthat can put permanency planning upfront and sustain change. Financialviability and continuing to build theskills, knowledge, and ownership ofclinicians and provider organizationsare critical. It is important toanticipate the costs of dissemination;the skills, time, and effort required fora lengthy, system-wide transition; andmanagement strategies at the system,agency, and supervisory levels. Nuts-and-bolts issues such as cost-effectivetraining, retraining and supervision;performance contracting; improveddata systems; attention to degree offidelity to the evidence-based practice;quality of care; and outcomes areimportant to consciously address andbuild adequate infrastructure forsustained change.

Keeping the FocusBudget realities may slow

implementation schedules, but therecontinues to be strong consensusamong the state’s advocacycommunity, the Medicaid authority,and sister agencies (Juvenile Justice,Alcohol and Other Drug Services,Vocational Rehabilitation, StateSheriff ’s Association and others) tosupport DMH leadership’s emphasis onproducing outcomes that matter in thelives of children, adolescents and theirfamilies in South Carolina. Resultsimprove when leaders establish a clearvision with a convincing reason toembrace the vision. The Center forInnovation in Public Mental Healthhas a key role to play, and the moreoften we can find ways to workcollaboratively to make “recoveryreal,” the better. ◆

For more information, contact John Morris,[email protected] orGeorge Gintoli, [email protected].

School-Based Services inthe Context of System of

Care DevelopmentBuilding Bridges Between the Home,School, & CommunityElizabeth V. Freeman, L.I.S.W., M.S.W.,Louise K. Johnson, M.S., and George Gintoli, M.S.South Carolina Department of Mental Health

The System of Care in South CarolinaIn South Carolina, the Department

of Mental Health (SCDMH) and theDivision of Children’s Services havetaken a stance to develop a seamless,state-wide system of services forchildren and families which is family-focused, community-based andculturally competent. SCDMH hasbeen instrumental in developing thevision of coordinated system of caresince as early as 1991. The process ofdeveloping this vision evolved in 1999through the Governor’s Safe SchoolsTask Force targeting evidenced-basedviolence prevention initiatives. Stateand non-profit organizationpartnerships have been strengthenedto focus on the system of care goalsand objectives to: 1) Improve clinicaloutcomes; 2) Cost-share/maximizeresources; 3) Promote culturallyappropriate community-basedinterventions; 4) Promote evidence-based practices through trainingprofessionals/organizations,developing and funding programs thathave proven effective with youth andare outcome driven; and 5)Decentralize crisis/acute care services.

Focusing on What WorksWithin this context, the

Governor’s Safe Schools Task Force

assessed activities and strategiesalready in place in the state proven todecrease and/or prevent youthviolence. A review of evidence-basedprograms and outcome data were usedto determine the additional resourcesneeded in SC to address youthviolence. The results of the task forceproduced the following goals:● Implement more school-based

prevention strategies/programs.● Increase community involvement in

preventing youth violence.● Identify high-risk students for

committing assaultive/violentbehavior and provide effectiveintervention/treatment strategies.

● Improve the system’s overalleffectiveness through increasedcoordination of policy development,training and technical assistance.

Why choose school-based services?South Carolina chose school-based

services as one mechanism for offeringcoordinated and evidence-basedservices within the system of care withseveral goals in mind.● To increase the accessibility of

mental health services for childrenand families in need of these servicesin a non-stigmatizing environment.

● To provide mental health programsthat address early intervention andprevention services for schools andthe community.

S O U T HC A R O L I N A




Data Matters

challenges include: obtainingstakeholder participation andpartnerships, limited resources,overcoming turf issues within schoolsand communities, understandinglimits and duties of each stakeholder,and overcoming mental health stigma.

Both the state level andcommunity level advisory teams playcritical roles in creatively addressingchallenges. First, a shared vision for allpartners at the state and communitylevels has been imperative. Monthlymeetings among advisory teamsprovide a vehicle for encouragement,support, and learning among thepartners to share challenges anddevelop strategies to address thebarriers. True partnerships have beenformed to overcome the historicalautonomy of schools and communityagencies. A concerted effort to obtainstakeholder participation andpartnerships at various levels (e.g.,sharing costs and duties of a program,acceptance of a ‘system of care’perspective, sharing cross-trainingresponsibilities between agencies andprofessionals) takes several years andis ongoing. In a time of budget deficits,both the state and community leveladvisory teams work to createmechanisms to share program coststhrough shared/blended funding streams.The state level advisory team soughtways to change policy and proceduresto share funding betweenagencies/non-profits (e.g., contracts,Memorandum of Agreements (MOA),state health/human servicedepartment policies), whilecommunity level advisory teamssought contracts and MOAs betweenschool districts, community mentalhealth centers, non-profits, citygovernment, foundations, etc.

Despite these challenges, thebenefits to students, families, andschools have been tremendous. In

● To provide consultation for teachersand other school staff on mentalhealth issues.

● To increase partnerships betweenthe school and community whichpromote emotional health.

These goals are part of the reasonwhy school-based services work for SC.Mental health services are providedunder DMH confidentiality guidelinesby mental health professionals (MHP),at the school (a non-stigmatizingenvironment), as requested with noappointment necessary.

School-based services: Fromresearch to practice

SCDMH guides communities/schoolsinterested in implementing school-based programs through severalimportant planning steps necessary fora successful partnership and theselection of an appropriate,community-specific, violenceprevention initiative as outlined below.● Contact the local community mental

health center to set up meetingswith the Director and ChildrenServices Director

● Develop a community advisory teamto assess the community/school’sstrengths and needs

● Outline the anticipated benefits ofmental health efforts for thecommunity/school

● Assess the population to be served,the cost of program services, theschool site, and partnership needs

● Based on needs assessment, selectthe most appropriate preventionprogram (further description below)

● Establish memoranda of agreementand/or contracts between agencies

Through this process,community/school advisory teamshave used resources within theircommunity to begin violenceprevention initiatives. As needed,

partnerships were also created todevelop new resources within thecommunity. After carefullyresearching the needs of theirparticular students and community,each community/school advisory teamchose a model that would best suittheir needs. The SCDMH and stateadvisory team members from theGovernor’s Safe Schools Task Forceprovided information on modelprograms (e.g., FAST, PACT and Youthleadership). The community/schooladvisory team also consideredprograms that had been promoted bytheir local school district. Each teamthen determined how the initiativewould be implemented in the school.Usually, the principal of each school setthe tone for successful implementationof the school-based program.

The following list is a sample ofprograms that were used in variousprojects across the state depending on the intervention that thecommunity selected:● School-wide Bullying Program (USC

Institute for Families in Society),● Get Real About Violence,

Peaceable Schools,● Seals Skills Streaming for elementary

and middle schools,● Positive Adolescent Choices Training,● Families And Schools Together

program,● Peer Mediation,● Prudential Youth Leadership

training,● Youth Courts,● Drug Courts,● Juvenile Arbitration programs, and● Diversion Programs with middle

school youth.

Benefits outweigh the challengesAs with any system change, there

have been challenges encountered atthe state and local level. Some of these

School-Based Services continued from page 27




Data Matters

and support needed by providers todevelop and sustain MST teams. CIP hasfostered steady growth in awarenessand development of MST teams. Thereare currently 9 counties with 8 teamsworking with CIP. Dissemination effortshave been steady and discussions withadditional communities are inprogress. In 2003, CIP will identifyand foster similar partnerships withother EBPs, such as Functional FamilyTherapy and the Oregon Model ofTherapeutic Foster Care.

CIP has been an active participantin inter-state and national discussionsrelated to implementation challengesof EBPs. CIP catalogs on an ongoingbasis its ‘Lessons Learned’ in regards tothe challenges of wide scaledissemination of EBPs. Some of the key lessons learned/challenges include:

● A ‘center’ model provides for an overarching structure that highlights EBP and allows for the developmentof a cohesive and quality basednetwork approach to dissemination.The CCOE model is designed tobundle skills and information into anaccessible resource for variousconstituents. A Center can also actas a ‘hub’ through which a variety ofentities (providers, planners, policymakers) can intersect and beconnected. This type of an organizedand outcome driven approach canhelp highlight and operationalize thedissemination process.

● Evidence of clinical and costeffectiveness is not sufficient toinfluence change in treatment andfunding patterns. Experience isshowing us that it is extremelychallenging to re-route fundingpatterns, despite cost and clinical evidence.

Patrick J. Kanary, M.Ed.Center for Innovative Practices, Ohio

The Center for Innovative Practices(CIP) was established with funding

from the Ohio Department of MentalHealth as a component of its overallCoordinating Centers of Excellence(CCOE) initiative. The CCOE initiative isdesigned to promote the disseminationof evidence based and best practices inthe field of mental health. CIP’s focusis those services and interventionsspecific to youth and familypopulations. The goals of CIP are:● To partner with organizations,

connected to or developers of, evidencebased and promising practices

● To integrate Evidence-Based/Promising Practices with Systems ofCare development by assistingcommunities and organizations withassessment of systems’ needs anduse of evidence-based services

● To identify other evidence-based/promising practices forpotential development in Ohio

● To participate and provide technicalassistance related to policy,financing, and program issues

● To be complementary to/supportiveof other CCOEs and initiatives thatpromote evidence-based practices.

The initial evidence-based practicewith which CIP is partnering forstatewide dissemination isMultisystemic Therapy (MST). CIP hasa partnership agreement with MSTServices, Inc., that provides theinfrastructure for statewidedissemination of MST. As a licensedtraining organization of MST Services,Inc., CIP is qualified to provide all theclinical and administrative consultation

Connecting Systems of Carewith Evidence-Based Practices

From a funding perspective, thecomplexity of public systems’funding and service patternsrequires an individualized approachto realize cost benefits and redirectfunds. Localities and states operatewith a variety of formulas andpatterns; therefore, dissemination ofEBPs requires understanding thosepatterns and working withstakeholders to find the right ‘fit.’ Amore cohesively funded system,serving youth and families at alllevels, would greatly enhance thecapacity of communities to developa broader array of effective services.

From a treatment perspective, weneed to further develop our levels ofcare and our array of services at eachlevel. We owe our youth and familieseffective interventions that arerationally organized in a true system;one that respects their individualityand responds in ways that strengthenthe goal of a healthy, strong family.Behavioral health outcomes achievedthrough best practices andinterventions need to be directly linkedto their impact and effectivenessacross all other critical life domainsof the youth and family. Strongadvocacy among families, partners,providers, and key stakeholders isneeded in order to disseminate the“outcome news” of EBPs.

● Identifying ‘champions’(national/state/local) who canprovide leadership for systemschange is critical. The System ofCare movement, for example, has thebenefit of inspiring champions ofsystems change, inclusive of parents,providers, funders, and policymakers. Advocating for more

O H I O




Data Matters

effective services requires that sameconfiguration of support from thesame base of stakeholders who seethe relationship between the systemof care and evidence-based practicesas highly interdependent. It is in thisway that systems of care and EBPsare complementary and reinforceone another. A good system of care,guided by clear values, should beorganized to support youth andfamilies and built on strong, proven,and effective interventions andpractices. This synergy between theorganizational and operationalcomponents results in a trulyeffective system of care.

As one of its primary goals, CIPsupports integrating evidence-based/promising practices withsystems of care development in itswork with local communities andchampions for systems change. Theselessons learned influence CIP’s role inproviding technical assistance andleadership in the ongoing discussion of how to support the developmentand dissemination of EBP in the fieldvia presentations and participation inkey forums. While CIP is a resource for the state of Ohio, it is an activeparticipant in the national dialogue on EBP through forums, conferences,and other venues focused on thedevelopment and dissemination of EBPs.

Future plans include developingmore system specific information fordecision-makers, launching a website,enhancing the network relationshipwith in-state MST teams, continuingto provide real world examples ofdissemination, and outreach to otherbest, promising, and evidence-basedpractices that will enhance the state’ssystem of care for youth and families. ◆

For additional information on CIP, please contact Patrick J. Kanary, Director, at [email protected] [email protected]

Rick Ferguson, M.S., L.M.H.PMid-Plains Center for BehavioralHealthcare Services

Beth Baxter, M.S.Region III Behavioral Health Services

Wraparound in Central Nebraskabegan in 1995 as a statewide

primary intervention initiative focusedon children and adolescents whoexperienced serious emotionaldisorders and their families, but whowere not Medicaid eligible. Throughour evaluation, a population of youth(juvenile offenders) was identified whowere not experiencing as strong,positive outcomes as other youth inthe program. In order to improve theeffectiveness of wraparound for theseyouth and their families,representatives from our local site—Region III Behavioral Health Services,the Nebraska Department of Healthand Human Services, the University ofNebraska, and a clinical consultantembarked on a process to select anappropriate therapeutic component forthese youth.

Selection CriteriaA good “fit” between the selected

intervention and wraparound inNebraska was important to the team tofacilitate a successful ecologicalapproach. Therefore certainintervention criteria were defined forthe selection process. Some of thecriteria included a family focus,community- and home-basedapproach, team oriented,demonstrated outcomes, and a

compliment to the wraparoundapproach. After a review of evidence-based interventions, Multisystemictherapy (MST) was selected to be thetherapeutic component.

The Process of Implementing EBPin Systems of CareThe process of implementing MST inwraparound involved issues at thesystem partners level, the wraparoundprovider level (Region III), thecontracting agency level (Mid-Plains),and the EBP developer level. Some ofthe issues are briefly highlighted below.

In order for the Nebraska model tobe effective, MST Services, Inc. (MSTS)and Region III Behavioral HealthServices had to work collaboratively toimplement and operationalize themodel. While the implementation ofMST within wraparound involvedmany challenges, the system partnerswere committed to making the modelwork. Conflict resolution and decision-making processes were developed tohelp address the differencesencountered between the proponentsof wraparound and MST.

Initially, there were some strugglesfor MSTS to integrate the interventionwithin the Wraparound Model ofservice delivery, most likely due to aninitial misunderstanding of thewraparound elements. There were alsoconcerns over who was “in charge” ofthe process and difficulty defining rolesand responsibilities of the wraparoundversus MST service providers. Thesystem partners and MSTS worked to overcome these barriers through lots of

Implementation of anEvidence-Based Intervention

in Systems of CareThe Evolution of the Nebraska Model

Ohio continued from page 29

N E B R A S K A



Data Matters

document was created for thewraparound/MST Nebraska Modelthat outlines the intake and referralprocess and the evaluation ofoutcomes, describes the integratedmodels, and defines the roles andresponsibilities of wraparound andMST therapists.

Evolution of MST in The Nebraska Model

The efforts of the initiative tointegrate MST and wraparound arecharacterized by four models, eachevolving to varying degrees throughour collective experience.● The “alternative model” provides

each approach independently from the other, either MST or wraparound. In part, this approach was developedearly in the Initiative due todifficulties conceptualizing how thetwo approaches could work together.While much progress has been madein determining the collaborativeroles of MST and wraparoundtherapists, some children and theirfamilies participate in this model dueto their unique needs.

● The “sequential model” was similarto the alternative model in that thetwo approaches would not beconducted at the same time.However, a family in wraparoundwho may benefit from MST wouldsuspend wraparound services tocomplete MST. After MST concluded,wraparound would resume. Thismodel was the least satisfying forboth the professionals and familiesand was ultimately discontinued.

● The “blended model” focused onproviding wraparound providerswith MST training in an effort toenhance or improve their abilities toprovide wraparound. The MSTecological orientation and use ofempirically supported, behaviorallyoriented treatments helped focus thewraparound process into adisciplined, deliberate approach.

● The “integrated model” focuses onhow MST as a clinical interventionfits into the wraparound process.The collaborative implementation ofMST as part of the wraparoundprocess is based on mutually agreedupon practices and procedures.

In the Nebraska Model, adherenceto the evidence-based intervention iscritical. Therefore, severalimplementation procedures areimportant to note:● A supervision/consultation

process was established with MST for the first three years ofprogram implementation

● Training requirements for alltherapists include an initial 5 daytraining with MST Services in SouthCarolina and ongoing training (2-day booster every quarter)

● Families complete a TherapistAdherence Measure on their therapists

● Weekly tapings of supervision and consultation are reviewed toprovide evidence of adherence to the model and provide opportunityfor skill development

As our therapists work with youthand families, the attitude that we havetoward them is a great determiningfactor to success. We approach each ofour families with a true belief thatthey are people of great value andhave wonderful resources andstrengths that have been untapped. We bring a ray of hope into their lives,seeking to find what is good aboutthem, rather that what is wrong withthem. Through the trial and error ofimplementing an EBP in an existingwraparound process, we found thatthe only effective way was workingtogether for the good of the family. The true reward of providing MST tothese extremely struggling families isbeing able to see their youth remain inthe home. ◆

frank and direct communication duringmeetings with our site’s leadership andMSTS, a clear articulation of our site’sexpectation of MST, and a centeredfocus on positive outcomes for childrenand their families.

Mid-Plains Center alsoencountered challenges in becoming alicensed MST provider. Initially, it wasdifficult to obtain administrativesupport for the funding of a bestpractices program that appears on thesurface to be expensive. While costcomparisons indicated that MST couldsave dollars, convincing the ones whopay the bill was a struggle. Fortunatelyin 1997, Mid-Plains Center receivedthe contract from Region III’s CMHSgrant to develop the capacity for MST.

Another upfront struggle was therecruitment of therapists who werewilling to work 24/7 with highintensive families. Finally, in February1998, two MST Therapy Teams weretrained and began serving familiessoon thereafter. Fortunately, the grantalso paid for the training for eachtherapist. Without this support, thetraining at $2500 per therapist wouldhave been difficult to provide. Seekingto become a self-sustaining programhas been a struggle, as the addedexpense of paying for all fees includinglicensing and consultation can quickly drain the budget. Thenegotiation of contracts with payorsources is also an ongoing event, asthe expenses need to be covered withthe rate. Mid-Plains Center alsoeducates referral sources to generateappropriated referrals to the MSTprogram. It took several months forthe program to have enough referralsand was a big budget concern.

Fortunately, the commitment ofthe systems partners to integrate MSTwith wraparound continued to moveforward despite these barriers. Theevolution of this integration becameknown as “The Nebraska Model.”Ultimately, a Program Guidelines


Data Matters

National TA Center for Children’s Mental HealthGeorgetown University Center for Child and Human Development3307 M Street, NW, Suite 401Washington, DC 20007

2002, over 12,000 children/youthreceived mental health services, with43% of these services provided in theschools. Positive outcomes for studentsinclude: increased school attendance(93%), decrease in discipline referrals(56%), increased length of stay infamily home and communityprograms (92%), decreasedinpatient/hospitalizations (12%), anddecreased juvenile justice referrals(99% remain out-of-trouble). Familieshave easy access to service, toteachers, and to student assistanceteams; crises episodes are handledimmediately; and treatment durations

CORRECTION TO ISSUE #5 LISTOF WEB-BASED RESOURCES

The Council on Quality and Leadershipis a non-profit organization with amission to increase the responsivenessand accountability of individuals, organizations and systems to people with disabilities. This is pursued throughaccreditation, monitoring, evaluation,training, and consultation to humanservice organizations. For over 30 years,the Council has worked to implement person-centered solutions for service andsupport organizations, state and national government agencies, regional systems,and networks, and professionals andself-advocates. The Council's web site is:www.the council.org

have decreased. Schools have a MHcounselor on site to handle crisesepisodes and work daily with studentswho have difficulties.

Currently, over 250 MHPs arelocated in over 448 South Carolinaschools (40%). SCDMH is dedicated tothe development of school-basedprograms and aims to provide a MHPin every school in South Carolina. Theimplementation of school-based servicesoffers the opportunity for evidence-based treatment services within eachcommunity to be accessible tostudents, families, and schools. ◆

School-Based Services continued from page 28


Date post:	13-Apr-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

AN EVALUATION NEWSLETTER SPRING/SUMMER 2003, SPECIAL … · AN EVALUATION NEWSLETTER SPRING/SUMMER...

Documents