An Overview of Performance Dashboards

An Overview ofPerformance Dashboards

Presented by:Desiree A. Crevecoeur-MacPhail, Ph.D.

Research Psychologist, UCLA ISAP

Loretta L. Denering, M.S.Project Director, UCLA ISAP

Why Conduct this Training?

• To review purpose of Los Angeles County Evaluation System (LACES)

• To review the recently implemented Performance Dashboards

• To review the purpose of the Dashboard and how to interpret and utilize the information it contains

2

What do We Hope You Will Gain

• By the end of this training, providers will:– Be more aware of what LACES is and does– Understand some terms used in program

evaluation– Understand how to read and interpret the

Dashboards – Be aware of the benchmarks and how they are

being used to assess performance for outpatient counseling programs

3

EVALUATING LOS ANGELES COUNTY SUBSTANCE USE

DISORDER SERVICESLACES and LACPRS

4

Los Angeles County Evaluation System (LACES)

• Evaluation of adult alcohol and other drug services provided by the Los Angeles County.– Data analysis, reports, articles– Training, presentations and conferences– Development and implementation of surveys & tools

• Partnership between SAPC, contractors for SAPC services, and UCLA/ISAP.

• On-going evaluation, not a temporary study.

Purpose of LACES• Evaluate LA County substance use disorder

treatment services• Assess treatment outcomes and program

performance• Disseminate evaluation data to the public• Evaluate and explore innovative programs• Analyze and report on drug trends• See www.LACES-UCLA.org for more info

http://www.laces-ucla.org/

Los Angeles County Participant Reporting System (LACPRS)

• LACPRS is the key to evaluating substance use treatment among those using County services

• All agencies are contractually required to input specific data into the LACPRS database

• For LACPRS to be effective, agencies MUST input data completely, accurately, and timely for every client!

7

LACPRS Admission & Discharge (A/D)

• LACPRS A/D has 141 questions– Questions 1-32: basic demographic asked

once– Questions 33-141: information asked at

admission, and again at discharge• LACPRS is ONLY for treatment services,

not prevention or DUI (adolescent programs have different set of questions)

• Provides data on those admitted to County funded AOD treatment 8

LACPRS Admission and Discharge

• Data from LACPRS A/D informs the following evaluation documents– Site Reports– Performance Dashboards– Annual Reports

9

ASSESS TREATMENT OUTCOMES AND PROGRAM PERFORMANCE

SITE REPORTS AND DASHBOARDS

10

Important Terms

• Four important terms:–Client/Treatment Outcomes–Program Performance–Engagement–Retention

11

Difference betweenOutcomes and Performance

• Client Outcomes– Client Outcomes are the result of what programs do– Can be measured– Examples: Changes in drug use and employment

• Program Performance– Program Performance refers to areas that are under

the control of the program– Can be measured– Examples: Length of stay and perception of care

12

Examples of Outcome Measures

Domains Measures

Alcohol/Drug Use Alcohol/drug use during past 30 days

Employment/Education

Employment/education in past 30 days

Crime & Criminal Justice

Criminal justice system-related activity in past 30 days, in terms of any CJS involvement arrests, jail days, and prison days

Stability in Housing Stable housing in past 30 days, in terms of homeless

Social Connectedness

Family/social problems in past 30 days in terms of serious family conflict

Examples of Potential Performance Measures

Domains MeasuresContinuity of Care • Clients who had a subsequent admission to

another service during 30 days after discharge from a prior service (treatment episode data with unique client ID).

Access • Self-reported wait list time at admission.Engagement • Stay in treatment at least 30 days and

participate in 4 or more sessions.Retention • Length of treatment stay (in days)

Completion • Clients with a treatment completion (referred/not referred) discharge status.

Site Reports

Details of Content and Use

15

What is a Site Report?

• Brief: six pages, double-sided.• Information is gathered from the admission

& discharge LACPRS.• Information is included on each agency site

and for all sites of the same program type (e.g. outpatient, residential, etc).

More on Site Reports

• Snapshot of program functioning and short-term treatment outcomes.

• Based on LACPRS admission & discharge data.

• Are used to provide information ONLY • Divided into two sections:

– Executive Summary– Full Report

17

Purpose of Site Report

• To provide information concerning key performance/outcome areas.

• To provide feedback to sites concerning how they are performing with regards to the focus areas of the evaluation.

• To report how other, similar programs are performing.

18

Contents of the Site Report

• Executive Summary– Admission and discharge totals.– Information on program functioning and

treatment outcomes.• Full Report

– Includes demographic information.– Includes more details on treatment outcomes.– Charts showing admission to discharge

changes in various areas covered by the LACPRS.

Site Reports and LACPRS

20

• Site Report Information is based on LACPRS– Late data or inaccurate data input at

admission and discharge will affect results• Engagement of clients is critical• Conducting Exit Interviews are essential

– Compares measures/indicators input at admission with those at discharge

– Lack of exit interviews impact accuracy of performance measures and program outcomes

Performance DashboardsWhat are they?

How were they developed?How are they used?

How do you gain access?

21

What is a Dashboard?

• Similar to a report card• An easy to read summary (typically single

page)• Provides information on provider

performance based on the identified measures

• Dashboard is based solely on LACPRS information input by the agency.

22

Difference betweenSite Reports and Dashboards

• Site Reports– Multi-page– Info on outcomes and some performance

measures– No benchmarks or required level of achievement

• Dashboards– Single page – in some cases– Info only on performance measures– Includes benchmarks and there is aRequired level of achievement

23

Similarities between Site Reports and Dashboards

• Accurate and timely data entry are important.– Inaccurate data will result in

• Incorrect reports (Site Reports/Dashboards)• Delayed payments

– Late data will result in • Incorrect reports

• Both reports only available to the executive directors and their designees

• If you need access, the executive director must contact Richard Lugo 24

Terminology Review

25

• Performance Measure: An indicator used to assess a provider's delivery of care as it conforms to guidelines or standards of quality. – Focus on program functioning – Performance measures do not directly

measure these outcomes.

More Terminology Review

• Performance Benchmark : A level of achievement in reaching the goal for a performance measure that generally represents an industry-best standard.– For SAPC that industry-based standard is the

average performance for all providers– Adherence to performance benchmarks is

expected to lead to desirable outcomes

26

Performance Measure (PM) Development

• A list of possible PMs were distributed to contracted treatment providers

• Meetings were held to discuss potential PMs, availability of the data and their utility

• Once performance measures were settled, data examined to determine benchmark

• Settled on three initial performance measures - for outpatient counseling (OC) ONLY

• Day Care Habilitative included with OC

27

Performance Measure (PM) Development

• Performance Measures agreed upon:– 30 Day Length of Stay (Engagement)– 90 day length of stay (Retention)– Exit Interviews

• All of these performance measures were familiar to provides since they are included in site reports

28

Why Engagement?

• In AOD treatment, a significant proportion of patients leave treatment during the first four weeks

• Engagement = first 30 days in treatment– Typically includes at least 4 contacts

• Low engagement may indicate problems with intake process, counselor rapport development, program process or policies

29

Why 30 Days?

• Most of the questions on the LACPRS that we use to measure outcomes have a 30 day time frame. For example– How many days in the last 30 did you…– This time frame is used at admission and

discharge• If clients are not engaged for at least 30

days, the time frame of the questions asked at discharge overlap the time period covered at admissions

30

Why Retention?

• Research shows that without an adequate amount of time in treatment, few improvements are found– 90 days is the magic number

• Retention = 90 days or more in treatment• Low retention rates may indicate problems

with treatment process (e.g., redundant), lack of rapport, inappropriate or ineffective policies for dealing with relapse, etc.

31

Interviews

• Responses to all discharge LACPRS questions.

• One of the most important of the PMs• Information from both admission and

discharge is required to measure treatment effectiveness

• No exit interviews = No measurement of changes that occurred during treatment

Impact of Exit Interviews and 30- days LOS Data

• Program ABC has 20 individuals discharged– 15 completed exit interviews– 10 of the 15 were in treatment at least 30 days

• ONLY have valid outcome data for 10 of the 20 discharged individuals– Only those in treatment at least 30 days and

who have completed exit interviews will have data that can show changes that occurred during treatment

33

Performance Benchmarks• Once PM were agreed upon, benchmarks

were developed• Standard levels of expected performance• SAPC benchmarks are based on average

from outpatient programs for the prior three years– 30 Day Length of Stay = 80%– 90 Day Retention = 65%– Exit Interviews = 50%

• Benchmarks are reported via “Dashboards”

34

HOW TO READ DASHBOARDS

• Three (or four) sections of dashboard– Introduction– Results– Next Steps (for fell below benchmark)– Definitions

35

HOW TO READ DASHBOARDIntroduction

Performance Measure

Current Quarter Discharges: 7

Report QTR

N

Total for Fiscal Year by Quarter (%)

CumulativePerformance (%) Performance

Benchmarks (%)1st 2nd 3rd 4th N Year 1 Year 2 Year 3

Participants in Treatment at least 30

Days 7 100 100 10 100 80


Days 5 100 71 8 80 65

Participants with Exit Interviews 6 67 86 8 80 50

36

Performance DashboardOutpatient Counseling Program

1234 Any Street, Los Angeles, CA 90025

This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.


Performance Measure


Report QTR

N



Benchmarks (%)1st 2nd 3rd 4th N Year

1Year

2Year

3

Participants in Treatment at least 30 Days 7 100 100 10 100 80



37





• Two main areas to note on all dashboards–The underlined comment above the

table–The table at the top of the dashboard

38


Performance Measure


Report QTR

N




1Year

2Year

3




39



This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short.According to this information, nothing further is required of your program at this time.


Performance Measure


Report QTR

N



Benchmarks (%)

1st 2nd 3rd 4th N Year 1

Year 2

Year 3


Days

7 100 100 10 100 80


Days

5 100 71 8 80 65

Participants with Exit Interviews

6 67 86 8 80 5040





• Performance table divided into 5 columns that provide information on discharged clients entered into LACPRS.

• These columns include:– Performance measure– Report Quarter– Total for Fiscal Year by Quarter (%)– Cumulative Performance (%)– Performance Benchmark (%)

41


Performance Measure


Report QTR

N



Benchmarks (%)


Year 2

Year 3


Days

7 100

100

10 100

80


Days

5 100

71 8 80 65


6 67 86 8 80 50 42





Performance Measure


Report QTR

N



Benchmarks (%)


Year 2

Year 3


Days

7 100

100

10 100

80


Days

5 100

71 8 80 65


6 67 86 8 80 50 43



This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly bass and to contact the County if the performance of this program falls short.According to this information, nothing further is required of your program at this time.


Table provides info on• Performance measures and benchmarks

– Benchmarks based on County average for outpatient counseling programs

• Current Quarter Discharges– Number of discharges for the reporting

quarter• Report Quarter (QTR) N

– Number of clients who met the performance measure 44


Performance Measure


Report QTR

N




1Year

2Year

3


Days 7 100 100 10 100 80


Days 5 100 71 8 80 65


45





• Total for Fiscal Year by Quarter– The number in these boxes are

PERCENTAGES– They tell us what percentage of discharged

clients met the performance measure.• In this case, 100% of discharged clients remained in

treatment at least 30 days.– Agencies would compare the percent for each

quarter with the performance benchmark at the end of the table.

46


Performance Measure


Report QTR

N




1Year

2Year

3



Participants with Exit Interviews 6 67 86 8 80 50 47





Performance Measure


Report QTR

N


CumulativePerformance (%)

Performance Benchmarks

(%)1st 2nd 3rd 4th N Year

1Year

2Year

3


Days

7 100 100 10 100 80


Days

5 100 71 8 80 65


6 67 86 8 80 50 48




HOW TO READ DASHBOARDResults

• The narrative description of the data is below the performance table under the section “Results”– There are two periods in which results are

listed:• Current Quarter• Cumulative Results

– There are two types of results:• Met/exceeded• Not met

49


RESULTSCurrent Quarter• This program has met/exceeded the County Benchmarks for 30

Day LOS. • This program has met/exceeded the County Benchmarks for 90

Day LOS. • This program has met/exceeded the County Benchmark for Exit

Interviews. Cumulative Results (Year to Date)• This program has met/exceeded the County Benchmarks for 30



Interviews. 50








Interviews. 51








Interviews. 52

HOW TO READ DASHBOARD“Fell Below: 1-19%”

Performance Measure


Report QTR

N




1Year

2Year

3




53

The performance of this program requires improvement of 1-19% on one or more of three performance measures.


• All sections are the same for all reports – only the data presented is different

– Look to “Results” and to learn what performance measure needs improvement.

Current Quarter• This program has not met the County Benchmarks for 30 Day LOS. • This program has not met the County Benchmarks for 90 Day LOS. • This program has met/exceeded the County Benchmark for Exit Interviews.

Cumulative Results (Year to Date)• This program has not met the County Benchmarks for 30 Day LOS. • This program has not met the County Benchmarks for 90 Day LOS. • This program has met/exceeded the County Benchmark for Exit Interviews.

54


• Also, look to Next Steps:

• Provides tips on how to improve performance in each of the performance areas

• Tips are straightforward and should not require additional assistance from SAPC

55

• 30-Day Length of Stay (LOS): Check out the NIATx website at www.niatx.net to learn of ways to improve your programs’ 30 day LOS.

• 90-Day Length of Stay (LOS): Program participants who are not in treatment at least 90 days may not fully benefit from treatment. The patient does not have to be in this program for the full 90 days if he or she transferred from a briefer treatment stay elsewhere. In order to ensure the LOS is calculated correctly, be sure that the client ID is identical to what was used with the patient in the prior program. If you are transferring the patient to another level of care, be sure to follow-up with the program to determine if the patient enrolled. If your program does not span for 90 days, please notify your program auditor.

56

HOW TO READ DASHBOARDNext Steps

http://www.niatx.net/

HOW TO READ DASHBOARDNext Steps

• Exit Interviews (Administrative Performance Measure): Exit interviews (or completed LACPRS discharges) are important in order to adequately measure how the patient improved over the course of treatment. If you are having problems with patients who leave treatment prior to the scheduled interview, try one of these strategies:– Inform the patient at admission that an exit interview is required prior to the

patient discharging from the program.– As the date of discharge nears, remind the patient that there is an exit interview

that needs to be completed prior to discharge – regardless of the patient’s discharge status.

– Have counselors complete the Concurrent Recovery form – this information can then be used if the patient leaves treatment prior to the scheduled discharge. If the patient is present, complete the discharge as usual – do not use the form, even if completed as it does not collect all of the required discharge information and should only be used if necessary.

• If this program requires additional assistance, please contact your program auditor. 57

HOW TO READ DASHBOARD“Fell Below 20% or more”

• This dashboard will be posted when any single measure is 20% or more below the benchmark

• All sections are the same for all reports – only the data presented is different– Look to “Results” and to learn what

performance measure needs improvement

58


Performance Measure


Report QTR

N




1Year

2Year

3




59

The performance of this program requires improvement of 20% more on one or more of three performance measures.


• Also look to “Expected Performance…” table– This process improvement project is designed to

assist with 30 Day engagement– For other performance measures look to “Next

Steps” for tips to improve

60

61

Performance Measure

Current Performance

(%)

ExpectedPerformance FQ

3 (%)

ExpectedPerformance

FQ 4 (%)

Participants in Treatment at least

30 Days 54 64 74

Participants in Treatment at least

90 Days --- --- ---

Participants with Exit Interviews 23 33 43

Also look to “Expected Performance” table


HOW TO READ DASHBOARDDefinitions

The dashboard also includes definitions of the terms used in the report•LOS: Length of Stay.

•Participants in Treatment at Least 30 Days: Are those individuals who are in treatment at least 30 days, as measured by the LACPRS admission date and discharge date (last face to face) and had four treatment sessions during that time. The treatment sessions can include the individual counseling sessions to complete the assessment and treatment plan as well as any form of group counseling.

62

HOW TO READ DASHBOARDDefinitions

• Participants in Treatment at Least 90 Days: Are those individuals who were in treatment at 30 days and remained in treatment for 90 days or more, as measured by the LACPRS admission date and discharge date (last face to face).

• Participants with Exit Interviews: This performance measure is more of an administrative performance measure in that it assesses the ability of the program to collect the information necessary to produce patient outcomes. This measure is collected based on the response to the LACPRS discharge question, “Is the client available for an exit interview?” This performance measure is only counted for those clients who remained in treatment at least 30 days or more.

63

HOW TO READ DASHBOARD

• In looking at the dashboard:– Compare total fiscal year by quarter with the

performance benchmark for each measure– Compare Cumulative Performance with the

performance benchmark– Read the results section for further clarification– Then, if necessary, read the next steps and

expected performance • ONLY NECESSARY WHEN ONE OR MORE

PERFORMANCE BENCHMARKS IS NOT MET64

Additional Dashboard Info• For the most accurate dashboards

– A/D data entered weekly– All data entered by the last day of the month

• Dashboards posting in same area on the LACPRS system as the site reports– Posted on the 10th of the month subsequent to

the end of each quarter– e.g. 3rd quarter dashboards will be posted by April 10th

(end of quarter is March 31st) – ONLY Executive Directors and their designee(s)

have access

65

Final Note on Dashboards

• Located in LACPRS, like the site reports• Based on LACPRS data

– Late data or inaccurate data will affect results– Advise executive directors to review reports

• If you want more info on LACES, outcome vs. performance measures, site reports, etc. go to www.laces-ucla.org.

66

http://www.laces-ucla.org/

Questions?

67

Contact Your Presenters

Desiree A. Crevecoeur-MacPhail, Ph.D.(310) 267-5207

email: [email protected]

Loretta L. Denering, M.S.(310) 267-5312

email: [email protected]

mailto:[email protected]

mailto:[email protected]

Date post:	23-Feb-2016
Category:	Documents
Upload:	akiva
View:	44 times
Download:	0 times

An Overview of Performance Dashboards

Documents