FY 2019 Quality Management Annual Report · 2020-01-27 · Quality Management Annual Report FY 2019...

January 10, 2020

FY 2019

Quality Management

Annual Report

Quality Management Annual Report FY 2019

1 | P a g e

Table of Contents

Table Listing ...............................................................................................................................................4

Figure Listing .............................................................................................................................................5

Section 1: Executive Summary ....................................................................................................................7

Reviews Completed by Fiscal Year ................................................................................................................ 7

FY 2019 Provider Performance ..................................................................................................................... 8

FY 2019 Quality Management Activities ...................................................................................................... 9

FY 2019 Key Findings .................................................................................................................................. 13

FY 2020 Areas of Focus ............................................................................................................................... 18

In Conclusion ............................................................................................................................................... 21

Section 2: Introduction ............................................................................................................................. 23

Section 3: Behavioral Health Quality Reviews ........................................................................................... 25

Background ...................................................................................................................................................... 25

Sampling Method ............................................................................................................................................. 26

Individual Records and Billing Review ........................................................................................................ 26

Individual and Staff Interviews ................................................................................................................... 29

BHQR Overall Review Scores ............................................................................................................................ 30

BHQR Overall Scores ................................................................................................................................... 30

BHQR Overall Scores by Tier and Provider Size ........................................................................................... 32

BHQR Billing Validation .............................................................................................................................. 35

BHQR Assessment and Planning ................................................................................................................. 44

BHQR Service Guidelines............................................................................................................................. 47

Office of Deaf Services Special Reviews ...................................................................................................... 55

BHQR Programmatic (Non-scored) ............................................................................................................. 57

BHQR Focused Outcome Areas (FOA) ......................................................................................................... 59

BHQR Staff and Individual Interviews ......................................................................................................... 64

Assertive Community Treatment (ACT) Quality Reviews ................................................................................. 65

ACT Quality Review Overall Scores ............................................................................................................. 66

ACT Billing Validation ................................................................................................................................. 68

ACT Assessment and Planning .................................................................................................................... 71

ACT Service Guidelines ................................................................................................................................ 72

ACT Focused Outcome Areas ...................................................................................................................... 75

Crisis Stabilization Unit Quality Reviews .......................................................................................................... 76

Crisis Stabilization Unit Sampling Method ................................................................................................. 77

Crisis Stabilization Unit Quality Review Overall Scores .............................................................................. 79


2 | P a g e

CSUQR Individual Record Review (IRR) ....................................................................................................... 80

CSUQR Service Guidelines ........................................................................................................................... 87

CSU Focused Outcome Area ....................................................................................................................... 90

CSUQR Staff Interview and Individual Interview ......................................................................................... 94

New Provider Reviews ...................................................................................................................................... 96

Reassessment Frequency Reviews ................................................................................................................. 101

BHQR Reassessment Review Findings ...................................................................................................... 101

CSU Reassessment Review Findings ......................................................................................................... 107

Technical Assistance/Exit Conference ............................................................................................................ 108

Summary of Findings and Recommendations for Behavioral Health Providers ............................................ 109

Provider Performance ............................................................................................................................... 110

FY 2019 BH Accomplishments .................................................................................................................. 113

BH Systems Strengths and Recommendations for Improvement ............................................................. 115

Section 4: Intellectual and Developmental Disabilities ............................................................................ 121

Background .................................................................................................................................................... 121

Sampling Method ........................................................................................................................................... 122

Person Centered Review (PCR) .................................................................................................................. 122

Quality Enhancement Provider Review (QEPR) ........................................................................................ 123

Review Processes ........................................................................................................................................... 124

PCR and QEPR ........................................................................................................................................... 124

Quality Technical Assistance Consultation (QTAC) .................................................................................... 126

Person Centered Review ................................................................................................................................ 127

PCR Scores by Tool .................................................................................................................................... 127

PCR Scores by Focused Outcome Area (FOA) ............................................................................................ 128

Low-Scoring Indicators by FOA (PCR) and Opportunities for Improvement ............................................. 129

PCR Scores by Tool and Focused Outcome Area (FOA) ............................................................................. 143

Low-Scoring FOAs by Perspective (Tool) ................................................................................................... 144

FOAs by Demographics ............................................................................................................................. 146

Individual Service Plan Quality Assurance Checklist (ISP QA) ................................................................... 149

ISP Expectations ........................................................................................................................................ 152

Results Using Original ISP QA Checklist .................................................................................................... 152

Results Using New ISP QA Checklist .......................................................................................................... 154

PCR Results by Service .............................................................................................................................. 155

PCR Strengths and Recommendations ..................................................................................................... 157

Quality Enhancement Provider Review (QEPR) ............................................................................................. 158

QEPR Scores by Size .................................................................................................................................. 158

Overall QEPR Score by Tool and Year ........................................................................................................ 159

Overall Crisis Provider Scores.................................................................................................................... 161


3 | P a g e

Qualifications and Training (Q & T) and Service Specific (DDSS) .............................................................. 161

QEPR Scores by Provider Size .................................................................................................................... 163

QEPR Scores by Focused Outcome Area (FOA) ......................................................................................... 164

QEPR Scores by FOA and Tool ................................................................................................................... 165

QEPR Scores by FOA and Provider Size ..................................................................................................... 167

Quality and Technical Assistance Consultation (QTAC) .................................................................................. 168

Intellectual and Developmental Disability Summary of Findings and Recommendations ............................ 170

FY 2019 IDD Accomplishments ................................................................................................................. 170

IDD System Strengths and Recommendations ......................................................................................... 173

Section 5: Feedback Survey Results ........................................................................................................ 183

Section 6: Conclusion ............................................................................................................................. 185

Appendix A: Abbreviations and Acronyms .............................................................................................. 189

Appendix B: Score Distributions ............................................................................................................. 191


4 | P a g e

Table Listing (Links available by ctrl + clicking on table name)

Reviews Completed by Fiscal Year

Table 1. BHQR Overall Scores by Tier

Table 2. BHQR Category Scores by Provider Size

Table 3. FY 2019 Amount Reviewed by Funding Source

Table 4. Non-Intensive Outpatient Services and Percent of Funds Justified

Table 5. Specialty Services and Percent of Funds Justified

Table 6. BHQR Billing Discrepancy Reasons by Fiscal Year

Table 7. BHQR Assessment & Planning Indicator Scores by Fiscal Year

Table 8. BHQR Service Guidelines Scores by Service Type

Table 9. FY 2019 Lowest Scoring Indicator(s) by Service

Table 10. Progress Note Indicator Results by Year and Service Type

Table 11. Office of Deaf Service Specialty Review Results

Table 12. BHQR and ACT Results by Category by Year

Table 13. ACT BHQR Billing Discrepancy Reasons by Year

Table 14. ACT Service Guidelines Scores

Table 15. Lowest Scoring Individual Record Review Indicators by Category

Table 16. Highest-Scoring Individual Record Review Indicators by Category

Table 17. FY 2019 Review Results by Category of New Providers

Table 18. Intensive Family Intervention Results by Indicator for New Providers

Table 19. FY 2019 Review Results by Category of Reassessed Providers

Table 20. FY 2019 Technical Assistance/Exit Conference Details

Table 21. BHQR, ACT, CSUQR Overall Averages by Year

Table 22. BHQR and CSUQR FOA Scores

Table 23. FY 2019 Reassessment Review Results …

Table 24. FY 2019 PCR Sample by Region

Table 25. QEPR Provider Sample by Size and by Year

Table 26. FY 2019 Number of Records by Review Tool and Review Type

Table 27. Low-Scoring Whole Health Indicators (PCR)

Table 28. Low-Scoring Safety Indicators (PCR)

Table 29. Low-Scoring Person Centered Practices Indicators (PCR)

Table 30. Low-Scoring Community Life Interview Indicators (PCR)

Table 31. Low-Scoring Community Life Record Review Indicators (PCR)

Table 32. Low-Scoring Choice Indicators (PCR)

Table 33. Low-Scoring Rights Indicators (PCR)


5 | P a g e

Table 34. ISP QA Life Indicator by Region and Year

Table 35. FY 2019 ISP QA Checklist Ratings by Expectation

Table 36. Average Rating by Expectation (0 - 4) by Year

Table 37. FY 2019 “New” ISP QA Checklist Ratings

Table 38. PCR PRR Results by Service and Year

Table 39. FY 2019 PCR Record Review Results by FOA and Service

Table 40. FY 2019 Top Strengths Identified During a PCR

Table 41. FY 2019 Top Recommendations Identified During a PCR

Table 42. FY 2019 FY 2019 Qualifications and Training Low-Scoring Indicators

Table 43. QTACs by Referral Source and Type

Table 44. QTAC Referral Reasons: FY 2018

Table 45. FY 2017 Summary by Tool and Review Type

Table 46. The Collaborative Provider Feedback Surveys

Figure Listing (Links available by ctrl + clicking on figure name)

FY 2019 Provider Performance

Figure 1. FY 2019 Diagnostic Categories Reviewed

Figure 2. BHQR Fiscal Year Results by Category

Figure 3. Overall Score by Provider Size by Year

Figure 4. Billing Score by Provider Size by Year

Figure 5. BHQR Billing Validation Amount Reviewed by Year Figure 6. BHQR Amount Reviewed by Funding Source and Year

Figure 7. FY 2019 Percent of Discrepancy Reasons

Figure 8. FY 2019 Percent of Dollars Reviewed by Non-Intensive Outpatient Service

Figure 9. FY 2019 Percent of Dollars Reviewed by Specialty Service Figure 10. BHQR Focused Outcome Areas Scores by Year

Figure 11. BHQR Focused Outcome Area by Year – Whole Health

Figure 12. BHQR Focused Outcome Area by Year – Safety

Figure 13. BHQR Focused Outcome Area by Year – Person Centered Practices

Figure 14. BHQR Focused Outcome Area by Year – Community Life

Figure 15. BHQR Focused Outcome Area by Year - Choice

Figure 16. BHQR Focused Outcome Area by Year - Rights

Figure 17. BHQR Interview Data by Year Figure 18. ACT Fiscal Year Results by Category and Year

Figure 19. ACT Billing Validation by Year

Figure 20. ACT Assessment and Treatment Planning Scores by Year


6 | P a g e

Figure 21. ACT FOA Scores by Category by Year

Figure 22. FY 2019 Diagnostic Categories Observed by Review Type

Figure 23. CSU Overall Scores by Year

Figure 24. Individual Record Review Category Scores

Figure 25. CSUQR Service Guidelines Staffing Adherence Indicators by Year

Figure 26. CSUQR Service Guidelines Policy Adherence Indicators by Year

Figure 27. CSUQR FOA Results by Year

Figure 28. CSUQR Individual and Staff Interview Results by Year

Figure 29. FY 2019 Overall and Billing Scores by New Provider

Figure 30. FY 2019 BHQR Overall Score Distribution of Reassessed Providers

Figure 31. FY 2019 BHQR Billing Score Distribution of Reassessed Providers

Figure 32. Overall Score Results by Reassessed CSU

Figure 33. PCR Scores by Tool and Year

Figure 34. PCR Scores by FOA and Year

Figure 35. FY 2019 PCR Scores by Tool and FOA

Figure 36. FY 2019 PCR Scores by FOA and Region

Figure 37. FY 2019 PCR Scores by FOA and Residential Setting

Figure 38. FY 2019 PCR Scores by FOA and Funding Source

Figure 39. PCR ISP QA Life Indicator by Year

Figure 40. FY 2016 - FY 2019 QEPR Samples by Provider Size and Type

Figure 41. Overall QEPR Scores by Tool and Year

Figure 42. Overall Crisis Provider Scores by Tool and Year

Figure 43. QEPR Qualifications and Training and DDSS Results by Year

Figure 44. Overall and Qualifications and Training Score by Provider Size and Review Year

Figure 45. Overall QEPR Scores by FOA and Year

Figure 46. FY 2019 QEPR Scores by Tool and Focus Outcome Area

Figure 47. FY 2019 PRR Low-Scoring Indicators - QEPR

Figure 48. FY 2019 QEPR Provider Scores by FOA and Size

Figure 49. FY 2019 QTAC Technical Assistance Provided


7 | P a g e

Section 1: Executive Summary

Since FY 2016, the Georgia Collaborative Administrative Services Organization (the Collaborative) has

assisted the Georgia Department of Behavioral Health & Developmental Disabilities (DBHDD) in its

management of services and supports for individuals receiving Community Behavioral Health and

Rehabilitation Services (CBHRS), New Options Waiver (NOW), Comprehensive Supports Waiver

(COMP), and state funded behavioral health (BH) and intellectual and developmental disabilities

(IDD) services. DBHDD delegates BH, Crisis Stabilization Unit (CSU), and IDD quality reviews to the

Collaborative. June 30, 2019 marked the Collaborative’s Quality Management Department’s fourth

year of the contract with DBHDD. See the table below for the number of BH and IDD quality reviews

and interviews conducted.

Reviews Completed by Fiscal Year

Fiscal Year Number of Reviews Individuals Interviewed Staff Interviewed

BH/CSU* IDD** BH/CSU IDD BH/CSU IDD

2016 177 584 643 1,151 657 1,186

2017 208 581 735 921 774 702

2018 197 584 616 920 652 759

2019 201 584 780 1,228 832 888

Total 783 2,333 2,774 4,220 2,915 3,535

*The additional Assertive Community Treatment (ACT) Quality Reviews are included in the BH/CSU totals (FY17: 20;

FY18: 21; FY19: 20). Calculations specific to the ACT Quality Reviews are in a separate section of the report.

** Crisis providers are included in the total number of reviews, but are not included in the figures/graphs below

(FY16: 1; FY17: 4; FY18: 1). Separate calculations of Quality Enhancement Provider Reviews (QEPR) and Person

Centered Reviews (PCR) are included below. A separate QEPR calculation for crisis providers can be found

throughout the report.


8 | P a g e

FY 2019 Provider Performance

Behavioral Health Quality Reviews Crisis Stabilization Unit Quality Reviews

IDD: Quality Enhancement Provider Reviews1

1 The QEPR sample is designed to ensure all providers are reviewed at least once every three to four years.

Overall Score

FY 17: 83.7% FY18: 84.8%FY19: 86.2%

Individual Interview

FY17: 90.7%FY18: 89.6%FY19: 92.3%

Staff Interview

FY17: 95.2%FY18: 95.3%FY19: 95.7%

Observation

FY17: 95.7%FY18: 96.0%FY19: 97.2%

Record Review

FY17: 69.6%FY18: 72.4%FY19: 73.2%

Qualifications and Training

FY17: 82.6% FY18: 84.6% FY19: 83.2%

DD Service Specific

FY17: 99.8% FY18: 99.8% FY19: 99.8%


9 | P a g e

IDD: Person Centered Reviews by Focused Outcome Area and Year

FY 2019 Quality Management Activities

Based on Fiscal Year (FY) 2018 analysis and dialogue with DBHDD, the following areas of focus were

determined for FY 2019:

Behavioral Health

Based on FY 2018 findings statewide, the minimum score threshold for the Behavioral Health

Quality Reviews (BHQR) and Crisis Stabilization Unit Quality Reviews (CSUQR) was increased

to 90 percent for both Billing (not applicable to CSUQR) and Overall scores. Any provider not

reaching the 90 percent threshold is scheduled for a second review (reassessment). Thus,

behavioral health Quality Assessors continue to assist the network in reaching or surpassing

the threshold through technical assistance, education, and various training methods and

materials.

Rights ChoiceCommunity

Life

PersonCenteredPractices

SafetyWhole Health

FY 2017 (N = 481) 90.2% 78.7% 71.7% 82.6% 90.3% 84.7%

FY 2018 (N = 484) 92.0% 82.3% 76.9% 84.4% 91.6% 87.4%

FY 2019 (N = 484) 93.4% 80.9% 77.3% 85.2% 92.5% 86.3%

0%

25%

50%

75%

100%


10 | P a g e

Behavioral Health Quality Review (BHQR) and Crisis Stabilization Unit Quality Review

(CSUQR) annual tool revisions were completed throughout FY 2019 with several additions for

clarity, alignment, and formalized criteria including:

o Addition of Substance Abuse Intensive Outpatient Program as a measurable service

o Inclusion of a non-scored category titled Programmatic, which measures various

program needs based on service type

o Differentiation between Peer Support Whole Health and Wellness - Individual versus

Group services

o Inclusion of a billing discrepancy entitled “No valid, verified diagnosis on date of

service provided”

o Addition of question/indicator detail to include any reason(s) an indicator(s) was not

met

The Collaborative met contractual requirements and conducted the FY 2019 Quality Training

Series calendar events related to BH and CSUs. The topics presented were preapproved by

DBHDD.

Successful collaboration occurred with the Collaborative’s Quality department in developing

Final and Exit summaries and assessments specific to the IDD review process for public

website viewing. This development occurred simultaneously with the alignment of IDD tool

revision changes and needs with categories reviewed within the BHQRs/CSUQRs.

Completion of the DBHDD requested Office of Deaf Services Specialty Review.

An additional forty percent of reassessments were completed in FY 2019 (N = 28) as

compared to FY 2018 (N = 20), due to the increased threshold for Overall and Billing scores to

90 percent from 85 percent with supplemental technical assistance provided by Assessors.

Quality Management met all quality review reporting contractual requirements as set forth

by DBHDD including but not limited to; quarterly provider score reporting, semi-annual

review data and score presentation, Medicaid discrepancy reporting, as well as any ad hoc

requests made by DBHDD.


11 | P a g e

Quality Management conducted various ad hoc reporting and data analyses which were

provided to DBHDD and included the following:

o Sample size Power Analysis (BHQR/CSUQR/ACT/ICM)

o Supported Employment Study (IDD Specific but included BH collaboration)

o Intensive Case Management (ICM) Analysis

o Individual/Staff Interview Analysis

o Yale RSA-R Report

o Intensive Family Intervention (IFI) Review/Claim Analysis

o Office of Deaf Services Review pilot

o Suicide Study proposal and sampling methodology

Intellectual and Developmental Disabilities

The Collaborative’s continuous improvement efforts for the IDD system focused on three key areas:

training for stakeholders, the new IDD Connects system, and review process changes including tools,

procedures and web-based application. Following are the initiatives for each of these areas.

Early in FY 2019, the Collaborative’s Quality Management team and DBHDD met to discuss

review findings from FY 2018 to identify training needs for the stakeholder community. As a

result, the following training sessions were developed and provided to stakeholders in the

2019 fiscal year:

o A total of 11 training sessions were developed centered around the Focused Outcome

Areas (FOA) and presented throughout the year:

Whole Health: How to Identify Potential Health Risks; Monitoring Health Risks

and Protocols

Person Centered: CMS Expectations for the Support Planning Process;

Creating a Person-Centered Culture in Any Setting

Community: Networking with the Community to Build Relationships and

Natural Supports; Community Life: How to Support Community Relationships


12 | P a g e

and Natural Supports; Supporting Individuals to Gain Competitive

Employment

Rights: How to Identify Rights Violations; How to Support Self Advocacy

Choice: What is Informed Choice?

Safety: Educating Individuals on Abuse, Neglect & Exploitation

The Human Services Research Institute (HRSI) developed and presented an

additional training focused on National Core Indicator (NCI) survey results and

the development of quality improvement initiatives.

Qlarant, a subcontracting partner within the Collaborative, created an additional 16 separate

training modules, with PowerPoint presentations and User Guides to assist with

implementation of the new IDD Connects system. Each session included information for the

different user roles developed to access the system. As a part of the training team, Qlarant

participated in numerous meetings and collaborative sessions with Beacon, its Information

Technology sub-contractor and DBHDD. Qlarant also helped train Beacon staff on the

presentations in preparation for training stakeholders. At the end of the fiscal year, one

training session was conducted for regional staff.

In collaboration with Qlarant, DBHDD conducted analysis on review tools. Based upon the

results of the analysis, efforts to help reduce administrative burden on providers, and

eliminate redundancy between tools, DBHDD requested the review processes be modified.

These changes included discontinuation of the Person Centered Review as well as several

review processes from the QEPR: the individual interview, provider staff interview, Support

Coordinator interview and observations. Consequently, in the last quarter of the year,

Qlarant’s efforts focused on modifying the record review tools, review procedures, the web-

based application and QEPR reports.


13 | P a g e

FY 2019 Key Findings

Based on data collected over fiscal year 2019, in conjunction with review and analysis of trends

associated with previous year’s data, the following areas of strength and improvement opportunities

have been identified.

Behavioral Health Key Findings

The statewide average Overall BHQR score improved from 88 percent to 90 percent from FY

2018. In other words, the behavioral health Provider Network, in aggregate, improved their

performance, as aided by the associated technical assistance provided during the reviews and

ongoing training. Additionally, each category demonstrated significant increases in its

statewide average with the exception of Service Guidelines which remained high at 90

percent for both FY 2018 and FY 2019.

The BHQR Assessment and Planning category showed significant improvements related to

the three indicators that scored the lowest in the prior year:

o “Co-occurring health conditions being addressed/included in the IRP/NCP”

(FY 2018 60%; FY 2019 70%)

o “Discharge plan defining criteria”

(FY 2018 66% FY 2019 76%)

o “All assessed needs are addressed”

(FY 2018 67%; FY 2019 77%)

The FY 2019 Assertive Community Treatment (ACT) scores remained consistent to the

previous year’s results related to Focused Outcome Areas (93%), Service Guidelines (84%),

and Overall score (89%) with a statistically significant three-point increase for Assessment

and Planning to 90 percent.

Fifty-three percent of services reviewed with the BHQR Service Guidelines demonstrated

improvement from FY 2018 to FY 2019. For example, Nursing Assessment and Health Services

scores continuously improved from year to year since FY 2017 to reach 89 percent. Three

indicators for Nursing Services that demonstrated the greatest improvement were those


14 | P a g e

indicating that education was being provided to individuals related to their identified health

issues, specific service goals being individualized with progress notes indicating individuals’

progress (or lack thereof), and individuals’ responses to treatment and planned

interventions.

All BHQR FOA categories exceeded 90 percent with the exception of Safety. For the second

year in a row, of the three BHQR FOA Safety indicators, the one most often scored “No” was

“Individuals (or their legal guardians) signed medication consent forms along with the

prescriber.” Although results of this indicator significantly increased by 5 points, the score for

this indicator only reached 73% for FY 2019.

All CSUQR Whole Health FOA results improved significantly demonstrating documented

safeguards for utilized medications known to have substantial risks or side effects (93%),

ongoing assessment to determine external referral needs (99%), as well as communication

with external referral sources and providers as needed (99%).

While no significant improvements were identified specifically in CSUQR Service Guidelines,

several measured standards or indicators related to staff access, staff ratios, and staff

availability were met at 100 percent.

Intellectual and Developmental Disabilities Key Findings

Person Centered Review results showed the following:

Individuals receiving services through the DBHDD system were satisfied with those services,

felt safe, and had access to supports and services to address health needs (98% or higher).

Approximately 99 percent or more of individuals had access to supports and services to

address health needs and access to needed medications.

Individuals reported being free from all types of abuse, staff providing services appeared to

be aware of how individuals self-preserve, and the individuals served felt safe in their work

and living environments (99% or higher).

Over 90 percent of Individuals reported they participate in life’s decisions, were offered

choices, or were involved in the development of their ISP.


15 | P a g e

Most individuals interviewed indicated having privacy (99.6%), did not have any rights

restrictions (99.9%), and were treated with respect (99.6%).

Since FY 2017, Safety and Rights have generated the highest scores, over 90 percent each year, which

suggests providers have systems and practices in place to help individuals be safe in their

environments and exercise and learn about their rights. Individuals receiving services have indicated

they do feel safe and their rights are upheld. In FY 2019, the Individual Interview, Staff Interview and

Observation tools for both the PCR and QEPR scored over 90 percent. Consistent with previous years,

scores on the interviews and observations were higher than documentation by providers and

Support Coordinators (record reviews).

Behavioral Health Assessed Areas of Need

While most categories of the BHQR improved, the category of Service Guidelines maintained

the FY 2018 statewide average, 90 percent. Twenty services were reviewed with

improvements noted in only half of those services; thus, additional focus is to be dedicated in

the upcoming fiscal year to various services and ongoing analysis of the additional

“Programmatic” section piloted in FY 2019.

Although Billing scores increased statewide by a minimal one point for BHQRs, 57 percent of

reassessed providers (or providers not meeting the required 90 percent threshold at review)

had a decline from their first FY 2019 review to their second FY 2019 review with several

providers more deficient in “content does not support units billed” than the previous year’s

discrepancy of “missing or incomplete service order.”

While the BHQR Assessment and Planning category resulted in significant improvements

related to three lowest indicators; “co-occurring health conditions being addressed/included

in the IRP/NCP”, “discharge plan defining criteria”, and “interventions/objectives being goal-

linked and service-consistent;” the exact opposite was found in the case of CSUQRs by which,

the same exact indicators declined to a result of 36 percent (FY18 54%), 48 percent (FY18

69%), and 90 percent (FY18 96%), respectively.

All BHQR and CSUQR FOA categories exceeded 90 percent in FY 2019 except for Safety. For the

second year in a row, the BHQR FOA Safety indicator most often scored “no” was “Individuals


16 | P a g e

(or their legal guardians) signed medication consent forms along with the prescriber”.

However, results of this indicator have continued to trend upward yet again by five points to a

result of 73 percent for FY 2019.

Specific to the CSUQR IRR, only Admission/Initial Evaluation/Initial Screening and Course of

Stay had categorical improvement; yet declines were evident in Treatment Planning (70%) and

Discharge/Transition Planning (56%) by four points and 36 points respectively.

Within the Transition/Discharge Summary subcategory of the Individual Record Review (IRR),

two new indicators were added in FY 2019 with both resulting in exceptionally low numbers of

records providing such evidence that “A discharge summary was entered into the

Collaborative’s Provider/Connect/batch system,” and “If individual was discharged to a

homeless shelter, documentation reflects alternative were explored;” 24 percent and 36

percent respectively. These indicators, although new to the measurement review tool for FY

2019, were not new standards set forth by DBHDD. The indicators were added to address

deficits identified throughout the past three years via assessor reviews and additional

comments identified.

Although part of a non-scored element within CSUQR IRR category, follow-up and connection

to continuing care statistically decreased (p < .01) by 12 points to a low result of 39 percent.

Moreover, vital signs being documented, also a non-scored indicator, improved (p < .01) from

62 percent in FY 2018 to 72 percent in FY 2019. Records pertaining to individuals aged 17 or

below met this standard in 57 percent of the records reviewed compared to those aged 18

and over at seventy three percent).

During the CSUQR, policy adherence declined from the previous year’s results for both

theraputic blood level monitoring (90% to 77%) and safe storage of medication (86% to 59%).

Five new providers were reviewed and analyzed in FY 2019 that had never before had a quality

review completed. Of the five providers, three scored within the ten lowest scoring reviews

for Overall score, with two of those providers receiving a zero percent for the Billing category.

Twenty-eight providers were reviewed for a second time in FY 2019 due to either low scores in

Overall, Billing, or both. While this was an influx due to increased thresholds, it was still less


17 | P a g e

than the number of additional reviews conducted in FY 2017 (N = 35). While it is anticipated

that providers who are reassessed are likely to obtain an elevated score when measured on

the same indicators, only 36 percent (N = 10) actually received an increased score by year end.

Intellectual and Developmental Disabilities Assessed Areas of Need

Since FY 2018, staff supporting individuals receiving services was significantly less likely to

describe how individuals are supported to learn about medications (57.3% compared to

68.4%) and significantly less likely to document how education is provided on all prescribed

medications (27.9% compared to 48.3%), their risks and side effects (37.5% compared to

52.1%), or show evidence psychotropic medications were prescribed by a psychiatrist or

psychiatric nurse practitioner (86.9% compared to 96.5%).

Approximately half of providers had at least one staff member who did not complete training

within the required timeframes, specific to medications and their side effects.

Over 40 percent of PCRs included recommendations related to providing education regarding

abuse, neglect and exploitation and helping individuals receiving services learn how to

manage safety situations by conducting “what if” scenarios based on the following reasons:

o Approximately 35 percent of individuals receiving services were not aware of what

constitutes exploitation or neglect.

o Approximately 54 percent of providers had not offered education on how to self-

preserve or develop effective resiliency skills according to the individual’s learning

style.

Fewer than half of Support Coordinators interviewed demonstrated a knowledge of talents or

strengths of people they served, or regularly reviewed goals with individuals receiving

services. In addition, providers and Support Coordinators both demonstrated a significant

decrease, since FY 2018, of documenting how individuals respond to services or supports.

Community Life remains the lowest scoring area in the PCR and QEPR:

o Developing new social roles and maintaining those valued roles are instrumental ways

to increase community involvement and integration. Information from individual

interviews suggests approximately 30 percent to 40 percent of individuals receiving


18 | P a g e

services had not been given the opportunity to learn about or develop new social

roles in the community, or had support to maintain valued social roles.

o Provider and Support Coordinator documentation often does not show support for

individuals to seek employment in integrated settings (only 27.9% and 44.1% met this

expectation respectively).

o Only 8.4 percent of ISPs were written to promote full participation in the community.

o The ISP Likert Scale indicators demonstrated 46.9 percent of ISPs scored in the lower

two ratings, Emerging or Needs Improvement, showing summaries often did not

reflect the person’s community life. In addition, 41.8 percent of the relationship maps

scored in the lowest two ratings, which shows a lack of demonstration of how the

person is connected to the community.

Approximately 50 percent to 70 percent of provider and Support Coordinator records lacked

evidence that informed choice was provided for competitive or supported employment, living

situations, environments, community participation/social interactions, or for opportunities

for education, exploration or experiences in the community. This seems related to findings

that half of provider staff interviewed had not actually provided options for competitive

employment.

Over the past several years, documentation by providers and Support Coordinators has

shown consistently lower scores when compared to other information gathering techniques

(interviews or direct observations).

FY 2020 Areas of Focus

Behavioral Health

Throughout the year, Quality Management monitors review scores, specific service details,

and indicators to evaluate provider trends. With these findings, the Collaborative identified

and recommended changes to the BHQR and CSUQR tools to further supplement the quality

review process and reporting. Therefore, a new section entitled “Quality Risk Items” is being

added to the behavioral health review tools for FY 2020. Quality Risk Items would result in a


19 | P a g e

reduction to the Overall score by two percentage points for each identified item with a 10-

point maximum reduction. For a complete list of Quality Risk Items, please refer to the

Quality Management chapter in The Georgia Collaborative Provider Handbook.

The Collaborative suggested the introduction of additional reasons claims are not justified

following FY 2019 analysis. These additional reasons would better align the Collaborative’s

scoring policies with other auditing entities. These additional discrepancy reasons are also

intended to increase the quality of documentation within the individuals’ records by asking

providers to align their documentation more fully with existing standards.

Continue annual review and evaluation of existing BHQR and CSUQR tools and requirements,

especially for new or recently-added indicators to align with current state requirements and

DBHDD recommendations in accordance with the DBHDD Provider Manual.

Develop and continue the FY2020 Quality Training Services based on FY2019 data analysis.

Three of the five new providers who have never been reviewed by Quality Management,

scored within the ten lowest of all 157 reviews conducted. Furthermore, analysis at the

indicator level provided an additional level of detail by which it became evident new

providers require further training or technical assistance prior to their initial review occurring.

Consequently, Quality Management is recommending a collaboration and inclusion of a

training model specifically tailored to first time provider reviews.

In FY 2019, DBHDD released its FY 2017 BH Suicide Mortality Report. Through review of this

document and analysis of provider review findings, the Collaborative began a Quality

Improvement Study on suicidal ideation in FY 2020. Utilizing a tracer methodology, the study

will track individuals who have been readmitted to a CSU and with suicidal ideation at

admission.

It is notable that all individual and staff interviews averaged above 96 percent since FY 2016.

As a direct result of such high four-year findings, it has been recommended by the

Collaborative and approved by DBHDD to eliminate staff interviews beginning in FY 2020.

Additionally, Quality Management collaborated with DBHDD’s Office of Recovery to revise

the individual interviews to a set of ten questions.

https://www.georgiacollaborative.com/providers/policies-procedures/


20 | P a g e

The Programmatic category pilot proved successful through FY 2019, supplying supportive

supplemental information related to certain services. Quality Management, at the request of

DBHDD will continue to collect data specific to these indicators and requirements as a non-

scored element.


Based on changes to the reviews outlined in the IDD section of Quality Management

Activities, the Collaborative made several improvements to the review processes, tools and

reports, which will be implemented in FY 2020. These modifications will allow the

Collaborative to review all providers over a two-year period, which is a substantial increase

from 100 reviewed each year previously. Other modifications included the following:

o Publicly posting QEPR Final Assessment reports to the Collaborative’s website.

o The Exit Conference and Final Assessment QEPR reports have been extensively

modified to present results of the QEPR with graphics and data tables. The reports

now include sections for information related to the following:

Immediate Action Items: areas the provider should address immediately and

typically relate to significant documentation issues or health, safety or rights

of the individual.

Key Findings: a new section which provides justification for the scores on the

tools.

Recommendations: identifies, by review tool, areas where the provider needs

to focus efforts related to documentation and compliance requirements.

Additional Consultation and Suggestions for Quality Improvement: allows the

Quality Assessor to document technical assistance provided during the QEPR

and offer suggestions on how the provider can improve practices, processes

and documentation.

o Review tools were reformatted to provide additional guidance to Quality Assessors,

specifically in the documentation, and the specific reasons an indicator may be scored

not present.


21 | P a g e

o New tools to score Service Guidelines were developed and include the format

mentioned above.

The Collaborative will work with the DBHDD Office of Health and Wellness to identify

initiatives promoting independence in maintaining health and staff training relating to the

individuals’ medications. The sessions should include ways to determine if individuals

receiving services understand, to the best level possible, how to maintain personal health

and the benefits/risks of prescribed medication.

Quality Management is recommending, with support from DBHDD, the Georgia Learning

Community could develop “communities of practice” and not only focus on person centered

practices but, also, social role development and community integration. Advocacy groups

and providers across all the regions could be solicited to help generate ideas of how to

improve an understanding of what social roles are, as well as how to explore and develop

new roles. Collaborative Quality Assessors could attend meetings and add additional support

and ideas as needed.

Quality Management recommends developing a training to address how to document

choices offered and how providers can support individuals to make informed decisions. This

training would target support staff and Support Coordinators.

New providers have joined the DBHDD service delivery team, new staff have joined existing

providers, and new requirements for providers may have impacted the quality of provider

documentation. Quality Management will identify key areas from this report where

documentation is clearly falling behind interview findings and recommends targeting these

specific to training for providers and training for Support Coordinators.

In Conclusion

During FY 2019, the continued collaboration between DBHDD and Quality Management resulted in

an increased awareness and education to the BH and IDD network specific to:

Justification of billing (BHQR)

Focused Outcome Areas documentation of Whole Health (BHQR/CSUQR)

Focused Outcome Areas documentation of Rights (BHQR/PCR/QEPR)


22 | P a g e

Education on health (QEPR)

Both IDD and BH quality reviews indicate areas of focus such as safety/crisis planning and medication

education. It has been the goal of DBHDD and the Collaborative to integrate quality measurement

processes relating to BH and IDD services. Side-by-side comparisons are difficult to evaluate due to

the use of different types of sampling methods for each discipline, as well as differences in the tools

and standards utilized. During FY2019, Quality Management, in conjunction with the DBHDD Office

of Quality and divisional subject matter experts, reviewed the behavioral health and intellectual and

developmental disability record review tools to identify modifications that will enhance the ability to

make these types of comparisons. Recommendations and tool revisions have been approved to be

implemented FY2020.


23 | P a g e

Section 2: Introduction

The Georgia Collaborative Administrative Services Organization (the Collaborative), contracted by

Georgia’s Department of Behavioral Health and Developmental Disabilities (DBHDD), continued to

employ 30 Quality Assessors to complete more than 3,100 quality reviews and interview over 13,400

individuals and staff across the state of Georgia since contract inception. The mission of Quality

Management within the Collaborative is to provide DBHDD with valuable and beneficial data for use

in decision-making regarding the quality of services, funding, and development of programs (among

other quality improvement initiatives).

This annual report is the result of assessing, gathering, compiling, analyzing, and measuring the

quality of the service-delivery system through assessment of and technical assistance provided to

DBHDD’s behavioral health and intellectual and developmental disability providers. Furthermore,

this report contains suggestions for modifications in some processes, methods, approaches, and

tools to measure the quality and impact of services. Quality Management recognizes that quality

assessment or measurement processes must have a quality review periodically to examine and

determine whether current tools are measuring those things that are most important and

meaningful to those receiving services and DBHDD.

This annual report includes behavioral health, intellectual and developmental disability, and crisis

stabilization unit (CSU) findings as well as a detailed explanation of the review processes, analysis,

and comparisons of network performance across providers and information from individuals about

the services received. A difference in proportions test was used to determine statistical significance

and, where applicable, determine change from year to year.2 Areas highlighted in blue within tables

reflect significance at a p < .01 or have the p-value defined. While there are similarities in the review

processes, there are also distinct differences; therefore, behavioral health, intellectual and

developmental disability, and CSU results are reported separately. The report is divided into the

following sections:

2 Bohrnstedt, George W. & Knoke, David. (1988). Statistics for Social Data Analysis, 2nd Edition. Itasca, Illinois. F.E. Peacock Publishers,

Inc., pgs. 198-200.


24 | P a g e

Behavioral Health Quality Reviews

o Assertive Community Treatment

o Crisis Stabilization Unit Quality Reviews

o New Provider Reviews

o Reassessment Frequency Reviews


o Person Centered Reviews

o Quality Enhancement Provider Reviews

o Quality Technical Assistance Consultations

Summaries of findings and recommendations after each service delivery system section to

address areas needing improvement or training/education programs to help improve the

quality of services provided to Georgians

Provider and Individual Feedback Surveys

Two appendices are attached

o Acronym and abbreviation list, Appendix A

o Distribution graphs for all tools used in the BHQR, CSUQR, and intellectual and

developmental disability reviews, Appendix B


25 | P a g e

Section 3: Behavioral Health Quality Reviews

Background

The purpose of the Behavioral Health Quality Review (BHQR) is to determine providers’ adherence to

DBHDD’s standards and to assess the quality of the service delivery system through individual record

and claims reviews.3 Review indicators are based on DBHDD and Medicaid requirements, and are

organized into four categories: Billing Validation, Assessment and Planning, Service Guidelines, and

Focused Outcome Areas (FOA). The score for each category represents the percent of relevant

indicators met or present. The BHQR Overall score is calculated by averaging the scores for the four

categories.4 Each category accounts for 25 percent of the Overall score. In collaboration with DBHDD,

minor alterations were made before and throughout the fiscal year (FY) 2019 review process to

adhere to DBHDD and Medicaid requirement changes (if any), as well as language modifications

specific to tools, providing more clarity for Quality Assessors and providers. Additionally, the BHQR

contains a fifth category that is for data gathering purposes only, BHQR Programmatic. Please refer

to the BHQR Programmatic section of this report for more details.

Consistent with previous fiscal year reviews, when a BHQR is completed the lead assessor convenes

the team to analyze data gathered and plan the exit conference. The exit conference provides

immediate, preliminary feedback of the BHQR findings to the provider. A report of these preliminary

findings is left with the provider in the form of an exit conference report outlining the provider’s

identified strengths and any opportunities for improvement from the four primary categories of the

review. Quality Assessors include any items falling outside the parameters of the review determined

to be an area of concern or risk. Extensive technical assistance is provided during the review and exit

conference.

3 Please refer to the following link to access a full description of the review process and review tools. The Georgia Collaborative - Quality Management BH 4 The FOA subcategories are individually scored and are not averaged for the final overall FOA result at the review level. The final overall FOA result is calculated by adding all “Yes” or “Present” responses of all FOA subcategories and dividing by the total “Yes” or “No” responses of the combined subcategories for each review.

http://georgiacollaborative.com/providers/prv-BH.html


26 | P a g e

Within 30 days of completion of a BHQR, a final assessment report is posted on the Collaborative’s

website and the provider is notified via email of the posting, along with the agency’s final score(s).

Like the exit conference report, the final assessment identifies strengths and opportunities for

growth in the four categories of FOA, Billing, Service Guidelines, and Assessment and Planning. The

final assessment highlights areas of concern falling outside the scope of scoring. The final

assessment also includes specific recommendations for improvement and comparisons to a

statewide average based on the previous fiscal year’s results.

Providers are offered an opportunity to appeal the BHQR findings. Appeal information, including

timeframes for submission, is provided upon notification of the final report and is made available to

providers by visiting the Collaborative’s website or reviewing the Georgia Collaborative Provider

Handbook.

Sampling Method

Individual Records and Billing Review

During the 2019 fiscal year, 131 providers were eligible for review, as determined by DBHDD, and

received a BHQR. This sample represents a slight reduction from the previous year in which 136

providers were reviewed. In FY 2019, five reviews were conducted for providers new to the DBHDD

network. An analysis of these new providers is contained within this report.

Whether an active DBHDD provider received a BHQR during a fiscal year is based on multiple factors,

including: volume of individuals served, claim volume, and type of services provided. Additionally,

while efforts to review all providers within the behavioral health (BH) network are made, some

providers may be classified as ineligible due to site closure, changes to location or site, or at the

direction of DBHDD. A sample of individuals was selected for each of the record reviews, and a

sample of those individuals’ claims was used for the billing review. To be included in the sample,

each individual selected must have had at least three claims within the three months (or longer, if

necessary) preceding the BHQR. Providers deemed ineligible because of minimal claim availability or

individuals served (< 5) within the timeframe are monitored periodically for reevaluation of

eligibility.

https://www.georgiacollaborative.com/reports/bh-final-quality-assessment-reports/

https://www.georgiacollaborative.com/providers/quality-management/




27 | P a g e

To select the individuals as part of the record review, an unduplicated list of individuals receiving

services from the provider was stratified by service and payer source: Medicaid, state-contract, and

fee-for-service paid claims. The sample was selected proportionate to the providers’ ratio of

individuals served by payer source, ensuring each service provided and approved for review by

DBHDD was represented. The sample for each provider consisted of up to 30 records for non-

intensive and specialty service providers, with an additional 15 service-specific records for providers

who also offered Assertive Community Treatment (ACT). The number of records selected per

provider was based on the number of individuals served during the six months prior to review,

thereby grouping a provider into one of the following categories.

Small Providers (serving 5-50 individuals) = 5-10 records selected

Medium Providers (serving 51-100 individuals) = 20 records

Large Providers (serving 101+ individuals) = 30 records

Crisis Stabilization Unit Quality Reviews = 15 records

Assertive Community Treatment Providers = 15 additional records

Providers serving fewer than five (< 5) individuals are monitored and added to the review process

once at least five individuals have been served; however, these providers can be selected for an ad

hoc review at the direction of DBHDD. Additional varying analysis occurred throughout the year by

Quality Management to evaluate appropriate service representation, which too is impacted by

individuals served (by the provider) as well as number of claims billed and is discussed further in the

Service Guidelines section of this report.

A representative sample of the population is preferred using standard sampling calculations. The

calculated sample size (even when calculated at the individual provider level based on consumer

utilization) was, in some instances, beyond the scope and resources available to both the provider

and the Collaborative (i.e. one provider may be required to have 200+ records reviewed). Thus, in

quarter one of FY 2019, a formal Power Analysis5 was conducted utilizing past provider results and

5 The Power Analysis was conducted utilizing a difference in Means test, within G*Power version 3.1.9.2 accessible via http://www.gpower.hhu.de, standardizing Power at .80 and alpha at .05.

http://www.gpower.hhu.de/


28 | P a g e

scores to decipher a statistically relevant sample size needed and to ensure the groupings

established above were, in fact, sufficient in having the sample represent the population.

Statistically relevant sample sizes by groupings (Small, Medium, Large, and CSU) were confirmed

after removing outliers.

Oversampling of individuals at each quality review began in FY 2018 to preserve the integrity of the

sample and to ensure a full sample is reviewed. The oversample includes additional, unique

individuals following the above outlined sampling method. Providers receive the list of individuals

(including the oversample) at the start of the review process. Individuals listed on the oversample,

but not reviewed, are deleted from the review.

During FY 2019, 3,353 individuals were sampled for record reviews, 66 (2%) of which were age(s) 17

and under. The total number of records is nearly 300 more than the previous year. This is most likely

due to the standardization and confirmation in sampling methods and requirements specific to

provider size, as well as results of past reviews. Furthermore, an increase in Overall score and Billing

score thresholds occurred for the beginning of FY 2019, thereby rendering additional review

frequency to the network of having a review completed every six months.

With the increase in the FY 2019 threshold to 90 percent, 28 providers were required to have a

reassessment. This is a 40 percent increase compared to the previous year (N = 20). However, this is

still fewer than the initial baseline year in which thresholds were introduced (FY 2017, 80 percent

Overall and Billing scores) in which an additional 35 providers were reassessed See the Reassessment

Frequency Review section for more information.

FY 2019 data also provided an additional level of detail specific to type of diagnosis reviewed by

individual, where 85 percent of all records had a confirmed mental health diagnosis. While the

population is not stratified at sampling specific to age, some differences are clearly identifiable

between those aged 17 and under compared to those 18 and over and are mentioned accordingly

throughout the report. Furthermore, individuals may have more than one diagnosis selected during


29 | P a g e

the record review process. The following information shown in Figure 1, demonstrates the diagnostic

categories.

Figure 1. FY 2019 Diagnostic Categories Reviewed

All records (N = 3,353) 17 and Under (n = 66)

For each record, a random sample of up to 10 paid claims was selected for a billing review for both

BHQR and ACT reviews and the total number of claims reviewed for FY 2019 was 28,935 compared

to 25,598 in FY 2018. The number of records reviewed at a provider directly affects the total number

of claims reviewed. As there was an increase in the number of records reviewed (298), there was

also an increase in the number of claims reviewed. When providers did not have adequate claims

submissions in the three months preceding their review, the claims selection timeframe was

extended, but did not precede the provider’s previous BHQR timeframe (when applicable).

BHQR Billing Validation claim(s) reviews focused on specific services. The services included are listed

in Table 4 and Table 5. All eligible providers were reviewed at least once during the fiscal year and

had at least one claim per billed service included in the claims review, ensuring the complete array of

services provided were included in the BHQR.

Individual and Staff Interviews

In FY 2019, 572 Individual Interviews and 620 Staff Interviews were completed. Samples used for the

Individual and Staff Interviews were selected by the provider and Quality Assessors conducting the

BHQR; services received or provided were not considered in the selection of interviewees. Interview

sampling methods remained the same for FY 2019 with no changes since inception. Quality

Assessors attempted to complete a minimum of five Individual and five Staff Interviews per BHQR.

However, the actual number interviewed fluctuated based on individual and staff availability, their

IDD

7%

Mental Health

85%

Substance Use

29%

Physical

41%

IDD

27%

Mental Health

96%

Substance Use

6%

Physical

17%


30 | P a g e

agreement to participate in the interview process, the number of employees, and the number of

individuals the provider served at the time of the review. If an individual or staff declined an

interview, the Assessors would select an additional individual or staff to be interviewed, when

possible. When reviewing providers who served individuals for both non-intensive outpatient

services (NIOP) and ACT services, Assessors would attempt to interview at least one individual

receiving ACT services and one staff member from the ACT team.

There was a decrease in both Individual and Staff Interview participation by approximately 76

interviews from the previous year. Analysis has shown that fewer interviews were conducted at

smaller provider locations (serving 5-50 individuals). Many Small providers (n = 7, 12%) had only one

interview completed, while several reviews (n = 47, 30%) had only three Individual/Staff Interviews

conducted. The primary reason(s) for fewer interviews completed was due to the lack of availability

of individuals served and staff availability. Results from interviews conducted are not included in the

BHQR Overall scores; however, results have provided valuable, qualitative feedback to promote

quality improvement activities. As an additional quality initiative, the results of interviews were

shared with providers and DBHDD to provide direct communication of individuals’ perceptions and

experiences of receiving services and staff’s knowledge and expertise related to providing services.

BHQR Overall Review Scores In this report, data is aggregated and presented by overall provider scores as well as by category. The

four main categories of Billing Validation, Assessment and Planning, Service Guidelines and Focused

Outcome Areas (FOA) each account for 25 percent of the Overall score. Each FOA (Whole Health,

Safety, Person Centered Practices, Community Life, Choice, and Rights) also has scores displayed and

discussed.

BHQR Overall Scores

FY 2017

84%

FY 2018

88%

FY 2019

90%


31 | P a g e

Figure 2 shows a graphical representation of the average Overall score for the 157 reviews and the

scores for each category compared to the previous two fiscal years to demonstrate areas of

improvement and decline. The mean, or statewide average, of Overall scores for FY 2019 was 90

percent. This is a two-point increase compared to FY 2018 and a six-point increase from FY 2017.

While the FOA category demonstrated the most consistent improvements across fiscal years (FY19

94%), Assessment and Planning continued to trend upwards with a FY 2019 result of 88 percent.

Moreover, where Assessment and Planning demonstrated a historically low result across years with a

final FY 2019 result of 88 percent, Billing became the FY 2019 lowest-scoring category with a result of

86 percent (Figure 2) .

The category of Service Guidelines, historically the highest category since inception (FY 2016),

continued to maintain a steady statewide average at 90 percent. The following are key findings of the

overall category scores:

Assessment and Planning had the greatest improvement from the previous fiscal year by four

points. With an eight-point difference since inception, and 11-point increase from FY 2017,

Assessment and Planning quality indicator results continued to rise from year to year.

Providers have demonstrated being receptive to education and technical assistance from

Assessors specifically in the areas of documentation that supports the incorporation of whole

health and wellness goals into Individual Recovery/Resiliency Plans (IRPs), addressing co-

occurring health conditions, creating clear and identifiable discharge planning criteria, and

addressing all assessed needs. See Assessment and Planning section for indicator results.

Billing Validation consistently trended upward for the fourth year with scores of 81, 84, 85,

and 86 percent respectively as depicted in Figure 2.

The FOA category continued to demonstrate an upward trend in results through FY 2019.

Given the relatively high scores in this category (exceeding 90 percent for the second year in

a row), it seems providers may be more attentive to the standards and measured outcomes

in this area than they were four years ago. For example, Whole Health scored 63 percent in

FY 2016, but demonstrated significant improvement over the years (FY17 74%; FY18 84%;

FY19 91%).


32 | P a g e

Service Guidelines’ statewide average was equal to the previous year’s score of 90 percent.

Nearly three quarters of the reviews (73%, N = 114) met or exceeded 90 percent with ten

reviews receiving a perfect score.

Figure 2. BHQR Fiscal Year Results by Category

BHQR Overall Scores by Tier and Provider Size

In 2014, DBHDD implemented a community behavioral health provider network structure in which

providers were classified using a four-tiered structure.6 Tiers are defined as follows:

Tier 1: Comprehensive Community Providers

Tier 2: Community Medicaid Providers

Tier 2+: Community State Funded Providers

Tier 3: Specialty Providers

6 Policies regarding the implementation and definition of the DBHDD Community Behavioral Health Provider Network Structure can be found at DBHDD’s PolicyStat website: https://gadbhdd.policystat.com

Overall Score FY17: 84%

(N = 167)

FY18: 88%(N = 156)

FY19: 90%(N = 157)

Billing Validation

FY17: 84%FY18: 85%FY19: 86%

Focused Outcome

Areas FY17: 89%FY18: 92%FY19: 94%

Assess. &Planning

FY17: 77%FY18: 84%FY19: 88%

Service Guidelines

FY17: 88%FY18: 90%FY19: 90%

https://gadbhdd.policystat.com/


33 | P a g e

Table 1 provides a snapshot of the BHQR Overall scores by tier and a distribution of scores by each

tier is presented in detail in Appendix B. Results increased from FY 2017 to FY 2018 across tier levels

1, 2, and 2+ but declined for Tier 1 providers from 90 to 88 percent in FY 2019. Both Tier 2 and 2+

providers continued to demonstrate improvement, scoring at 91 and 92 percent respectively. Lastly,

consistent with results of previous years, scores remained unchanged and generally lower for the

group of Tier 3 providers (84%) compared to the other tiers.

Table 1. BHQR Overall Scores by Tier*

Fiscal Year Tier 1 Tier 2 Tier 2+ Tier 3 Statewide Average

FY 2017 85%

(n = 28)

85%

(n = 115)

86%

(n = 6)

83%

(n = 18)

84%

(N = 167)

FY 2018 90%

(n = 29)

89%

(n = 93)

87%

(n = 6)

83%

(n = 28)

88%

(N = 156)

FY 2019 88%

(n = 27)

91%

(n = 102)

92%

(n = 6)

84%

(n = 22)

90%

(N = 157) *N represents the total number of reviews conducted statewide; n represents the subset number of reviews per Tier level. Some providers may

have had a reassessment review thereby increasing the number as compared to previous years.

Provider sizes are established as follows with the Overall score results by size grouping displayed in

Figure 3;

Small Providers (serving 5-50 individuals)

Medium Providers (serving 51-100 individuals)

Large Providers (serving 101+ individuals)


34 | P a g e

Figure 3. Overall Score by Provider Size by Year

Consistent with FY 2018 findings, Medium providers maintained the result of 91 percent and

exceeded the statewide average of 90 percent. Large providers improved in the year to also exceed

the statewide average and obtain a mean result of 91 percent as well. Small providers improved

across the year finishing with a mean result of 88 percent. While there is noted improvement of

three points for the Small group, the group remained three points below the other groups and two

points below the 90 percent Overall score.

Figure 4. Billing Score by Provider Size by Year

80%(n = 71)

90%(n = 29)

90%(n = 56)

84%(n = 58)

88%(n = 31)

88%(n = 68)

70%

75%

80%

85%

90%

95%

Small Medium Large

2018

2019

2019

2019

2018

2018

85%(n = 71)

91%(n = 29) 89%

(n = 56)88%(n = 58)

91%(n = 31)

91%(n = 68)

70%

75%

80%

85%

90%

95%

Small Medium Large

2018

2019

2019

2019

2018

2018

Statewide

Average

FY 2019

90%

(N = 157)

Statewide

Average

Billing

Score

FY 2019

86%

(N = 157)


35 | P a g e

In Figure 4, Small providers struggled to meet the threshold or statewide average of a 90 percent

Overall score, Medium and Large providers in FY 2019 exceeded the Overall result and the same was

the case specific to Billing scores. Small providers, while improving from FY 2018 to FY 2019 by four

points, continued to fall below the FY 2019 statewide result of 86 percent, where Medium and Large

providers, although declining both by two points, continued to exceed the network result of 86

percent. Further details regarding billing result declines will be explained further in the Billing

section of this report.

Table 2. BHQR Category Scores by Provider Size FY 2018 / FY 2019

Provider Size Overall Billing Service

Guidelines Assessment &

Planning FOA

Small 85% / 88% 80% / 84% 89% / 88% 82% / 85% 91% / 94%

Medium 91% / 91% 90% / 88% 92% / 92% 88% / 90% 95% / 94%

Large 89% / 91% 90% / 88% 91% / 92% 85% / 89% 92% / 93%

Statewide 88% / 90% 85% / 86% 90% / 92% 84% / 88% 92% / 91%

In review of all category scores across provider sizes, Table 2, Small providers had increased results of

three to four points across all categories with the exception of Service Guidelines. Medium and

Large provider improvement/declines were minimal if at all. However, Medium providers resulted in

a one-point average decline in their FOA score whereby Large providers increased in their average

FOA score by one point. While Small providers make up 45 percent of the reviews conducted (N =

71, or 61 unique providers), they have fewer individuals reviewed (5-10); thereby increasing the

impact of missing elements or quality indicators scored negatively.

BHQR Billing Validation

The Billing score for each BHQR is the percent of justified paid dollars divided by the total paid

dollars for the reviewed claims. Billing scores are averaged across the network of annual reviews to

FY 2017

84%

FY 2018

85%

FY 2019

86%


36 | P a g e

obtain an annual statewide average. This category continued to trend upward from contract

inception, beginning in FY 2016 with a statewide average of 81 percent and ending FY 2019 with an

average of 86 percent. Improvement at the provider level is evident as in FY 2019 where 13

providers achieved a perfect 100 percent, compared to nine providers in FY 2018. One provider, not

previously reviewed, scored 0 percent in FY 2019. See the distribution of BHQR Billing scores in

Appendix B. Since an increase in Billing scores continued in FY 2018, the billing threshold was raised

to 90 percent for FY 2019 to align with DBHDD’s contractual Key Performance Indicators (KPIs). Only

56 percent of reviews (N = 88) met this threshold in FY 2019 and, interestingly, the average Billing

score of reassessed providers declined from 87 percent at the start of the year to 80 percent at year-

end. See Reassessment Frequency Review section for more details.

Figure 5 shows the total dollar amount reviewed through an analysis of paid claims for all providers

reviewed during FY 2019 ($2,892,206.46) compared to FY 2018 ($2,430,951.50) and FY 2017

($2,934,560.52) as well as the dollar amounts found to be justified and unjustified. In FY 2019, the

increase in funds reviewed may be due to several factors, including but not limited to: provider

sample size, claims availability, or the types of service reviewed. Although the total funds reviewed

for FY 2019 were nearly $500,000 more than the previous year, the amounts unjustified and

susceptible to recoupment continued to remain steady with a one-point increase for the year.


37 | P a g e

Figure 5. BHQR Billing Validation Amount Reviewed by Fiscal Year7

Figure 6 provides the percent of reviewed claim amounts by funding source. In FY 2019

$2,417,070.35 in Medicaid funds were reviewed compared to the state funded amount of

$475,136.10. The proportions of claims reviewed for FY 2019 by funding source is comparable to

previous fiscal years.

7 The percent of justified and unjustified dollar amounts, in Figure 5, are total dollar amounts justified or unjustified divided by the total amount of funds reviewed in total for the year. The annual statewide average Billing score is not equivalent to the percent justified/unjustified depicted in the figure; statewide Billing score is calculated based on averaging all provider scores at year-end.

$0.00

$500,000.00

$1,000,000.00

$1,500,000.00

$2,000,000.00

$2,500,000.00

$3,000,000.00

FY 2017FY 2018

FY 2019

$2,471,510.59 84% $2,136,207.68

88%

$2,513,252.73 87%

$463,049.93 16%

$294,743.82 12%

$378,953.72 13%

Total $ Justified

Total $ Unjustified

Total Amount Paid Claims Reviewed in FY 2017: $2,934,560.52Total Amount Paid Claims Reviewed in FY 2018: $2,430,951.50Total Amount Paid Claims Reviewed in FY 2019: $2,892,206.46


38 | P a g e

Figure 6. FY 2018 BHQR Amount Reviewed by Funding Source

For the first year since inception, state funded claims were found to be significantly (p < .01) more

justified than Medicaid claims. Table 3 depicts the justifiable and unjustifiable amounts by funding

source in which 87 percent of Medicaid claims reviewed were found justified compared to 89

percent of state funded claims.

Table 3. FY 2019 Amount Reviewed by Funding Source State Funds Medicaid Total

Amount Justified $422,116.12 $2,091,136.60 $2,513,252.72

(89%) (87%) (87%)

Amount Unjustified $53,019.98 $325,933.75 $378,953.73

(11%) (13%) (13%)

Total Amount Reviewed $475,136.10 $2,417,070.35 $2,892,206.45

The highest funded service(s) reviewed during FY 2019 were Individual Counseling ($507,716.34)

followed by Assertive Community Treatment ($318,879.82), and lastly, Intensive Family Intervention

($312,947.42). While most services had both sources of funding reviewed, some services had

substantially less state funds reviewed for FY 2019 compared to Medicaid funds. However, services

such as Substance Abuse Intensive Outpatient, Community Support Team (CST), and Group

Counseling had over $5,000 more in state funding claims reviewed than Medicaid funded claims.

Medicaid, 84%

State Funded,

16%

Total Amount Reviewed

$2,892,206.45


39 | P a g e

Table 4 and Table 5 provide the amount and percentages of funds justified by service, non-intensive

outpatient and specialty services of all funds combined, for fiscal years 2018 and 2019, respectively.

Table 4. Non-Intensive Outpatient Services Percent of Funds Justified

FY 2018 FY 2019

Service Amount Justified

Amount Reviewed

Justified Amount Justified

Amount Reviewed

Justified

Individual Counseling $380,030.45 $409,683.46 93% $475,262.18 $507,716.34 94%

Psychiatric Treatment $67,178.02 $75,386.25 89% $72,447.94 $78,804.19 92%

Diagnostic Assessment $37,270.98 $39,762.22 94% $37,230.03 $40,854.73 91%

Community Support $128,832.09 $146,364.10 88% $176,844.96 $197,659.23 89%

Behavioral Health Assessment

$75,895.66 $83,962.92 90% $96,288.18 $108,559.58 89%

Group Outpatient Services $107,754.77 $121,168.70 89% $83,356.02 $95,383.66 87%

Family Counseling/Therapy $140,012.07 $166,185.24 84% $178,706.41 $204,926.79 87%

Service Plan Development $51,843.00 $59,530.24 87% $69,778.54 $80,834.59 86%

Medication Administration $3,494.80 $3,876.51 90% $2,821.40 $3,430.62 82%

Case Management $109,814.09 $134,652.93 82% $148,094.90 $184,138.38 80%

Nursing Services $44,389.22 $51,337.63 86% $50,784.83 $64,149.30 79%

Psychological Testing $29,166.90 $35,869.25 81% $19,853.01 $25,261.70 79%

Crisis Intervention $9,301.10 $12,123.05 77% $7,822.65 $11,418.59 69%

Community Transition Planning

$3,103.01 $3,312.21 94% $1,936.18 $3,777.14 51%

Total $1,188,086.16 $1,343,771.74 88% $1,421,227.23 $1,606,914.84 88%

*Areas highlighted in blue within table reflect significance at a p < .01.


40 | P a g e

Table 5. Specialty Services Percent of Funds Justified*

FY 2018 FY 2019

Service Amount Justified

Amount Reviewed

Justified Amount Justified

Amount Reviewed

Justified

Opioid Maintenance $9,430.70 $11,063.50 85% $4,809.60 $4,809.60 100%

Community Support Team (CST)

$13,042.33 $14,985.88 87% $16,742.65 $17,901.24 94%

MH Peer Support - Individual $19,806.24 $26,289.03 75% $27,820.63 $30,251.62 92%

Assertive Community Treatment (ACT)

$295,649.45 $325,851.60 91% $281,284.86 $318,879.82 88%

Addictive Disease Support Services (ADSS)

$54,703.17 $64,477.32 85% $94,439.45 $108,062.11 87%

MH Peer Support Program $83,508.23 $90,261.59 93% $99,132.88 $113,524.56 87%

Intensive Case Management $16,751.46 $19,101.80 88% $26,504.41 $30,534.94 87%

Psychosocial Rehabilitation Program

$66,408.84 $78,011.70 85% $71,760.93 $83,863.64 86%

Psychosocial Rehabilitation - Individual

$151,633.45 $172,628.40 88% $198,060.49 $233,412.49 85%

Peer Support Whole Health & Wellness - Individual

$5,606.28 $5,606.28 100% $3,849.18 $4,553.81 85%

Substance Abuse Intensive Outpatient

Not Reviewed $19,495.66 $24,674.84 79%

Intensive Family Intervention $221,099.15 $267,624.22 83% $246,963.16 $312,947.42 79%

AD Peer Support - Individual $399.30 $1,052.70 38% $1,161.60 $1,875.52 62%

Peer Support Whole Health & Wellness - Group

$10,082.92 $10,225.74 99% Not Reviewed

Grand Total $948,121.52 $1,087,179.76 87% $1,092,025.50 $1,285,291.61 85%

*Areas highlighted in blue within table reflect significance at a p < .01.

When a claim was found to be unjustified, Assessors selected all applicable reasons a reviewed claim

was identified as a discrepancy; therefore, one unjustified claim may have multiple reasons. Figure 7

provides the percent of discrepancies within the year. Of reviews completed that incorporated a

review of the service Community Transition Planning (51% justified funds), the most common

discrepancies identified were “Content does not support units billed,” “Content does not support


41 | P a g e

code billed,” and “Content of note does not match service definition.” Additionally, reviews

conducted for the service of AD peer Support – Individual (62%) also had high discrepancies of

“Content does not support units/code billed,” but also “More contacts/units per day than allowed,”

and “Missing/incomplete service order.” Figure 7 provides the percent of discrepancies within the

year of all service types reviewed.

Figure 7. FY 2019 Percent of Discrepancy Reasons

Other discrepancy reasons not included in Figure 7 include:

Individual does not meet admission criteria for service billed (1.8%)

Date of entry missing (1.6%)

Intervention outside the scope of practice for staff (1.3%)

Non-billable activity (1.3%)

Diversionary activities billed (< 1%)

Printed name missing (< 1%)

Multiple services billed at same time (< 1%)

Signature missing (2.2%)

Location missing (out-of-clinic) (3.1%)

Units billed exceeded time and/or units (3.4%)

Progress note is missing (3.6%)

Billing code is missing or different from code billed (3.9%)

Staff credential missing (5.9%)

Content is not unique to the individual (6.2%)

Content does not support code billed (8.7%)

Content of note does not match service definition (9.3%)

No valid, verified diagnosis on date service provided (9.4%)

Missing/incomplete service order (10.0%)

Staff credential not supported (11.4%)

Content does not support units (11.8%)

0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%


42 | P a g e

More contacts/units per day than allowed (< 1%)

Consistency requirements missing (< 1%)

Date of service incorrect/missing (< 1%)

Time in/Time out missing (< 1%)

Mutually exclusive services billed (< 1%)

Record not submitted within timeframe (< 1%)

While there were nearly 500 fewer unjustified claims identified in FY 2019 some of the most

prevalent discrepancy reasons identified in FY 2019 were similar to FY 2018 and include:

Content did not support units billed – For example, billed eight units for a medication check

with no further documentation of interventions provided.

Content did not support code billed – For example, the out-of-clinic modifier was billed but

the session took place in the office.

Content of note not did not match the service definition – For example, the provider billed

Case Management, but the documentation indicated the practitioner provided Psychosocial

Rehabilitation - Individual.

Historically, common billing discrepancy reasons were individuals not meeting admission criteria,

missing progress notes, and missing signatures. However, improvements have been made in these

areas from FY 2018 to FY 2019 by 72 percent, 56 percent, and 53 percent respectively. Table 6

provides the number of all discrepancy reasons, in ranking order for FY 2019 by the number of

instances, and the change in percent of the instance count from year to year. A negative difference

demonstrates improvement from year to year with highlighted areas demonstrating a statistically

significant change. Overall, 14 (58%) of the observed indicators (N = 24) improved from the previous

fiscal year. One new discrepancy reason was added for the FY 2019 measurement year specific to a

valid/verifiable diagnosis being on file for the specific service date reviewed. For this discrepancy,

there were 417 instances of 28,935 claims or 1.4 percent in FY 2019.


43 | P a g e

Table 6. BHQR Billing Discrepancy Reasons by Fiscal Year*

FY 2017 FY 2018 FY 2019

Instances Instances Instances

Claims Reviewed 29, 602 25,598 28,935

Missing/incomplete order 496 (1.7%) 285 (1.1%) 444 (1.5%)

No valid/verified diagnosis on service date N/A N/A 417 (1.4%)

Does not meet admission criteria 748 (2.5%) 254 (1.0%) 82 (1.8%)

Quantitative

Progress note is missing 676 (2.3%) 326 (1.3%) 161 (0.6%)

Code is missing/different than code billed 379 (1.3%) 244 (1.0%) 172 (0.6%)

Staff credential missing 295 (1.0%) 236 (0.9%) 263 (0.9%)

Signature missing 194 (0.7%) 182 (0.7%) 98 (0.3%)

Date of entry missing 185 (0.6%) 160 (0.6%) 71 (0.2%)

Units billed exceed time/units documented 112 (0.4%) 90 (0.4%) 152 (0.5%)

Consistency requirements missing 57 (0.2%) 31 (0.1%) 28 (0.1%)

Credential not supported by documentation 276 (0.9%) 211 (0.8%) 505 (1.7%)

Record not submitted within timeframe 19 (0.1%) 10 (< 0.1%) 2 (< 0.1%) Location missing (out-of-clinic) 155 (0.5%) 106 (0.4%) 139 (0.5%)

Time in / time out missing 68 (0.2%) 14 (0.1%) 13 (<0.1%)

Date of service incorrect / missing 42 (0.1%) 34 (0.1%) 25 (0.1%)

Printed name missing 2 (<0.1%) 8 (<0.1%) 38 (<0.1%)

Performance Standards

Content does not match service definition 489 (1.7%) 413 (1.6%) 412 (1.4%)

Content does not support code billed 397 (1.3%) 403 (1.6%) 386 (1.3%)

Content does not support units billed 518 (1.8%) 441 (1.7%) 525 (1.8%)

Intervention outside staff's scope/practice 79 (0.3%) 120 (0.5%) 59 (0.2%)

Content is not unique to the Individual 243 (0.8%) 140 (0.5%) 273 (0.9%)

Multiple services billed at the same time 79 (0.3%) 45 (0.2%) 32 (0.1%)

Non-billable activity 59 (0.2%) 34 (0.1%) 56 (0.2%)

Diversionary activities billed 26 (0.1%) 111 (0.4%) 82 (0.3%) *Areas highlighted in blue within table reflect significance at a p < .01 and indicators with n < 90 were not tested due to low volume.

Over the course of the quality review process, providers have become acclimated to the billing

discrepancy reasons and recoupment process that has been established and implemented since

contract inception as evidenced by the consistent declines in discrepancy instances recorded. As

scores specific to Billing continued to increase statewide, the Collaborative and DBHDD have agreed


44 | P a g e

to introduce other areas of policy specific to Billing and to incorporate additional discrepancies in FY

2020 to include the following:

High utilization without clinical justification

Intervention unrelated to IRP without clinical justification

Monthly contacts not met per DBHDD Service Guidelines

No overall progress documented

Note does not include response to intervention

Progress note was not filed within seven calendar days

BHQR Assessment and Planning

Assessment and Planning consisted of ten indicators answered once per record reviewed. Table 7

contains the indicators and percent “Yes” in ranking order. All indicators increased or remained equal

for the second year in a row with areas of growth having continued through FY 2019 to include at

least a ten point, significant increase in each of the four lowest-scoring indicators of Assessment and

Planning:

All assessed needs are addressed

o C&A 62 percent, Adult 77 percent

Co-occurring health conditions addressed in the IRP


Discharge plan defines criteria


Whole health and wellness in IRP


FY 2017

77%

FY 2018

84%

FY 2019

88%


45 | P a g e

Table 7. BHQR Assessment & Planning Indicator Scores by Fiscal Year*

Indicator FY 2017 FY 2018 FY 2019

Current medical screening is present 97% 98% 99%

Individual meets admission criteria 97% 97% 98%

Interventions/objectives are goal-linked & service-consistent 91% 92% 96%

Current behavioral health assessments of needs 94% 94% 96%

IRP is individualized in personalized language 87% 94% 94%

Goals/objectives honor hopes, choice, preferences, outcomes 89% 94% 94%

Whole health & wellness in IRP 56% 69% 82%

All assessed needs are addressed 48% 67% 77%

Discharge plan defines criteria 58% 66% 76%

Co-occurring health conditions addressed in IRP 34% 60% 70%

Assessment & Planning Statewide Average 77% 84% 88% *Areas highlighted in blue within table reflect significance at a p < .01 as tested from the previous year.

Data continues to suggest that, although records contained required assessment documentation

such as medical screenings (99%) and current behavioral health assessments (96%), a vast

percentage of recovery/resiliency plans lacked the documentation of discharge planning, co-

occurring health conditions being addressed, or a plan for increased whole health and wellness. On

July 1, 2018, an additional level of data collection was implemented and tailored to provide

additional detail for reason(s) that indicators within Assessment and Planning were “not met.” In

the case of the one percent of medical screenings not present, 36 records (88%) did not have an

annual medical/wellness update. Also, 77 percent of 137 records did not have an annual update of

behavioral health assessment of needs. The lowest-scoring indicators in the category of Assessment

and Planning, all with significant improvements (p < .01) for the first time in FY 2019, held a wide

array of reasons (new for FY 2019) for having not been met and have the reasons outlined below.


46 | P a g e

• Plan does not address all areas/types of need (60%, n = 464)

• Psychiatric issues not addressed (21%, n = 164)

• Substance use issues not addressed (18%, n = 140)

• Developmental issues not addressed (14%, n = 112)

• Plan is expired (10%, n = 81)

Assessed needs are addressed (23% "Not Met", N = 776)

• Areas identified in assessment but not addressed in IRP: Physical (56%, n = 426) IDD (18%, n = 133) Substance Use (17%, n = 126) Psychiatric illness (16%, n = 120)


Co-occurring health conditions addressed (30% "Not Met", N = 756)

• No whole health & wellness on plan (82%, n = 501)


• Whole health and wellness on plan; not individualized (5%, n = 32)

Whole health & wellness in IRP (18% "Not Met", N = 611)

• No clinical benchmarks indicated (44%, n = 352)

• Specific step-down service(s) not indicated/planned (35%, n = 285)

• Discharge Plan is expired (13%, n = 102)

• Date of discharge/transition no documented/planned (12%, n = 96)

• No plan at all (11%, n = 85)

• Discharge criteria does not align with goals/objectives (5%, n= 43)

Discharge/Transition Planning (24% "Not Met", N = 805)


47 | P a g e

BHQR Service Guidelines

Indicators within the Service Guidelines section of the BHQR were answered once per record

reviewed. The number of indicators answered varied, specific to the service. The service reviewed

for each record was dependent upon the services contained in the billing claims sample for the

respective individual; therefore, multiple services could be reviewed within one individual’s record.

A total of 16 NIOP services and 11 specialty services were identified as measurable services for FY

2019 as shown in Figure 8 and Figure 9. However, some services (Opioid Maintenance or

Medication-Assisted Treatment, MAT) may not have been reviewed for specific providers based on

lack of services billed within the eligible period or at the direction of DBHDD, thereby excluding

service(s) from some reviews.

The percent of dollars reviewed per individual service in FY 2019 was comparable to FY 2018 for both

non-intensive outpatient and specialty services. For example, in FY 2019 Individual Counseling

represented 26 percent of all sampled services, compared to 27 percent in FY 2018.

In analysis of funds reviewed specific to NIOP services, Individual Counseling had the most funds

reviewed (see Figure 8). Psychological Testing, Crisis Intervention, Community Transition Planning,

and Medication Administration all had the least amount of funds reviewed per service, each with

one percent or less reviewed in FY 2019.

FY 2017

88%

FY 2018

90%

FY 2019

90%


48 | P a g e

Figure 8. FY 2019 Percent of Dollars Reviewed by Non-Intensive Outpatient Service

Figure 9 indicates that Assertive Community Treatment (ACT) had the greatest amount ($318,879.82

or 34%) of funds reviewed specific to specialty services; hence ACT is considered a subset sample

with an additional 15 records sampled per ACT rendering provider/review. Intensive Family

Intervention (IFI) ranked second in FY 2019 with 33 percent of all specialty funds reviewed

($948,817.01). However, year-end analysis of data and its findings provided evidence that 37

providers offered IFI services some time during the year, yet only 27 providers had the service

reviewed by Assessors. Again, this was due to either a directive (prepayment review) made by

DBHDD or limited unique individuals being served/claims billed to generate a sample extraction in

line with sampling methods.

Individual Counseling (26%)

Psychosocial Rehabilitation -Individual (12%)

Family Counseling/

Therapy (11%)

Community Support (10%)

Case Management

(9%)

Individual Counseling (26%)

Psychosocial Rehabilitation - Individual (12%)

Family Counseling/ Therapy (11%)

Community Support (10%)

Case Management (9%)

Addictive Disease Support Services (ADSS) (6%)

Behavioral Health Assessment (6%)

Group Counseling/Therapy (5%)

Service Plan Development (4%)

Psychiatric Treatment (4%)

Nursing Assessment & Health Services (3%)

Diagnostic Assessment (2%)

Psychological Testing (1%)

Crisis Intervention (1%)

Community Transition Planning (<1%)

Medication Administration (<1%)


49 | P a g e

Figure 9. FY 2019 Percent of Dollars Reviewed by Specialty Service

Table 8 provides the scores by service type and year of review and represents the number of reviews

assessed for the service, based on the claims sample where “r” references the number of reviews,

and “n”, the number of records.

Assertive Community Treatment (ACT) (34%)

Intensive Family Intervention (33%)

MH Peer Support Program (12%)

Psychosocial Rehabilitation Program (9%)

Assertive Community Treatment (ACT) (34%)

Intensive Family Intervention (33%)

MH Peer Support Program (12%)

Psychosocial Rehabilitation Program (9%)

Intensive Case Management (3%)

MH Peer Support - Individual (3%)

Substance Abuse Intensive Outpatient Program (3%)

Community Support Team (CST) (2%)

Opioid Maintenance (1%)

Peer Support Whole Health & Wellness - Individual (<1%)

AD Peer Support - Individual (<1%)


50 | P a g e

* Areas highlighted in blue, represent significant differences where p < .01 from the previous year. Additionally due to low response volumn, Peer Support Whole Health & Wellness, AD Peer Support – Individual, and Community Support Team (CST) were not evlauted for statisical significance. Also, “r” represents the number of reviews whereas “n” respresnts the number of records.

Service Guidelines scored highest of the four categories following the first year of contract inception

at 90 percent. Although it declined slightly in FY 2017 by two points, it has since returned to 90

percent and remained there throughout FY 2019. Moreover, nearly three quarters of the reviews

Table 8. BHQR Service Guidelines Scores by Service Type*

Service Type FY 2017 FY 2018 FY 2019

AD Peer Support – Individual 48% (r = 1, n = 13) 80% (r = 2, n = 9) 84% (r = 4, n = 16)

Addictive Disease Support Services 85% (r = 69, n = 293) 80% (r = 61, n = 252) 90% (r = 62, n = 337)

Assertive Community Treatment 88% (r = 20, n = 334) 84% (r = 21, n = 336) 84% (r = 20, n = 313)

Case Management 84% (r = 90, n = 746) 88% (r = 85, n = 644) 89% (r = 84, n = 745)

Community Support 83% (r = 114, n = 637) 86% (r = 102, n = 446) 85% (r = 105, n = 550)

Community Support Team 92% (r = 7, n = 13) 99% (r = 10, n = 22) 93% (r = 10, n = 29)

Community Transition Planning N/A 100% (r = 17, n = 18) 94% (r = 17, n = 18)

Family Counseling/ Training 91% (r = 129, n = 904) 94% (r = 109, n = 625) 95% (r = 114, n = 786)

Service Type FY 2017 FY 2018 FY 2019

Group Counseling/ Training 90% (r = 78, n = 561) 95% (r = 75, n = 535) 92% (r = 74, n = 509)

Individual Counseling 94% (r = 150, n = 1980) 96% (r = 132, n = 1390) 97% (r = 137, n = 1583)

Intensive Case Management 94% (r = 13, n = 53) 94% (r = 11, n = 46) 91% (r = 17, n = 81)

Intensive Family Intervention 85% (r = 37, n = 185) 90% (r = 33, n = 149) 88% (r = 32, n = 159)

MH Peer Support - Individual 95% (r = 6, n = 38) 90% (r = 20, n = 92) 95% (r = 27, n = 138)

MH Peer Support Program 86% (r = 25, n = 199) 91% (r = 36, n = 210) 93% (r = 38, n = 238)

Nursing Assessment & Health 80% (r = 116, n = 897) 87% (r = 104, n = 649) 89% (r = 110, n = 734)

Opioid Maintenance Treatment 98% (r = 3, n = 69) 92% (r = 6, n = 52) Not Reviewed

Peer Support Whole Health & Wellness (Group)

68% (r = 12, n = 67)

80% (r = 2, n = 3) Not Reviewed

Peer Support Whole Health & Wellness (Individual)

93% (r = 2, n = 28) 89% (r = 6, n = 25)

Psychiatric Treatment 92% (r = 128, n = 950) 97% (r = 111, n = 717) 97% (r = 119, n = 749)

Psychosocial Rehabilitation Program 93% (r = 26, n = 179) 96% (r = 26, n = 153) 93% (r = 26, n = 804)

Psychosocial Rehabilitation - Individual 85% (r = 98, n = 904) 90% (r = 83, n = 679) 91% (r = 89, n = 143)

Substance Abuse Intensive Outpatient Program Not Reviewed Not Reviewed 88% (r = 9, n = 61)


51 | P a g e

(73%, N = 114) met or exceeded 90 percent with ten reviews receiving a perfect score, all of whom

were classified as a Small provider (N = 5-10 records per provider).

Individual Counseling was evaluated in 87 percent (r = 137) of all BHQRs conducted in FY 2019 and

continued to demonstrate the highest rate of compliance for the second year compared to all

services measured, with 97 percent of records meeting the measured indicators. Improvements in

scores were found evident for both MH Peer Support – Individual (p < .01) and AD Peer Support –

Individual in FY 2019, increasing results by at least five percentage points. Interestingly, the MH Peer

Support Program was evaluated more than any other specialty service spanning 38 of the 157 BHQRs

while AD Peer Support – Individual as a specialty service was reviewed the least (r = 4). Service areas

demonstrating the greatest decline(s) from FY 2018 to FY 2019, were Peer Support Whole Health

and Wellness – Individual (-4.0%), Community Transition Planning (-5.6%), and Community Support

Team (-6.1%). Although not significant, Intensive Case Management (ICM) had over a three percent

decline in records meeting indicator standards at 91 percent compared to the previous year’s 94

percent, with an additional six reviews (r = 17) conducted in the year.

Concerns over appropriate representation of ICM records across reviews completed in FY 2018 (r =

11) prompted Quality Management to evaluate statewide utilization reports and claims submission.

Through said process, it was evident the current BHQR process did not adequately include medical

records with ICM services. Only one percent of records (n=81) in FY 2019 with ICM services were

included in the BHQR samples, yet 15.3 percent of unique individuals received ICM services

statewide. In FY2020, Quality Management will begin including a set number of ICM records in each

BHQR.

Substance Abuse Intensive Outpatient Program (IOP), a newly reviewed service for FY 2019, resulted

in 88 percent compliance (r = 6, n = 61) with four of the six indicators fully meeting the compliance

with Service Guidelines (100%) necessary standards related to:

Clinical coordination/planning to avoid duplication of intervention evident for those

receiving ACT, CST, or Crisis Residential

Individual meets admission or continued stay criteria


52 | P a g e

Individual receiving services/support/skill development necessary for sobriety/use

reduction/recovery maintenance

Progress notes document individual response to the staff intervention provided

Although more than half (11 of 19, 57.9%) of the reviewed services met or exceeded the statewide

average of 90 percent for this category, specific indicators at the service level suggest areas for

improvement. Table 9 provides for the lowest-scoring indicator(s) identified, by service, through the

FY 2019 review process.

Table 9. FY 2019 Lowest-Scoring Indicators by Service* “r” represents the number of reviews and “n” the number of records

Service Name and FY 2019 lowest-scoring indicator(s): FY 2017 FY 2018 FY 2019

Assertive Community Treatment 88% (r = 20) 84% (r = 21) 84% (r = 20)

The ACT Team completes a Treatment Plan Review with the staff, the individual, and his/her family/informal supports prior to the reauthorization of services

59% (n = 307)

28% (n = 321)

28% (n = 304)

There is documented evidence that the ACT Team is working with informal support systems/collateral contacts at least 2-4 times per month (with or without the individual present) to provide support and skills training to assist the individual in his/her recovery.

39% (n = 283)

33% (n = 306)

47% (n = 273)

The ACT Team has all required staff. (List missing staff in comment box.)

91% (n = 334)

72% (n = 336)

65% (n = 312)

Addictive Disease Support Services 85% (r = 69) 80% (r = 61) 90% (r = 62)

Coordination with family and significant others is documented and

with adult individual’s permission. (Review authorization period.)

42% (n = 187)

59% (n = 143)

60% (n = 224)

Community Support 83% (r = 114) 86% (r = 85) 85% (r = 105)

There is evidence of service and resource coordination.

50% (n = 572)

63% (n = 445)

62% (n = 541)

Community Support Team 92% (r = 7) 99% (r = 10) 93% (r = 10)

CST Team Meeting logs document that the individual was discussed, even briefly, at least one time per week. (New for FY19)

Not Reviewed Not Reviewed 59%

(n = 29)

Intensive Case Management 94% (r = 13) 94% (r = 11) 91% (r = 17)

There is a joint development of a crisis plan to include the Provider and individual. The Provider is listed as primarily responsible.

85% (n = 52)

65% (n = 46)

69% (n = 80)

Intensive Family Intervention 85% (r = 37) 90% (r = 33) 88% (r = 32)


53 | P a g e

Table 9. FY 2019 Lowest-Scoring Indicators by Service* “r” represents the number of reviews and “n” the number of records

Service Name and FY 2019 lowest-scoring indicator(s): FY 2017 FY 2018 FY 2019

Documentation reflects a tapering of services. 37%

(n = 63) 75%

(n = 44) 63%

(n = 65) * Areas highlighted in blue within table reflect significance at a p < .01 from the previous year.

The age at which individuals are classified as part of a “child and adolescent (C&A) population” often

depends on the type of service the individual is receiving as well as a multitude of environmental,

physical, and mental factors. Findings were evident in analysis of individuals age 17 and under as

compared to individuals 18 and over. Age group differences were observed in the services of Family

Counseling (n = 25), Group Counseling (n = 6), Individual Counseling (n = 45), Psychiatric Treatment

(n = 17), and Intensive Family Intervention (n = 8). Several indicators were met at 100 percent for

C&A individuals as compared to adults (18+). Indicators not meeting 100 percent for the C&A

individuals across the five service types identified above relate to progress notes documentation and

progress (or lack of) as well as staff intervention in the progress notes relating to interventions listed

on the treatment plan. Results of the two indicators for C&A ranged from 88 percent to 96 percent

for Family, Individual, and Group Counseling yet only 63 percent (n = 5) of IFI progress notes

contained documented progress or lack thereof. Comparatively, these same two indicators were met

at 100 percent for all individuals ages 17 and under for all reviewed records (n = 19) within Nursing

services.

All services reviewed in FY 2019 were scored based on three specific indicators related to

documentation within progress notes. The results indicated significant improvement from the

previous fiscal year (p < .01). Table 10 contains detailed results of these similar indicators and the

results of said indicators by service type for all records reviewed.


54 | P a g e

Table 10. FY 2019 Progress Note Indicator Results by Service Type* “n” represents the number of records

FY 2017 FY 2018 FY 2019

Progress notes document progress (or lack of) toward goals/objectives identified on the IRP.

86% (n = 7,916) 90% (n = 6,875) 89% (n = 7,974)

Staff interventions reflected in the progress notes are related to staff interventions listed on the IRP.

88% (n = 7,869) 92% (n = 6,897) 93% (n = 7,988)

Progress notes document individual response to staff interventions provided.

98% (n = 7,934) 99% (n = 6,812) 98% (n = 7,996)

FY 2019 Progress Note Indicator Result(s) by Service Type

Service Type

Progress notes document progress (or

lack of) toward goals/objectives

identified on the IRP.

Staff interventions reflected in the

progress notes are related to staff

interventions listed on the IRP.

Progress notes document individual

response to staff interventions

provided.

AD Peer Support - Individual 69% 75% 100%

Addictive Disease Support Services 91% 94% 100%

Assertive Community Treatment 95% 96% 99%

Service Type

Progress notes document progress (or

lack of) toward goals/objectives

identified on the IRP.

Staff interventions reflected in the

progress notes are related to staff

interventions listed on the IRP.

Progress notes document individual

response to staff interventions

provided.

Case Management 90% 93% 98%

Community Support 91% 93% 98%

Community Support Team 93% 100% 100%

Community Transition Planning N/A N/A 100%

Family Counseling/ Training 89% 91% 98%

Group Counseling/ Training 81% 90% 93%

Individual Counseling 94% 96% 99%

Intensive Case Management 90% 93% 100%

Intensive Family Intervention 82% 88% 97%

MH Peer Support - Individual 86% 91% 100%

MH Peer Support Program 78% 93% 99%

Nursing Assessment & Health Services 88% 95% 98%


55 | P a g e

Peer Support Whole Health & Wellness – Ind. 83% 84% 100%

Psychiatric Treatment 94% 97% 99%

Psychosocial Rehabilitation - Individual 88% 92% 99%

Psychosocial Rehabilitation Program 81% 92% 98%

Substance Abuse Intensive Outpatient Program 69% 70% 100% * Areas highlighted in blue within table reflect significance at a p < .01 from the previous year.

As providers have continued to maintain a high statewide average in the category of Service

Guidelines over the past several years with the existing array of services, Quality Management began

evaluating specific service data as it relates to both services reviewed and services not historically

under review. Over the course of FY 2019, Quality Management consistently reviewed various

detailed reports, such as the Service Mix Report, as well as weekly claim files to ensure accurate

representation of service type and sampling occurred. Quality Management also provided specific

service information and data related to Supported Employment, Crisis Respite Apartments, and

Intensive Family Intervention to DBHDD. Additional data collection was provided to DBHDD related

to individuals who are identified as deaf and hard of hearing (see Office of Deaf Services Special

Review). As a direct result of the reports made available, DBHDD has requested the Collaborative

proceed in FY 2020 to include a set of indicators within the behavioral health tool to include the

additional and previously non-reviewed services: Supported Employment and Crisis Respite

Apartments. Furthermore, Intensive Case Management (ICM) will be evaluated, starting in FY 2020,

as a “carve out” or subset service review (similar to the treatment of Assertive Community

Treatment) using a sample of 15 records unique to that service.

Office of Deaf Services Special Reviews

Included within the Collaborative’s review process, DBHDD may request a “Special Review” that does

not fit into the review categories previously defined. In October 2018, DBHDD requested a special

review of records and services for individuals identified as deaf or hard of hearing using both the

provider manual and DBHDD policies8 15-111, 15-112, and 15-114 as references for review tool

8 DBHDD policies can be viewed in their entirety via DBHDD PolicyStat website, https://gadbhdd.policystat.com/.

https://www.georgiacollaborative.com/reports/utilization-management/

https://gadbhdd.policystat.com/


56 | P a g e

development. The purpose of these special reviews was to gather information regarding the

provision of Non-Intensive Outpatient (NIOP) services to deaf and hard of hearing individuals. As the

purpose was to gather data, no scores or final report was provided and no exit conference was

conducted with the provider. Additionally, the results of these special reviews were not posted to the

Collaborative’s public website. Results of the special review were provided to DBHDD.

Sixteen Tier 1 providers were selected to be included in the special review. A separate notice specific

to this review was sent to providers via email 24 hours prior to the start of the review. When

applicable, the special review occurred in conjunction with a regularly-scheduled BHQR or CSUQR to

reduce administrative burden for the provider.

The DBHDD Office of Deaf Services (ODS) special review consisted of a review of individual records,

review of provider policy and procedures related to services for individuals identified as deaf or hard

of hearing, and site visits for each outpatient site. The results were informative in that results of the

questions (compiled across all 16 agencies and 69 individual records) were low for all record review

questions and agency-level programmatic questions as illustrated in Table 11.

While reviewing records, it was apparent that providers had not been consistently informing ODS

when an individual who is deaf or hard of hearing presented for services. None of the medical

records reviewed had a copy of the Communication Assessment Report filed; however, a handful of

IRPs contained information regarding the individual’s communication preference.

Table 11. Office of Deaf Service Special Review Results

ODS Agency Programmatic Review Results (N = 16)

The DBHDD Office of Deaf Service’s (ODS) American Sign Language Poster prominently displayed in all community behavioral health provider’s lobbies or waiting rooms (NIOP sites only).

81% (n = 16)

ODS Record Review Results (N = 69)

Provider notified ODS w/in two business days of first contact w/individual served (for individuals who entered into services after March 2017).

21% (n = 33)

Communication Assessment Report (CAR) is in the medical record. 0% (n = 60)

If individual indicated a change in preference to utilize (or not) ASL-fluent services, provider contacted ODS.

NA

Communication Assessment Report is addressed on IRP to include individual's preferred mode of communication.

3% (n = 60)


57 | P a g e

As a direct result of findings, the Collaborative, with DBHDD’s approval, has crafted a set of

indicators to collect and report data regarding services for individuals who are deaf or hard of

hearing to be added to the Overall Programmatic and Service Guidelines tools for use in scoring

BHQRs.

BHQR Programmatic (Non-scored)

In FY 2019, a category entitled “Programmatic” was piloted as part of the quality review process as a

non-scored element. This was as a direct result of commonly asked indicators across various services

that vary based on the standards set specifically for the service (i.e. policies and procedures being

present, staff caseloads, and credentialing requirements). This Programmatic section encompasses a

separate series of indicator(s) answered once per BHQR and based on the service(s) reviewed and

provided by a provider. By implementing indicators reviewed only once per service rather than per

record, Quality Management avoids possible redundancy in deductions or increased inflation in

Service Guidelines scoring.

Programmatic-level indicators exist for 11 different services and are answered based upon whether a

provider offers that particular service. There is also a set of three indicators answered at each

review for all providers. The results from FY 2019 are as follows, noting in parentheses the number

of reviews (N) and the percentage of met:

All Provider Indicators (N = 157)

Registered Nurse (RN) is present 10 hours per week (91%)

Staff safety and protection policies and procedures are present (96%)

Where applicable, all services provided at approved Medicaid sites (94%)

Addictive Disease Support Services, ADSS (N = 56)

• Individual to staff caseload ratio is met (98%)

Case Management, CM (N = 80)

• Oversight of Case Manager (CM) is provided by an independently licensed practitioner (99%)

Notification of Right to Free American Sign Language Services and Accommodations form is in the individual’s medical record (for individuals who have been in contact with the provider since August 24, 2018).

8% (n = 24)


58 | P a g e

Community Support Team, CST (N = 11)

• CST has all required staff (64%) • CST individual to staff ratio is within parameters (100%)

Peer Support Whole Health & Wellness Individual (N = 8)

• Minimum monthly team meetings are documented and include the Certified Peer Specialist

(CPS), Certified Peer Specialists in Whole Health (CPS-WH) and Registered Nurses (RN) (38%)

• The CPS-to-individual ratio is not exceeded (100%)

• The CPS-WH is supervised by a licensed practitioner (100%)

• The service is provided by a RN or CPS-WH (100%)

Intensive Case Management, ICM (N = 17)

• Minimum staff requirements are met (47%)

• Oversight of ICM is provided by an independently licensed practitioner (94%)

• Staff-to-individual ratios for ICM are met (75%)

• The organization has policies that govern the provision of services in natural settings… (75%)

Intensive Family Intervention, IFI (N = 32)

• All team members are dedicated to this IFI organization and one specific team (88%)

• No more than 50% of staff are "contracted"/1099 team members (78%)

• The IFI team's family-to-staff ratio has not been exceeded (100%)

• The organization has policies that govern the provision of services in natural settings (94%)

MH Peer Support Program (N = 36)

• At least two Georgia CPS' are on staff (92%)

• Individual-to-CPS ratio is not exceeded (100%)

• Individual-to-direct service/program staff ratio is not exceeded (100%)

• Program leader is present at least 75% of the time (94%)

• The CPS Program leader is supervised by an appropriately-credentialed professional (92%)

• The Program is operated the required amount of time (97%)

• The Program Leader is a CPS who is also a Certified Peer Recovery Specialist (CPRP) (66%)

Assertive Community Treatment, ACT (N = 19)

• ACT has policies and procedures governing outreach services and staff safety (79%)

• At least one full-time RN is employed by the ACT team 32-40 hours per week (95%)


59 | P a g e

• Emergency procedures/protocols present including 24/7 availability of staff skilled in crisis

intervention (100%)

• On-call coverage is provided by ACT staff (100%)

• The ACT team maintains a small consumer-to-clinician ratio (78%)

• The psychiatrist participates in weekly team meetings (85%)

Psychosocial Rehabilitation Program (N = 25)

• At least one CPRP is on site face-to-face at all times (88%)

• Individual-to-direct service/program staff ratio is not exceeded (91%).

• The program is operated the required amount of time (96%)

• The program is supervised by an appropriate-licensed practitioner (100%)

While success is evident that several services received a perfect 100 percent “met”, some programs

hold deficits in areas such as employing all required staff for a service or program (CST, MH Peer

Support Program, and ICM) or maintaining the prescribed staff to consumer ratio (ACT). Since the

pilot project proved successful and informative through FY 2019, Quality Management is continuing

to collect this information as non-scored data through FY 2020.

BHQR Focused Outcome Areas (FOA)

The indicators within the Focused Outcome Areas tool are answered once per record reviewed. Each

FOA has a different number of indicators for a total of 21 indicators. Tool revisions implemented in

the first quarter of FY 2019 included adding an additional element of data collection for reasons

within the FOA subcategories for indicators “not met.” Figure 10 displays results of the overall FOA

results for FY 2019 compared to previous years. Additionally, each respective section contains a

figure specific to the subcategory FY 2019 results as compared to previous years and a synopsis of

highlights and trends occurring as a result of FY 2019 reviews.

FY 2017

89%

FY 2018

92%

FY 2019

94%


60 | P a g e

Figure 10. BHQR Focused Outcome Area(s) Scores by Year

Figure 11. BHQR Focused Outcome Area by Year

Whole Health indicators address whether the records reviewed demonstrated individuals were

treated holistically, with their physical health needs being assessed, documented, and monitored. Of

the six FOAs, Whole Health (with results shown in Figure 11 and historically the lowest-scoring) has

surpassed the subcategory of Safety for the second year in a row. Whole Health, for FY 2019, had a

statistically significant (p < .01) increase in the average score of 91 percent compared to the 84

percent result in FY 2018. Important to note is that one indicator was eliminated from FY 2019 data

collection due to redundancy as the area is intentionally addressed in the Assessment and Planning

FY17: 89%(N = 167)

FY18: 92%(N = 156)

FY19: 94%(N = 157)

Whole Health FY17: 74%FY18: 84%FY19: 91%

Safety FY17: 83%FY18: 78%FY19: 80%

Person Centered FY17: 91%FY18: 95%FY19: 94%Community

FY17: 93%FY18: 96%FY19: 97%

Choice FY17: 96%FY18: 97%FY19: 97%

Rights FY17: 93%FY18: 93%FY19: 94%

91%

84%

74%WholeHealth

FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)


61 | P a g e

category: The person has current medical conditions and documentation demonstrates these

conditions are assessed, monitored, and recorded. As the lowest-scoring indicator of Whole Health

(FY 2018, 77%), it is likely that the removal of this indicator from the set generated a slight

improvement in the overall provider FOA scores as it pertains to Whole Health.


Safety indicators address whether providers were attending to certain risk factors for individuals

such as their tendency to experience a crisis, have suicidal or homicidal thoughts or actions, and

receive information and education about the risks and benefits of prescribed medications. Safety

represents the lowest-scoring FY 2019 FOA, yet slightly increased by two points to 80 percent; see

Figure 12.

For FY 2019, the indicator most often scored “No” was “Individual (or their legal guardians)

has been educated on the risk/benefits of all medication prescribed and there is a signed

consent form.” Results of this indicator have trended upward (p < .01) since contract

inception to a result of 73 percent (58% in FY 2017; 68% in FY 2018). Of the 27 percent of

records not meeting this criteria, 44 percent, or 272 of the 622 responses identified the

“form was not present” while 259 records (42%), had a form present but, “lacked at least one

medication prescribed within the last six months” (C&A 85 percent, Adult 72 percent).Results

of documentation as it relates to how providers ensured that a, “Safety/Crisis plan was

developed, as needed, that directs, in advance, the individual's

desires/wishes/plans/objectives in the event of a crisis,” significantly improved (p < .01) from

FY 2018 to FY 2019 with a score of 86 percent. Of those scored not met, thirty percent were

80%

78%

83%

Safety

FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)


62 | P a g e

missing the plan and forty percent did not have the necessary signatures of both the

individual and a clinician (C&A 85 percent, Adult 86 percent).


Person Centered Practices measure whether documentation indicates individuals actively

participated in creating or modifying the plans as needed and desired. While the score of Person

Centered Practices improved in FY 2018, a slight decline of one point occurred in FY 2019. Figure 13

provides results for fiscal years 2017 through 2019. The indicator, “Plan reassessed based upon any

changing needs, circumstances or response by the individual” had the lowest score in the

subcategory for the second year in a row at 88 percent. Only 76 percent of records of individuals 17

and under met this requirement as compared to 89 percent of record of individuals 18 and over (p

< .01). Furthermore, of the 445 records reviewed specific to this indicator, 71 percent or 335 records

did not meet the standard as the plans had not been updated based on the individuals’ current

needs/circumstances. Additionally, documentation demonstrating the individual received

individualized services also declined (p <.01) from 99 percent to 96 percent in FY 2019. This decline

was due to a variety of reasons such as duplicated notes or repeated interventions (n = 55), the plan

did not address identified needs (n = 25), interventions were not meeting needs (n = 22), there were

significant duplicated elements (n = 21), and planned services or service intensity was not meeting

individuals’ needs (n = 20).

94%

95%

91%PersonCenteredPractices

FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)


63 | P a g e


Community Life indicators address how individuals were engaged in their communities of choice and

whether they held valued social roles. Figure 14 graphically presents the annual fiscal year result for

the Community Life FOA. All indicators within Community Life exceeded 90 percent again in FY 2019

with significant improvement noted (p < .01) in “Documentation demonstrates the individual has

age-appropriate responsibilities/valued roles in the community as desired.” Minimal differences

were found in Community Life indicators for C&A versus adult records reviewed.


Choice indicators address how, and if, providers offered individuals options of services and

encouraged individuals to make educated choices concerning supports and services provided. Figure

15 provides results of the Choice FOA by fiscal year. Results at the indicator level remained equal to

FY 2018 results wherein:

If a barrier was identified, documentation demonstrated alternatives were explored (C&A 94

percent, Adult 95 percent).

97%

96%

93%Community

Life

FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)

97%

97%

96%

Choice

FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)


64 | P a g e

The individual’s known preferences and differences were followed (98% both populations)

Documentation demonstrated how the individual was provided with options of supports and

services (98% both populations)


Rights indicators address whether individuals were apprised of their rights while receiving services at

the point of admission to services and on an annual basis thereafter, and whether they had been

informed of their rights under Federal HIPAA laws. There was minimal change from FY 2017 to FY

2018 with an overall subcategory score of 93 percent; however, an additional percentage point

increase occurred in FY 2019, see Figure 16. Nearly all records in FY 2019 (99%) demonstrated,

“HIPAA Privacy and Security Rules were reviewed with the individual,” and 97 percent of records

“had a signed formal acknowledgment of rights and responsibilities at the onset of services,

supports, and treatments.” Attention is still needed specific to “Individuals informed of rights at

least annually,” as only 79 percent of Adult records and 85 percent of C&A records reflected annual

updates; however, this was an increase (p < .01) of six points to the indicator for FY 2019.

BHQR Staff and Individual Interviews

Six FOA categories comprise the Staff and Individual Interviews. Individual Interviews served to

assess an individual’s perception of care with the provider, services rendered, and support in

working toward personal goals. Staff Interviews helped determine whether a person-centered

approach was used in providing services and empowering individuals. Data in Figure 17 represent

the average scores for the individual and staff FY 2019 interviews.

94%

93%

93%

Rights

FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)


65 | P a g e

Figure 17. BHQR Interview Data by Fiscal Year

It is notable that all FOA areas, when reviewed separately, scored in the 90th percentile since contract

inception, indicating a high level of satisfaction for individuals served by the providers. Furthermore,

little difference was identified between scores of similar indicators asked of both individuals and

staff. As a direct result of such positive findings related to the individual and staff interviews over a

four-year period, it has been recommended by the Collaborative (and approved by DBHDD) that staff

interviews be eliminated beginning FY 2020 and to edit and minimize the Individual Interview to a

set of ten questions. This recommended change to the interview processes establishes a new aim of

continuing to assess/identify the individual’s perception of care and satisfaction with their treatment

provider while simultaneously decreasing any administrative burden to the staff and individual

served that the original interview process may have caused.

Assertive Community Treatment (ACT) Quality Reviews Quality Management conducted 20 ACT reviews in FY 2019 specific to 18 unique providers. The ACT

reviews contained 313 records and 3,249 billing claims. Seventeen BHQRs were completed for

providers who offered both Non-Intensive Outpatient (30 records) and ACT services (additional ACT-

specific 15 records). One provider offered only ACT services and the sample was thirty individuals.

FY 2017

96.2%

(N = 735)

FY 2018

97.6%

(N = 616)

FY 2019

97.8%

(N = 572)

INDIVIDUALINTERVIEW

FY 2017

98.1%

(N = 774)

FY 2018

98.9%

(N = 652)

FY 2019

99.0%

(N = 620)

STAFF INTERVIEW


66 | P a g e

Two providers offering ACT services fell below the 90 percent threshold for Overall and

Billing score and required a second review over the course of the year. The two providers

offered both ACT and Non-Intensive Outpatient (NIOP) services. Both were on the once per

year frequency in FY 2018 (Three different ACT providers fell under the threshold in FY

2018).

Eighty-eight percent of ACT records reviewed documented a mental health diagnosis ( the

same percentage as FY 2018). Overall, the ACT records documented the possibility of co-

occurrence of other conditions as well within the population sampled:

o Fourty-four percent identified a substance use diagnosis

o Fourty-four percent identified a physical diagnosis

o Four percent of records confirmed an IDD diagnosis

ACT Quality Review Overall Scores

Figure 18 provides ACT statewide averages, by category, for the past three years of quality reviews.

The Overall score of 89 percent in FY 2019 was the same as in FY 2018.


67 | P a g e

Figure 18. Assertive Community Treatment (ACT) Results by Category and Year

OverallFY17: 87%(N = 20)*

FY18: 89% (N = 21)*

FY19: 89%(N = 20)*

Billing Validation

FY17: 90%FY18: 91%FY19: 88%

Focused Outcome

Areas FY17: 90%FY18: 93%FY19: 93%

Assessment &Planning

FY17: 80% FY18: 87%FY19: 90%

Service GuidelinesFY17: 88%FY18: 84%FY19: 84%

*ACT only providers: FY17: 2; FY18: 1; FY19: 1


68 | P a g e

Table 12. BHQR and ACT Results by Category by Year

Billing Validation FOA

Assessment & Planning

Service Guidelines Overall

BHQR FY 2017 84% 89% 77% 88% 84%

BHQR FY 2018 85% 92% 84% 90% 88%

BHQR FY 2019 86% 94% 88% 90% 90%

ACT FY 2017 90% 90% 80% 88% 87%

ACT FY 2018 91% 93% 87% 84% 89%

ACT FY 2019 88% 93% 90% 84% 89%

Table 129 provides ACT review results by category for FY 2017 through FY 2019 compared to the

BHQRs to demonstrate how ACT quality review results align with trending of all BHQR results. Of

note is that results of the ACT FOAs and Service Guidelines remained static when compared to FY

2018. Furthermore, Assessment and Planning, the lowest-scoring area since inception, has steadily

trended upward since FY 2017 reaching a result of 90 percent, which is comparable to BHQR’s

Assessment and Planning result of 88 percent. Lastly, while Billing results trended upward from

contract inception in BHQR’s averages, ACT results and averages as a subset of BHQRs decreased

from a 91 to 88 percent in FY 2019.

ACT Billing Validation

Figure 19 shows the total dollar amount reviewed through ACT claims analysis, as well as the dollar

amount found to be unjustified and justified across the years. While fewer funds were reviewed in FY

2019 ($318,879.82), the rate of justified amounts has declined from previous years to 88 percent.

9 ACT services are reviewed as a part of the BHQR process. For the purpose of this annual report, ACT scores have been extracted as an additional

subset population only for specific service evaluation, monitoring, and analysis.


69 | P a g e

Figure 19. ACT Billing Validation by Fiscal Year

Specific billing discrepancy reasons found during the ACT reviews are provided in Table 13 for the

3,249 claims reviewed in FY 2019. Assessors select all applicable discrepancy reasons for all reviewed

claims; therefore, one claim may have multiple discrepancy reasons. While the prevalence of

previously identified common discrepancies continued to decline, other reasons for discrepancies

became more pronounced. These include failure to document location of services or “Missing

location” (97 instance in FY 2019 [3.0%]), “Content does not support code billed” (35 instances in FY

2019, [1.1%]) or “Content does not support units billed” (52 instances in FY 2019 [1.6%]) as well as

“Staff credential not supported by documentation” (100 instances [3.1%]). Moreover, several

discrepancies became null for the second year in a row with zero instances identified and include:

Individual receiving services does not meet admission criteria for service billed

Missing/incomplete order

Time in/out missing

$0.00

$50,000.00

$100,000.00

$150,000.00

$200,000.00

$250,000.00

$300,000.00

FY 2017 FY 2018 FY 2019

$307,720.24 90%

$296,038.97 91%

$281,284.86 88%

$32,707.80 10%

$29,806.63 9%

$37,594.96 12%

Amount Justified

Amount Unjustified

Total Amount Reviewed in FY 2017: $340,428.04Total Amount Reviewed in FY 2018: $325,845.60Total Amount Reviewed in FY 2019: $318,879.82


70 | P a g e

Table 13. ACT BHQR Billing Discrepancy Reasons by Year

FY 2017 FY 2018 FY 2019 Percent Change FY17–FY18

Percent Change FY18-FY19

Claims Reviewed 3,221 3,427 3,249

Not meeting admission criteria for service 20 (0.6%) 0 (0.0%) 0 (0.0% -100% 0%

Missing/incomplete order 12 (0.4%) 0 (0.0%) 0 (0.0%) -100% 0%

No valid, verified diagnosis NA NA 3 (0.1%) NA NA

Quantitative Instances Instances Instances Change Change

Staff credential not supported NA NA 100 (3.1%) NA NA

Location missing (out-of-clinic) 32 (1.0%) 32 (1.0%) 97 (3.0%) 0.00% > 100%

Progress note is missing 17 (0.5%) 35 (1.0%) 12 (0.4%) -80.0% -63.1%

Code is missing / different than code billed 19 (0.6%) 13 (0.4%) 6 (0.2%) -33.3% -53.8%

Staff credential missing 40 (1.2%) 15 (0.4%) 5 (0.2%) -66.70% -61.5%

Units billed exceed time / units documented 24 (0.7%) 10 (0.3%) 3 (0.1%) -57.1% -69.2%

Date of service incorrect/missing 3 (0.1%) 1 (<0.1%) 2 (0.1%) -69% > 100%

Date of entry missing 22 (0.7%) 4 (0.1%) 1 (<0.1%) -85.7% -90.0%

Signature missing 25 (0.8%) 4 (0.1%) 1 (<0.1%) -87.5% -74.4%

Time in / time out missing 10 (0.3%) 0 (0.0%) 0 (0.0%) -100% 0%

Performance Standards Instances Instances Instances Change Change

Content does not support units billed 51 (1.6%) 32 (0.9%) 52 (1.6%) -43.8% 77.8%

Content does not support code billed 24 (0.8%) 34 (1.0%) 35 (1.1%) 25.0% 7.7%

Intervention outside staff's scope of practice 0 (0.0%) 42 (1.2%) 31 (1.0%) 100% -21%

Content is not unique to the individual 5 (0.2%) 56 (1.6%) 29 (1.6%) > 100% -44.2%

Non-billable activity 8 (0.2%) 8 (0.2%) 11 (0.3%) 0.0% 69.3%

Multiple services billed at the same time 2 (0.1%) 6 (0.2%) 9 (0.3%) 100% 38.5%

Content does not match service definition 5 (0.2%) 3 (0.1%) 7 (0.2%) -50.0% > 100%

* Due to the low response volume, Quality did not conduct statistical difference in proportions testing on the above table.

The Billing Validation score is assigned to each provider as the percent of justified billed dollars

divided by the total paid dollars for the reviewed claims. The statewide average ACT Billing score was

88 percent in FY 2019. This remained consistent with the FY 2017 and FY 2018 results of provider

scores demonstrating ACT providers had minimally fewer funds identified as unjustified (12%)

compared to all combined services of the BHQR reviews overall (13%).


71 | P a g e

ACT Assessment and Planning

Figure 20 provides the ACT Assessment and Planning score results by specific indicator annually since

FY 2017. Similar to BHQR results, these indicators remain the lowest-scoring in FY 2019:

Co-occurring health conditions addressed on IRP

All assessed needs are addressed on the IRP

Whole health and wellness addressed on IRP


Whole health and wellness improved by eight points, co-occurring condition documentation

increased by 25 points, while both assessed needs and discharge criteria significantly improved (p

< .01) by 26 and 31 points respectively. All other indicators reviewed specific to ACT met or

exceeded 90 percent for a consecutive third year.


72 | P a g e

Figure 20. ACT Assessment and Treatment Planning Scores by Year

ACT Service Guidelines

The ACT Service Guidelines score contained 16 indicators in FY 2019 with the following indicators

added or eliminated from the tool for FY 2019, based on the DBHDD Provider Manual and

discussions with DBHDD clinical leadership:

Indicator added: “Documentation reflects individual is seen face to face at least 8 times per

month (excluding those individuals transitioning to a lower level of care or enrolled in CTP

during the month)” (96%)

99%

99%

97%

97%

94%

91%

46%

76%

51%

49%

100%

100%

97%

96%

97%

89%

66%

86%

68%

70%

100%

99%

97%

95%

94%

90%

87%

84%

77%

74%

Individual meets admission criteria and the diagnosisis verified at least annually.

Current medical screening is present

Interventions/objectives are goal-linked & service-consistent

IRP is individualized in personalized language

Goals/objectives honor hopes, choice, preferences,outcomes

Current behavioral health assessment


Whole health & wellness in IRP

All assessed needs are addressed

Co-occurring health conditions addressed in IRP

FY 2017 (N = 20) FY 2018 (N = 21) FY 2019 (N = 20)


73 | P a g e

Indicator added: “Following admission to a psychiatric facility, the ACT Team is involved in

each individual’s discharge planning” (88%)

Indicator added: “If provided in conjunction with SAIOP, documentation includes evidence

that, without SAIOP, the individual is unable to maintain sobriety and there is clinical

coordination/ planning to avoid duplication of service interventions” (89%)

Indicator eliminated: “There is documentation of individual's responses and a discussion to

the agreement of services identified in the treatment planning/individual recovery planning”

Table 14 shows the item-level detail to illustrate the percent change across similar indicators and

compare fiscal year results within the ACT Service Guidelines. The descending ranked order of

indicators generated the same exact ranked order for similar indicators from FY 2018 to FY 2019.

Furthermore, the ACT Service Guidelines category score decreased significantly (p < .01) from FY

2017 (88%) to FY 2018 (84%) yet remains unchanged from FY 2018 to FY 2019. Yet again in FY 2019,

the same three indicators scored at 100 percent for all three fiscal years:

“Individual meets admission or continuing stay criteria”

“One of the contacts per month addresses the symptom assessment and management of

medications (once a month)”

“For individuals who were discharged, there are multiple documented attempts to locate and

make contact with the individual prior to discharge (over a 45-day period)”

While success is evident with 11 of the 16 indicators exceeding 90 percent, minimal fluctuations did

occur by approximately one percent either positively or negatively for indicators resulting in 90-99

percent. Important to note is that the ACT team with all required staff substantially declined (p < .01)

in FY 2018 by 21 percent and further declined by an additional six points in FY 2019. “ACT team

completes a treatment plan review with the staff, the individual, and his/her family/informal

supports prior to the reauthorization of services” was the lowest-scoring indicator for the second

consecutive year, with evidence in only 28 percent of records. Furthermore, evidence the “ACT team

is working with informal support/contacts at least two to four times per month prior to

reauthorization” showed improvement (p < .01) by thirteen points to a result of 47 percent.


74 | P a g e

Table 14. ACT Service Guidelines Indicator Scores

FY 2017 FY 2018 FY 2019

Percent Change

FY17-FY18

Percent Change

FY18-FY19

The individual meets admission / continuing stay criteria / diagnostic criteria.

99% 100% 100% 1.0% 0.0%

For discharged individuals, there are multiple documented attempts to locate and make contact with the individual prior to discharge (over a 45-day period).

100% 100% 100% 0.0% 0.0%

One of the contacts per month addresses the symptom assessment and management of medications (1/month).

99.7% 100% 100% 0.3% 0.0%

The progress notes document individual response to the staff intervention provided.

99% 98% 99% -1.0% 1.0%

The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.

97% 96% 96% -1.0% 0.0%

There is documentation of individual’s involvement in transition planning.

90% 96% 95% 6.7% -1.0%

Progress notes contain documentation of the individual’s progress (or lack of) toward specific goals/objectives on the treatment plan.

99% 94% 95% -5.1% 1.1%

The ACT team is working with the individual toward educational or vocational needs, interests, per IRP (once per authorization).

97% 94% 96% -3.1% 2.1%

There is documentation to support when substance use services are needed and are integrated into the treatment plan.

93% 92% 92% -1.1% 0.0%

Following admission to a psychiatric facility, the ACT team is involved in each individual’s discharge planning.

95% 91% 88% -4.2% -3.3%

The ACT team has all required staff. 91% 72% 66% -20.9% -8.3%

There is evidence the ACT team is working with informal support systems/collateral contacts at least 2-4 times per month with/without the individual present to provide support and skills training to assist the individual in his/her recovery.

39% 34% 47% -12.8% 38.2%

The ACT team completes a treatment plan review with the staff, the individual, and his/her family/informal supports prior to the reauthorization of services.

59% 28% 28% -52.5% 0.0%

ACT Service Guidelines Result 88% 84% 84% -4.5% 0.0%


75 | P a g e

Table 14. ACT Service Guidelines Indicator Scores

FY 2017 FY 2018 FY 2019

Percent Change

FY17-FY18

Percent Change

FY18-FY19

(N = 20) (N = 21) (N = 20)

ACT Focused Outcome Areas

Focused Outcome Area (FOA) indicators are answered once per record, with the same FOA indicators

(21) at the BQHR. The FY 2018 Overall score for ACT FOA increased (p < .01) from FY 2017 by three

points (93%) and remained steady at 93 percent in FY 2019. The same FOAs met or exceeded 95

percent in FY 2019 as compared to prior years (see Figure 21). This included the FOAs of Whole

Health, Person Centered Practices, Community, and Choice. A minimal decline in the subcategories

of Safety and Rights occurred in FY 2018 for ACT specific services, yet in FY 2019 results slightly

improved for the two FOAs. Indicators identified as requiring improvement due to low scores across

all ACT reviews in these subcategories include “Annual updates to rights” and “Documentation of

medication education and consent.” In FY 2018, both of these indicators resulted in 70 percent of

ACT records meeting the necessary criteria and while annual rights updates declined to 67 percent,

medication education and consent documentation improved from the previous year to a FY 2019

result of 71 percent. The greatest improvement in the FOA subcategory scores specific to ACT

reviews continues to be Whole Health. This FOA, specific to individuals receiving ACT services,

continued to significantly (p < .01) trend upward for the fourth year. Whole Health significantly (p

< .01) increased from 90 percent in 2017, to 93 percent (p < .01) in FY 2018, and further

demonstrated positive efforts to a FY 2019 result of 96 percent.


76 | P a g e

Figure 21. ACT FOA Scores by Category by Year

Crisis Stabilization Unit Quality Reviews Crisis Stabilization Unit Quality Reviews (CSUQR) began in January of FY 2016. The CSUQR assesses

the provider’s overall practices, quality of service delivery, as well as determine adherence to DBHDD

standards through individual record reviews.10 When possible, Quality Management conducted the

CSUQR and BHQR simultaneously; Crisis Stabilization Unit (CSU) providers received separate CSUQR

scores and final assessment reports.

The CSUQR indicators are based on DBHDD policies and the Provider Manual for Community

Behavioral Health Providers and are organized into three review categories: Individual Record Review

(IRR), Service Guidelines, and FOAs. The provider review score for each category represents the

percent of applicable indicators met or present within that category. The CSUQR Overall score is

10 Please refer to the following link to access a full description of the review process and review tools. http://georgiacollaborative.com/providers/prv-BH.html

FY17: 90%(N = 20)

FY18: 93%(N = 21)

FY19: 93%(N = 20)

Whole Health FY17: 90%FY18: 95%FY19: 96%

Safety FY17: 82%FY18: 80%FY19: 81%

Person Centered FY17: 91%FY18: 96%FY19: 95%Community

FY17: 94%FY18: 97%FY19: 96%

Choice FY17: 97%FY18: 97%FY19: 96%

Rights FY17: 87%FY18: 86%FY19: 88%

http://georgiacollaborative.com/providers/prv-BH.html


77 | P a g e

calculated by averaging the three categories, with each category accounting for 33.3 percent of the

Overall score.

Quality Management conducted reviews at 19 eligible CSU providers with three of the 19 providers

requiring a second review during the measurement year as a result of not meeting the threshold (90

percent Overall score). Results for the reassessment reviews were included within the overall results

for a total of 22 reviews. Five of the 19 providers offered services to children and adolescents (C&A)

(age[s] 17 and under) as well as adults; the remaining CSU providers served adults only.

Crisis Stabilization Unit Sampling Method

A random sample of 15 individuals who had received services within the three months preceding the

review was selected for each record review. However, when providers did not have an adequate

number of individuals served in the three months (a minimum of three days’/nights admission)

preceding the review, samples were pulled from individuals served up to six months preceding the

review. In FY 2019, all CSUQRs had the maximum 15 records reviewed per review. While

stratification of CSUQR samples does not occur to account for specific diagnoses type or other

demographics, the information is collected throughout the year with findings reported here within

the annual report. Figure 22 demonstrates the co-occurrences of diagnoses identified within the

sample for FY2019 compared to co-occurrences of diagnoses identified within BHQRs, as significant

differences (p < .01) have been identified. Some of these differences (p < .01) included:

Mental health diagnoses were most prevalent in 85 percent of the BHQR sample, whereas a

mental health diagnosis was evident in 80 percent of CSUQR individuals.

Substance use diagnoses were nearly double in CSUQRs (52%) than in BHQRs (29%).

Important to note: Individuals are often admitted to a CSU for detoxification purposes.

Co-occurring physical diagnoses identified were evident in 41 percent of BHQR records

compared to 17 percent of CSU records sampled.

Diagnoses related to IDD were more evident in the BHQR sample than the CSUQR sample,

with only 2% contained in the CSUQR.


78 | P a g e

Figure 22. FY 2019 Diagnostic Categories Observed by Review Type

Also noted throughout the CSUQR section of this report are differences noted between adults (ages

18+) and C&A. CSUs serving both populations of adults and C&A have an attempted sample of up to

ten adults and five C&A selected. FY 2019 CSUQRs consisted of 330 individual records, 316 adult

records, and only 14 C&A records. The limited number of C&A records sampled was likely due to

limited individuals served respective to the population as only five CSUs provide services to the C&A

population. Additional reasons for the limited number of C&A records included minimal claims

billed; use of oversample needed for reasons such as individuals’ length of stay was less than three

days, or for other reasons established during sample development or at the direction of DBHDD.

The CSUQR mirrors the BHQR scoring and sampling processes for the Individual Interviews and Staff

Interviews (selected by the provider and Quality Assessors). Results from the interviews were not

included in calculating the provider’s overall CSUQR score. Quality Assessors attempt at each review

to complete a minimum of five Individual and five Staff Interviews per CSUQR; however, the actual

number fluctuated based on individual and staff availability, their agreement to participate in the

interview process, the number of staff, individual’s stability or acuity level, and the number of

individuals the provider served at the time of the review. Individuals selected for interviews were

currently at the CSU, and the staff selected were providing services on the CSU. If an individual or

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

FY 2019 BHQR

FY 2019 CSUQR

IDD, 7%

IDD, 2%

Physical, 41%

Physical, 17%

Substance Use, 29%

Substance Use, 52%

Mental Health, 85%

Mental Health, 80%

Mental Health Substance Use Physical IDD


79 | P a g e

staff declined an interview, CSU staff selected a different individual or staff, if possible. In FY 2019,

104 interviews of individuals receiving CSU services and 106 staff interviews were conducted.

Crisis Stabilization Unit Quality Review Overall Scores

Figure 23 shows the Overall and category scores for reviews completed the past three years (FY

2017, FY 2018, and FY 2019). While CSUQRs category scores have progressively increased over the

years since contract inception, FY 2019 showed declines in both the Individual Record Review (p

< .01) as well as the Service Guidelines score(s), thereby decreasing the Overall statewide score

result from an 88 percent (FY 2018) to 87 percent (FY 2019). See distributions in Appendix B. The

FOA category continued to trend positively to be the highest amongst all categories reviewed across

all review types (BHQR, ACT, and CSUQR), for the third consecutive year. In FY 2019, the CSUQR

result for FOAs siginificantly improved (p < .01) 93 percent compared to 91 percent in FY 2018.

Figure 23. CSU Overall Scores by Fiscal Year

Overall ScoreFY17: 86%

(N = 23)FY18: 88%

(N = 21)FY19: 87%

(N= 22)

Individual Record Review

FY17: 80%FY18: 83%FY19: 79%

Focused Outcome

Areas

FY17: 91%FY18: 91%FY19: 93%

Service Guidelines

FY17: 87%FY18: 91%FY19: 89%


80 | P a g e

Three CSU providers had a second review during FY 2019 due to Overall scores that were below the

threshold (90 percent). The results of these three specific CSUQR reassessment reviews are

contained within the reassessment review section of this report in Figure 32.

CSUQR Individual Record Review (IRR)

Individual Record Review (IRR) indicators were answered once per record reviewed. The IRR is

comprised of six subcategories: Assessment and Planning, Admission and Initial Evaluation, CSU

Course of Stay, Documentation, Transition/Discharge Summary, and CSU Planning. Each of the six

subcategories had a unique number of indicators, with 32 total scored within the IRR category.

Additionally, there is a non-scored subcategory that includes four important elemental standards

used for data collection and reporting. In FY 2019, significant change (p < .01) was identified for all

subcategories except Documentation which did in fact increase by one point. Figure 24 identifies the

subcategories and results for each by fiscal year.

Figure 24. Individual Record Review Category Scores

* The IRR subcategories were individually scored and are not averaged for the final overall IRR score. The final overall IRR score is calculated by adding all “yes” or “present” responses of all IRR indicators and dividing by the total “yes” or “no” responses combined for each record review.

86%

89%

69%

85%

72%

62%

85%

85%

78%

88%

74%

92%

90%

86%

84%

82%

70%

56%

Crisis StabilizationCourse of Stay

Documentation

Admission/Initial Evaluation/Screening for Risk

Assessment/Treatment Planning

Crisis StabilizationSpecific Treatment Planning

Transition/Discharge Planning

FY 2017 (N = 23) FY 2018 (N = 21) FY 2019 (N = 22)


81 | P a g e

The overall IRR statewide average increased over the past two years to 83 percent in FY 2018, and

yet declined for FY 2019 to the baseline measurement year (FY 2016) of 79 percent. Several

indicators significantly (p < .01) declined within each subcategory regardless of sampled age

grouping (C&A versus Adult) lending to additional room for improvement, highlighted in the section

below. Moreover, Transition/Discharge Planning resulted in significant declines (p < .01) from FY

2018 to FY 2019 as a direct result of FY 2019 indicator changes, as seen in Table 15. Differences

between the populations sampled (Adult and C&A) are included with the corresponding indicator,

although statistical testing could not differentiate between significant changes in populations due to

the low volume of C&A records sampled. Differences were reviewed at the indicator level for each

indicator between populations and also between changes from the previous year. The lowest-scoring

indicators by each IRR category are listed in Table 15 which includes the following information:

current and previous year result, result of indicator by population (C&A versus Adult), statistical

significance of any at the p < .01 level (highlighted in blue), and the total number of instances

(percentage instances per indicator) per reason the indicator was not met.

Table 15. Lowest-Scoring Individual Record Review Indicators by Category*

Category Name and FY 2019 lowest-scoring indicator(s): FY 2018 FY 2019

Crisis Stabilization Course of Stay 85% 90%

Indicator: Co-occurring disorders as assessed and addressed simultaneously

82% C&A 33%, Adult 83%

79% C&A 33%, Adult 83%

FY 2019 reason(s) not met:

Medical, 21 instances (37%)

MH/Psychiatric, 18 instances (32%)

SU Diagnosis, 16 instances (28%)

IDD diagnosis, 2 instances (4%)

Documentation 85% 86%

Indicator: Community Transition Plan is present if readmitted within 30 days of discharge

19% C&A N/A, Adult 19%


FY 2019 reason(s) not met: • Of the nineteen individual(s) re-admitted within 30 days, 16 did not contain a Community

Transition Plan.

Indicator: Individual's MAR has a legend with all required criteria

65% C&A 68%, Adult 64%

61% C&A 79%, Adult 61%


82 | P a g e




Missing staff credentials, 99 instances (56%)

Missing staff signature, 40 instances (23%)

Missing staff initials, 13 instances (7%)

MAR was illegible, 13 instances (7%)

Means to link signature and initials missing, 8 instances (5%)

Name of medication was omitted without explanation, 4 instances (2%)

Admission/Initial Evaluation/Screening for Risk 78% 84%

Indicator: Comprehensive Nursing Assessment present upon admission

80% C&A 88%, Adult 80%

82% C&A 93%, Adult 82%


Missing fall risk assessment, 39 instances (61%)

Incomplete assessment, 10 instances (16%)

Missing nursing assessment, 6 instances (9%)

Missing flight risk assessment, 5 instances (8%)

Missing Infectious disease assessment, 2 instances (3%)

Nursing assessment was completed after the first 24 hours of admission, 2 instances (3%)

Assessment & Planning 88% 82%

Indicator: Co-occurring health conditions are included in the IRP or NCP

54% C&A 25%, Adult 55%

36% C&A 33%, Adult 36%


Medical, 76 instances (43%)

MH/Psychiatric, 46 instances (26%)

SU Diagnosis, 30 instances (17%)

Missing IRP/NCP, 21 instances (12%)

IDD diagnosis, 4 instances (2%)

Indicator: Discharge plan defines criteria 69% C&A 56%, Adult 71%

48% C&A 29%, Adult 49%


Step-down service missing, 99 instances (44%)

Clinical benchmarks missing, 80 instances (35%)

Missing discharge plan, 48 instances (21%)

Indicator: Interventions/objectives are goal-linked 96% C&A 100%, Adult 96%

90% C&A 100%, Adult 90%


Missing IRP/NCP, 25 instances (78%)

Interventions /objectives not related to goals, 5 instances (16%)


83 | P a g e



Interventions are outside the CSU scope of service, 2 instances (6%)

Crisis Stabilization Specific Treatment Planning 74% 70%

Indicator: IRP/NCP is reviewed following seclusion or restraint

100% C&A 100%, Adult 100%



Lack of a needed update of the IRP/NCP after seclusion/restraint of the three individuals sampled

Discussion of/with the individual in treatment team did not occur in 2 of the 3 individuals

Indicator: IRP/NCP is individualized and in personalized language

82% C&A 84%, Adult 82%

72% C&A 79%, Adult 72%


Individualized and specific to the individual and includes duplicated IRP/NCP, 63 instances (61%)

Written in individual’s language (not devoid of clinical terms), 32 instances (31%) Inclusive of a specific preference of the individual, 8 instances (8%)

Indicator: IRP/NCP incorporates medical updates as indicated

68% C&A 100%, Adult 65%



Urinary tract infection, 6 instances (27%)

Wound care as indicated, 2 instances (9%)

Post-trip to emergency room, 1 instance (5%)

New medical diagnosis after physical on CSU, 1 instance (5%)

Irregular labs, 1 instance (5%)

Updates as appropriate due to medical condition, 11 instances (50%)

Indicator: IRP/NCP addresses safety issues 55% C&A 88%, Adult 51%

43% C&A 58%, Adult 42%

FY 2019 reason(s) not met (missing safety issues):

Falls, 75 instances (41%)

Flight, 50 instances (27%)

Suicide precautions, 36 instances (20%)

Physical aggression risk, 9 instances (5%)

Infection precautions, 8 instances (4%)

Sexual aggression risk, 4 instances (2%)

Indicator: Plan of care discussed every 72 hours 60% C&A 96%, Adult 61%

60% C&A 92%, Adult 58%



84 | P a g e



Treatment team not held every 72 hours, 64 instances (42%)

Documentation of treatment team missing, 50 instances (33%)

Unable to determine which treatment team members were present, 30 instances (20%)

Individual not present/no documentation of refusal, 8 instances (5%)

Transition/Discharge Planning 92% 56%

Indicator: Discharge summary/note documentation includes listed criteria

92% C&A 96%, Adult 92%

89% C&A 92%, Adult 89%

FY 2019 reason(s) not met (missing):

Post-discharge appointment, 26 instances (46%) Living situation at time of discharge, 10 (18%) Specific instructions for ongoing care, including appointment date/time, 7 instances (13%)

Method of discharge, 6 instances (11%) Date, time, and method of discharge, 3 instances (5%)

Discharge summary incomplete or missing, 3 instances (5%) Specific instructions, 1 instance (2%)

Indicator: A discharge summary was entered into the Collaborative’s ProviderConnect/batch system

N/A 24% C&A 43%, Adult 23%


See below for more details

Indicator: If individual was discharged to a homeless shelter, documentation reflects alternatives were explored

N/A 36% C&A 100%, Adult 35%


See below for more details *C&A and Adult data was not tested in FY 2019 for significance due to low C&A volume. Areas highlighted in blue were identified as having a significant difference in year comparisons.

Specific to the Transition/Discharge Planning subcategory, two new indicators were added in FY

2019. These indicators, although new to the CSUQR review tool for FY 2019, were not new standards

set forth by DBHDD but have been standards within the DBHDD Provider Manual for several years.

The questions were added to address consistent deficits identified throughout the past three years

via additional observations during quality reviews. Specific details for each new indicator are as

follows:


85 | P a g e

A discharge summary was entered into the Collaborative’s ProviderConnect/batch

system (24%)

o One CSU provider had entered all discharge summaries

o FY 2019 reason(s) not met:

No discharge summary found in record at the time of review (65%)

Discharge summary submitted after 48 hours (35%)

If an individual was discharged to a homeless shelter, documentation reflects

alternatives were explored

o Forty-four individuals were discharged to a homeless shelter from the CSU

(within the records reviewed)

o Of these, 16 records documented alternatives were explored with individuals

within the medical record (36% met)

CSUQR Individual Record Review (IRR) FY 2019 Highlights

Albeit several indicators within various IRR subcategories declined, several records reviewed met

indicators at or above 90 percent. These indicators are outlined in Table 16 by IRR category with

statistical differences (p < .01) from the previous year shaded in blue.


Service Name and FY 2019 highest-scoring indicator(s): FY 2018 FY 2019

Course of Stay 85% 90%

Individuals participated in training/therapy 86% 90%

Individual is offered groups as needed 88% 95%

Documentation 85% 86%

Date of transfer to a transitional bed documented by physician/physician extender

100% 100%

Progress notes document individual response to interventions provided

94% 96%

Medical progress notes document progress toward goals/objectives on IRP/NCP

90% 92%

RN documents status of individual at least once per day 93% 92%

Staff interventions in progress notes are related to interventions on IRP/NCP.

94% 91%


86 | P a g e


Service Name and FY 2019 highest-scoring indicator(s): FY 2018 FY 2019

Admission/Initial Evaluation 78% 84%

*No indicators met this criteria (>90%) Null Null

Assessment and Planning 88% 82%

Individual meets admission criteria for CSU 100% 100%

Interventions/objectives are goal linked 96% 90%

Crisis Stabilization Treatment Planning 74% 70%

Individual evaluated by physician/physician extender within 24 hours

99% 99%

Transition/Discharge Planning 92% 56%

*No indicators met this criteria (> 90%) Null Null

While the subcategory of Documentation showed minimal change from FY 2018 to FY 2019 (with a

one-point increase for a FY 2019 result of 86 percent), several indicators demonstrated progress for

the second year in a row. These indicators remained at 90 percent or above and were related to

progress notes documenting the individual’s response to interventions (96%), status of individual

documentation by a registered nurse at least once per day (92%), and that staff interventions were

related to interventions on IRP/NCP (91%).

Lastly, four indicators were collected as non-scored elements of the IRR.

One hundred percent of one-on-one observation occurred, when required, while the

individual was in seclusion or restraint/experiencing severe withdrawal/detox or psychiatric

symptoms/behaviors that could cause harm to self or others (n = 124)

Individuals were provided with a reconciled list of medications upon discharge in 98 percent

of records reviewed (n = 315)

Individuals had vital signs documented every eight hours in 72 percent of records reviewed (n

= 308)

o Important to note is that although no significant difference could be found between

the C&A population as compared to the Adult population, only 57 percent of C&A


87 | P a g e

records (n = 14) compared to 73 percent of adult records (n = 301) had vital signs

documented as required.

Only 39 percent of the records contained evidence of follow-up and continuing care (n = 319)

o Increased from FY 2018 (27%)

o C&A population (77%, n = 13); adult population (38%, n = 306)

While the overall IRR category decreased from FY 2018 to the baseline year (FY 2016) result of 79

percent in FY 2019, there are areas of significant improvement such as those outlined previously in

the categories of Admission/Initial Evaluation, Course of Stay, as well as the non-scored indicators

section across the CSU network. It is clear, however, that deficits or the introduction of new

indicators such as those within the Transition/Discharge planning section likely require additional

monitoring or tailoring of the elements within the tools as well as ongoing education and technical

assistance by the Collaborative.

CSUQR Service Guidelines

The 16 CSU Service Guidelines indicators (14 scored and 2 non-scored) were answered once per

review to assess the CSU program, see Figure 25 and Figure 26 where “n” represents the number of

reviews evaluated.


88 | P a g e

Figure 25. CSUQR Service Guidelines Staffing Adherence Indicators by Fiscal Year

Adult CSU staffing requirements were met in 95 percent of the reviews conducted. Of the CSUs

serving children and adolescents, all met the staffing requirements and ratios for FY 2018 and FY

2019. An indicator was added to the FY 2017 review process specific to the “Availability of a

psychiatrist for consultation in cases in which the CSU physician(s) is/are not specialized in

psychiatry” has scored 100 percent since inception. In FY 2019, while more than half of individuals

sampled had a substance use diagnoses identified (52%), having access to an addictionologist was

the lowest-scoring indicator of Service Guidelines (82%) demonstrating room for improvement.

Further analysis and ongoing CSU education and technical assistance will continue through FY 2020.

100%

100%

83%

100%

100%

100%

96%

96%

100%

100%

100%

100%

100%

100%

95%

86%

100%

100%

100%

100%

100%

100%

95%

82%

C&A Minimum Staff Present(FY17 n = 5, FY18 n = 5, FY19 n = 4)

C&A Nursing Staff Ratio(FY17 n = 5, FY18 n = 6, FY19 n = 4)

C&A Psychiatrist (Non-scored)(FY17 n = 5, FY18 n = 5, FY19 n = 4)

C&A Staff Ratio Met(FY17 n = 5, FY18 n = 5, FY19 n = 4)

Physician Availability for 3.7Withdrawal Management

(FY17 n = 22, FY18 n = 21, FY19 n =22)

Psychiatrist Available for Consultation(FY17 n = 18, FY18 n = 21, FY19 n = 22)

Adult CSU Staffing Requirements Met(FY17 n = 23, FY18 n = 21, FY19 n = 22)

Access to Addictionologist(FY17 n = 23, FY18 n = 21, FY19 n = 22)

FY 2017 (N = 23) FY 2018 (N = 21) FY 2019 (N = 22)


89 | P a g e

Figure 26. CSUQR Service Guidelines Policy Adherence Indicators by Fiscal Year

While the majority of policies reviewed improved from the previous year(see Figure 26), both

“Medication notification policy adherence” and “Model/Curriculum for substance use treatment

(non-scored)” continued to excel to reach a perfect score of 100 percent across all FY 2019 reviews.

Policy adherence related to Infection control plan (86%), handling licit and illicit drugs (95%), as well

as seclusion and restraint (86%) all remained quite static when compared to the previous year (FY

2018).

83%

91%

87%

87%

91%

83%

83%

86%

90%

95%

86%

85%

90%

86%

100%

100%

95%

86%

86%

77%

59%

Adherence to Medication Notification Policy(FY17 n = 23, FY18 n = 21, FY19 n = 22)

Model/Curriculum for SU treatment (Non-scored)

(FY17 n = 22, FY18 n = 21, FY19 n = 22)

Protocols for Handling Licit and Illicit Drugs(FY17 n = 23, FY18 n = 21, FY19 n = 22)

Infection Control Plan Adherence(FY17 n = 23, FY18 n = 21, FY19 n = 22)

Seclusion & Restraint Policy Adherence(FY17 n = 23, FY18 n = 20, FY19 n = 22)

Therapeutic Blood Level Monitoring(FY17 n = 23, FY18 n = 21, FY19 n = 22)

Adherence to Safe Storage of Medication Policy(FY17 n = 23, FY18 n = 21, FY19 n = 22)

FY 2017 (N = 23) FY 2018 (N = 21) FY 2019 (N = 22)


90 | P a g e

Policy measurement indicators related to “Adherence to safe storage of medication” and

“Therapeutic blood level monitoring” declined to results of 59 percent and 77 percent, respectively,

with absolutely no changes in measurement or criteria of these two indicators since inception.

However, the additions made to the CSUQR tools in FY 2019 supplied an additional level of detail as

to why the indicator had not been met.

Adherence to safe storage of medications (59%):

o Substances other than medication had been found in the medication refrigerator (n =

5)

o The refrigerator temperature was out of the recommended range (n = 3)

o The documentation reviewed on the temperature log indicated staff did not follow

the policy (n = 1)

Therapeutic blood level monitoring policy adherence standard (77%):

o The blood glucose monitors were not calibrated (n = 5)

o Documentation in reviewed records indicate staff did not follow policy regarding lab

levels (n = 1)

Overall, the statewide average for Service Guidelines increased by four points (91%) in FY 2018 to

decline by three points to a FY 2019 result of 89 percent. From contract inception, scores typically

increased across the CSU providers specific to Service Guidelines with each year, with more than half

of reviews scoring a perfect 100 percent through FY 2018. However, now with the Collaborative in its

fourth year of the contract, scores at the provider level are declining with a median score of 90

percent compared to the previous year’s median score of 100 percent. The Collaborative will

continue to monitor, provide feedback, and provide additional technical assistance and training to

the CSU network specific to the standards and requirements surrounding policy and staffing

adherence.

CSU Focused Outcome Areas

Focused Outcome Area (FOA) indicators were answered once per record reviewed. Each FOA (Whole

Health, Safety, etc.) had a unique number of indicators for a total of 20 indicators assessed overall.

Please refer to the BHQR FOA section for a definition of the six FOAs.


91 | P a g e

The Overall FOA score for CSU providers increased from contract inception result of 88 percent to 91

percent in FY 2017 and remained at 91 percent through FY 2018. An additional two-point significant

(p < .01) increase for CSUQRs (93%) overall became evident in FY 2019 likely as a direct result of

increases in each of the subcategories with the exception of Rights which remained unchanged at 97

percent. The highest subcategories in FY 2018 were Choice (97%), Rights (97%), and Community

(92%). Although Choice scores increased significantly (p < .01) further to 99 percent in FY 2019,

Rights remained unchanged. The Whole Health FOA (96%), significantly improved (p < .01) from FY

2018, thereby replacing Community as the third highest CSUQR FOA following the completion of the

FY 2019 review process. See Figure 27 for FOA results by year.

Figure 27. CSUQR FOA Results by Fiscal Year*

* The FOAs were individually scored and are not averaged for the final overall FOA result at the review level. The final overall FOA result is calculated by adding all “yes” or “present” responses of all FOAs and dividing by the total “yes” or “no” responses combined for each review.

The following were findings based on each FOA, at the indicator level, for FY 2019:

Whole Health documentation mirrored BHQR results related to this FOA for FY 2019. This

FOA has continually improved across all indicators since contract inception to the highest

result yet (p < .01), 96 percent, compared to last year’s 88 percent. Both indicators related to

external resource communication and external coordination needs significantly increased (p

FY17: 91%(N = 23)

FY18: 91%(N = 21)

FY19: 93%(N = 22)

Whole Health

FY17: 89%FY18: 88%FY19: 96%

Safety FY17: 85%FY18: 82%FY19: 84%

Person Centered FY17: 88%FY18: 88%FY19: 90%

Community FY17: 91%FY18: 92%FY19: 93%

Choice FY17: 98%FY18: 97%FY19: 99%

Rights FY17: 92%FY18: 97%FY19: 97%


92 | P a g e

< .01) for CSUQRs by eight and five points, respectively. Additionally, there was significant

improvement (p < .01) identified in 93 percent of records reviewed in FY 2019 versus FY 2018

(83%) having documented safeguards being utilized for medications known to have a

substantial risk for undesirable effects. Most oftentimes records reviewed not meeting this

requirement did not include the necessary Abnormal Involuntary Movement Scale (AIMS) (16

instances). Moreover, 100 percent of records reviewed pertinent to C&A populations:

o Demonstrated ongoing assessment to determine external referrals (n = 13),

o Documented communication with external resources to obtain results of testing,

treatment, and follow-up (n = 14), and

o Documented safeguards for medications known to have a substantial risk for

undesirable effects (n = 11).

Safety documentation demonstrated improvement, although not significant, by two points

overall for this FOA in FY 2019 resulting in 84 percent.

o Signed medication consent forms documenting education on the risks and benefits of

all medications prescribed declined in FY 2019 to a result of 70 percent. This was

mainly as a direct result of medications not being listed on the consent form (35%),

required signatures (physician, nurse, or individual/guardian) not being included

(33%), or the entire consent form missing from the record (32%). Also, fewer C&A

records had a signed consent as having been educated on the risks & benefits of all

medications prescribed (64%) compared to adult records (70%).

o Thirty-nine records did not contain evidence of a safety/crisis plan developed, as

needed, that directs in advance the individual’s desires/wishes/plans/objectives in

the event of a crisis. However, this has improved by two points to 88 percent from FY

2018. Of the records not meeting criteria to justify the evidence needed, 51 percent

of the records had the crisis/safety plan completely missing while 46 percent may

have had a crisis/safety plan on file but lacked an individual/guardian signature.

o Services offered in an environment that ensures individual safety significantly

improved (p < .01) from FY 2018 from 87 percent to 95 percent. Environmental safety


93 | P a g e

concerns/observations are on a case-by-case basis at all CSU locations. All safety

concerns identified for FY 2019 were from one CSU. Environmental observations

included: lack of security camera in the outside common area, dim lighting due to

missing lightbulbs, furniture in the day rooms was not bolted down, and lack of

individual’s access to bathroom/bedroom. DBHDD was notified of the specific

concerns noted from this review.

Person Centered Practices in FY 2019 continues to depict a deficiency in documentation as it

relates to plans being reassessed based on changing needs of the individual. This has resulted

in a three-point increase to 79 percent. This indicator remains the lowest of the measured

indicators.

o Specific to the C&A population, 100 percent of records documented evidence of

individual/guardian participation in planning of services (n = 14), the

individual/guardian participation in the modification of plan/services (n = 4) and

documentation within C&A records to support the plan is reassessed upon changing

needs (n = 3).

o However, only 62 percent of C&A records demonstrated evidence that the individual

was receiving individualized services as compared to 93 percent of adults. This was

mostly identified within the IRP/NCP by which Assessors found duplication across

individuals of similar IRP/NCP documentation or that some other type of assessed

need was identified elsewhere within that individual’s record, but not documented

within the IRP/NCP.

Community documentation supported a seven-point improvement in the individual and

provider discussion of aftercare placement for discharge during the course of stay (97%).

However, a decline of six points, although not statistically significant, was identified in the

individual’s informed choice driving selection of housing options (89%, n = 190). This was

most often found in cases by which the individual was discharged to a homeless shelter.

o No significant differences were found between the C&A population when compared

to the indicators of the adult population on this FOA (due to low C&A response


94 | P a g e

volume). However, 100 percent of C&A records reviewed for this FOA were met across

all four Community indicators.

Choice reflected all records reviewed (N = 330) had appropriate documentation

demonstrating how the individual is provided with options of supports/services for FY 2019.

This is a significant improvement from the FY 2018 result of 97 percent.

o It is important to note that although not statistically significant (due to low volume of

responses), 100 percent of C&A records reviewed as compared to Adult records met

the criteria for:

• Documentation demonstrates individual’s known preferences and differences

are followed to the extent possible (Adult 99%)

• When barriers are identified, documentation demonstrates that alternatives

are explored (Adult 97%)

Rights was supported by 98 percent of records demonstrating, for the second year in a row,

that “Rights and responsibilities acknowledgement is readily available, well-prepared, and

written in language understandable to individual.” The lowest-scoring indicator from FY 2018,

“Individual/guardian has signed formal acknowledgement of rights and responsibilities at the

onset of services, supports, and treatment” did demonstrate a slight improvement to a FY

2019 result of 95 percent.

The C&A records reviewed (n = 14) demonstrated 100 percent compliance in all areas

of Rights for the second year as analyzed, FY 2019.

CSUQR Staff Interview and Individual Interview

The CSUQR interview indicators were similar to the BHQR interview indicators and divided into the

six FOAs. Interviews were analyzed and a score derived for reporting purposes; however, the score

obtained does not calculate into a provider’s “Overall” score or any other scored category. Individual

Interviews were used to assess the individual’s perception of care with the provider, services

rendered, and support in working toward personal goals. Staff Interviews helped determine if a

person-centered approach was used in providing services and empowering individuals. The data and


95 | P a g e

anecdotal information gathered from the Individual and Staff Interviews supplemented the record

review and enhanced the review process by providing a tailored perspective to illustrate the quality

of care the agency provided using both a quantitative and qualitative approach.

Data in Figure 28 illustrates the results of Staff and Individual Interviews by fiscal year. Individual

Interview scores declined slightly from FY 2018 (95.1%) to FY 2019 (94.6%) while Staff Interview

scores increased minimally by three-tenths of a percentage point to 99.5 percent for FY 2019.

Figure 28. CSUQR Individual and Staff Interview Results by Fiscal Year

Similar to BHQR interview results, all but one Interview FOA scored in the 90th percentile. The

lowest-scoring FOA for Individual Interviews for the fourth consecutive year was Whole Health

(85%). Moreover, it is evident over the course of all BHQRs since contract inception, both individual’s

and staff’s perception of care received, offered, and or coordinated is substantiated at a high level.

Areas of deficits can be and are addressed from interview information collected based on the

supplemental data. However, as results continue to remain high and the process of interview

scheduling, completion, and collection of indicators being somewhat timely and burdensome on the

part of individuals/staff, it has been recommended by DBHDD and Quality Management for FY 2020

that Staff Interviews be eliminated from the review process. Additionally, the collection of Individual

Interview data be reduced to a set of only 10 carefully-crafted questions.

FY 2017

93.0%

(N = 114)

FY 2018

95.1%

(N = 102)

FY 2019

94.6%

(N = 104)

INDIVIDUAL INTERVIEW

FY 2017

98.7%

(N = 115)

FY 2018

99.2%

(N = 105)

FY 2019

99.5%

(N = 106)

STAFF INTERVIEW


96 | P a g e

The recommendation mentioned above eliminates 56 separate Staff Interview indicators and a total

of 47 Individual Interview indicators at a rate of approximately 30 minutes per interview. Staff

Interview elimination would allow for a reallocation of resources for additional ICM records, new

services to be reviewed, and inclusion of the Office of Deaf Services questions to the BHQR. The

reduction of Individual Interviews indicators from 47 to 10 questions also reduces time and burden

while still providing supplemental information and allowing for the individual to provide their

perception and satisfaction with the care and services supplied by the agency.

New Provider Reviews New Provider Review Findings

In FY 2019, 131 unique BH providers were reviewed. Each fiscal year, it is anticipated that all

behavioral health providers have at least one quality review. As stated previously in this report,

exclusions are a possibility and provider eligibility is based on multiple factors including volume of

individuals served, volume of submitted claims, and type of services provided.

Of the 131 providers reviewed in FY 2019, five new providers (never before reviewed in prior fiscal

years by Quality Management) received a BHQR. New providers are reviewed approximately three

times within the first two years. All five providers were classified as “Small” serving less than 50

individuals per year, with 5-10 records reviewed per agency. Four of the five providers were Tier 3

(Specialty) providing Intensive Family Intervention services only. The fifth was a Tier 2 provider who

offered Non-Intensive Outpatient Services (NIOP). Results for each of the five providers’ Overall and

Billing scores are demonstrated in Figure 29 with remaining category scores listed in Table 17.


97 | P a g e

Figure 29. FY 2019 Overall and Billing Scores by New Provider

Overall Score: The average Overall Score for the group of new providers reviewed in FY 2019 was 64

percent; the overall statewide average was 90 percent. See Table 17. Only one of these new

providers met the 90 percent threshold as it relates to Overall Score (Tier 2 provider of NIOP

services). This provider scored a 94 percent and a 93 percent for the Overall score and Billing score,

respectively.

Important to note: Of the 157 reviews completed in FY 2019, three new providers were within the 10

lowest-scoring providers for both Overall and Billing scores. Due to this finding, rankings were

assigned to each category score in FY 2019, where “1” = lowest Overall score and “157” = highest

Overall score. See Table 17 for the New Providers’ scores by category, including the rank if the score

was within the 10 lowest scores for each category across all BHQRs.


Overall Score Billing Assessment &

Planning Service FOA

Provider 1 32%

Lowest Statewide

0% Lowest

Statewide

35% Lowest

Statewide

34% 2nd Lowest Statewide

60% Lowest

Statewide

Provider 2 36%

2nd Lowest Statewide

0% Lowest

Statewide


23% 1ST Lowest Statewide


Provider 3 70% 38% 85% 79% 79%

32%0%

36%

0%

70%

38%

86% 87%94% 93%90% 86%

0%

20%

40%

60%

80%

100%

Overall Score Billing

New Provider 1 New Provider 2 New Provider 3

New Provider 4 New Provider 5 State Average


98 | P a g e


Overall Score Billing Assessment &

Planning Service FOA

7th Lowest Statewide



Provider 4 86% 87% 63%


92% 100%

Provider 5 94%

93% 95% 98% 89%

New Providers 64% 44% 65% 65% 81%

State Average 90% 86% 88% 90% 94%

Billing Score: Of the four categories, Billing was the lowest for the five providers. New providers

averaged a score of 44 percent with a median of 38 percent. The only providers in FY 2019 that

received a zero percent Billing Score for FY 2019 were two new providers. The top three billing

discrepancies replicate the initial baseline year results (FY 2016) of the entire network in which

providers were most often cited for the following;

Progress note is missing

Billing code is missing or different than code billed

Missing/incomplete service order

Assessment and Planning Score: New providers averaged 65 percent for Assessment and Planning.

The highest-scoring indicators were similar to the statewide findings related to, “individual meets

admission criteria with the diagnosis verified at least annually,” (93%) and “current medical

screening present” (100%). Additionally, the lowest-scoring FY 2019 indicators across the new

providers were the same improvement areas identified across the entire network. Results of the

initially reviewed providers for FY 2019 were:

All assessed needs are addressed (43%; Statewide BHQR 77%)

Whole health & wellness in IRP (50%; Statewide BHQR 82%)

Discharge plan defines criteria (47%; Statewide BHQR 76%)


99 | P a g e

Focused Outcome Areas (FOA) Score: The average for the five new providers specific to the Focused

Outcome Areas category was 81 percent for FY 2019; however, one provider did score 100 percent

for every FOA (Tier 3 Intensive Family Intervention specialty services provider). Both Whole Health

and Safety had fewer indicators met than any other FOA; 60 percent and 53 percent, respectively.

The lowest-scoring indicators for this set of providers in FY 2019 across all FOAs are listed below.

Whole Health

Documentation of communication with external referral sources and providers to obtain

results of testing, treatment, and follow up (55%; Statewide BHQR 91%).

There are documented safeguards utilized for medications known to have substantial risk or

undesirable effects (lab work, assessments, AIMS, etc.) (50%; Statewide BHQR 88%).

Safety

Individual/guardian & prescriber has signed consent as having been educated on the risks &

benefits of all medications prescribed (50%; Statewide BHQR 73%).

Safety/Crisis plan developed, as needed, that directs, in advance, the individual's

desires/wishes/plans/objectives in the event of a crisis (57%; Statewide BHQR 86%).

Person Centered Practices

Documentation demonstrates the plan is reassessed based upon any changing needs,

circumstances or response by the individual (63%; Statewide BHQR 88%).

Rights

Documentation reflects that individuals were informed of their rights and responsibilities and

HIPAA (83%; Statewide BHQR 99%).

Choice

Documentation demonstrates individual's known preferences and differences are followed to

the extent possible (77%; Statewide BHQR 98%).


100 | P a g e

Community

Documentation of transition planning (including individual, family, other supports) is evident

throughout service delivery and includes specific objectives to be met prior to discharge or

decrease in intensity of services (78%; Statewide BHQR 94%).

Service Guidelines Score: The FY 2019 result of Service Guidelines for newly reviewed providers was

the same as the Assessment and Planning score of 65 percent. As four of the new providers

reviewed offered Intensive Family Intervention (IFI) services (Providers 1 – 4), Table 18 supplies

results of each indicator and the percent met for these providers. The lowest statewide Service

Guidelines score was 23 percent. This is the same new provider that ranked as second out of 157 as

based on Overall score. Eight indicators scored 50 percent or below with the lowest indicators

related to IFI team and team leader minimum contacts as well as documenting assessed progress

within progress notes. See Table 18 for the complete results of IFI indicators for the four new

providers offering only the IFI service.


Intensive Family Intervention (n = 4) % Met

The individual meets admission / continuing stay criteria / diagnostic criteria. 90%

The progress notes document individual response to the staff intervention provided. 90%

There is evidence that the provider is helping the parents/responsible caregivers increase capacity to care for their children. (Review authorization period.)

85%

The Team Leader is licensed/credentialed or CAC-II or equivalent. 75%

The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.

60%

Safety planning with the family and all parties involved evident in the record from the onset of services.

50%

The IFI team has all required staff. 50%

A team approach is used, as evidenced by more than one person and at least one licensed team member. (Review authorization period.)

45%

Services are a mix of individual/family counseling and skill development according to the needs of the individual/family. (Review authorization period.)

45%

The Team Leader is meeting with families at least 2x/month. (Review authorization period.) 45%


101 | P a g e


Documentation reflects a tapering of services. Answer N/A if not appropriate. 40%

Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the treatment plan.

30%

The team is making at least three (3) contacts a week and at a frequency that is clinically appropriate. (Review authorization period)

25%

Services over six (6) hours are related to a crisis and has supporting documentation signed by the Team Leader.

NA

Reassessment Frequency Reviews

BHQR Reassessment Review Findings

Beginning in FY 2017, the frequency of a BHQR and CSUQR was based on minimum scoring

thresholds for both Overall and Billing scores. Providers scoring above the minimum threshold

received one BHQR/CSUQR per fiscal year; providers scoring below the threshold received two

reviews. Providers scoring below the threshold remain on a frequency schedule of approximately

every six months until two subsequent review scores meet threshold requirements. The six months’

time frame between reviews allow ample time for claims submission and documentation to reflect

any changes made by the provider based on previous review findings.

Based on the analysis each fiscal year, the thresholds increased, per DBHDD direction:

FY 2017 thresholds (baseline year)

o BHQR: 80 percent Overall score or 70 percent Billing score

o CSUQR: 80 percent Overall score

FY 2018 thresholds

o BHQR: 80 percent for both Overall score and Billing score


FY 2019 thresholds

o BHQR: 90 percent for both Overall score and Billing score



102 | P a g e

Fifteen percent of providers required a reassessment in FY 2018 (N = 20) and 22 percent (N = 28)

required a reassessment in FY 2019. Five of these providers were also on the twice per year

frequency schedule in FY 2018. As in FY 2018, most reassessed providers in FY 2019 were Tier 2 (N =

19) and there was an average of 191.1 days between reassessed reviews.

Figure 30 displays the distribution of Overall scores obtained during the first and second reviews in

FY 2019 for the reassessed providers. The average change in Overall score was six points (either

increase or decrease) from the first to second review. The most visible change is in the 81 percent to

90 percent range. For the first review of FY 2019, 43 percent of providers scored in the 81 to 90

percent range, which reduced to 39 percent for the second FY 2019 review.

Figure 30. FY 2019 BHQR Overall Score Distribution of Reassessed Providers

Overall Score: The average result for the reassessed providers at first review of FY 2019 was 89

percent compared to 88 percent later in the year. When compared to FY 2018, more providers were

placed on the reassessment frequency in FY 2019 due to an Overall score below the threshold at the

initial annual review. A third of providers (n = 7) reassessed in FY 2018 were on the twice per year

frequency as a direct result of falling below the FY 2018 Overall score threshold as compared to half

4% 4%

43%50%

7% 4%

39%50%

0%

20%

40%

60%

80%

100%

1st BHQR (N = 28; Mean = 89%; Median = 90%)

2nd BHQR (N = 28; Mean = 88%, Median = 91%)


103 | P a g e

(n = 14) of the reassessed providers in FY 2019. The following are details specific to the FY 2019

reassessed providers and their Overall results:

Eight providers exceeded the 90 percent threshold for both FY 2019 reviews, thus removing

them from the bi-annual frequency.

The greatest increase in Overall score for a provider was a 15-point change from initial to

subsequent review. This provider’s Billing score increased from initial review (50%) to

subsequent review (94%).

Thirty-six percent (N = 10) of providers increased the Overall score upon subsequent review.

Three reassessed providers (11%) had the exact same Overall score for both their initial and

subsequent FY 2019 reviews.

Fifteen providers (54%) had a decline in Overall score from time of initial review to

subsequent FY 2019 review.

The largest decrease in Overall score was 26 percent and the average decrease was 5.7

percent in Overall score for these fifteen providers. Quality Management did not have review

tool changes between the first and second reviews.

Table 19 provides the average result by category for the group of reassessed BHQR providers,

both at the initial review as well as at the subsequent review.

Table 19. FY 2019 Review Results by Category of Reassessed Providers

Overall Score Billing

Assessment & Planning Service FOA

Initial Score Average

89% 87% 84% 89% 93%

Subsequent Score Average

88% 80% 87% 91% 94%

State Average 90% 86% 88% 90% 94%

Billing Score: Unlike FY 2018 where 80 percent of reassessed providers had an increase in their

Billing score, only 43 percent of providers reassessed in FY 2019 saw an increase from their initial


104 | P a g e

review. The subsequent BHQR Billing score fell below the State Average of 86 percent and is the

lowest category score. Overall, the average Billing scores for providers reviewed twice during FY 2019

decreased from 87 percent (at the initial review) to 80 percent subsequently (see Figure 31 for the

distribution scores of reassessed providers). As in previous years, the two most prevalent reasons

claims were unjustified: “Missing or incomplete service orders” and “Content does not support the

units billed.” The most substantial improvement seen in a provider’s subsequent Billing score was a

44 percent increase (initial to subsequent review results from 50 to 94 percent).

Figure 31. FY 2019 BHQR Billing Score Distribution of Reassessed Providers

Assessment and Planning was the lowest category for the reassessed providers in FY 2017 and FY

2018; however, this changed during FY 2019. The initial review result in this category maintained this

pattern with a score of 84 percent when averaged across providers, but with an increase of three

points (87%) by second review. Assessment and Planning was no longer the lowest category score.

Similar to providers who met the threshold requirement for FY 2019 and did not require a

reassessment, the lowest scored indicators were “All assessed needs addressed on the IRP”, “Co-

occurring health conditions addressed in IRP”, “Whole health and wellness within the IRP”, and

“Discharge plan defines criteria.” These indicators were also the lowest-scoring in this category in FY

2018. Of the 61 percent of providers who demonstrated improvement in Assessment and Planning,

0% 4% 0%7%

50%

32%

11%0%

7% 11%

43%29%

0%

20%

40%

60%

80%

100%

1st BHQR (N = 28; Mean = 87%; Median 89%)

2nd BHQR (N = 28; Mean = 80%; Median = 85%)


105 | P a g e

41 percent increased by 10 points or more. One provider had an increase of 29 points, with an

increased Billing score of 66 percent at initial review to 95 percent by year-end. When compared to

the statewide average of 88 percent, 54 percent of providers scored at or above this result during

their first FY 2019 review and then increased to 61 percent at the subsequent FY 2019 review.

Extensive technical assistance was provided during the quality reviews related to Assessment and

Planning. During subsequent reviews, it was found that providers implemented recommended

changes based on their previous BHQR.

Service Guidelines: The BHQR Service Guidelines state average score remained the same at 90

percent for FY 2019. The average score for reassessed providers at initial review was 89 percent, just

slightly below the statewide average. The average score improved to 91 percent at second review;

thereby exceeding the statewide average by one percent. Eighteen services were reviewed

throughout the reassessments. Services not included in these reviews were due to the agency either

not providing these services or claims not available in the date range sampled for either one or both

of the FY 2019 reviews. Reassessments did not include a review of:

AD Peer Support Program

AD Peer Support - Individual

Opioid Maintenance

Peer Support Whole Health and Wellness – Group

Peer Support Whole Health and Wellness – Individual

Most Service Guidelines scores experienced minimal change between the initial and subsequent FY

2019 review with 59 percent of services experiencing an increase or decrease of two percent or less.

A new service reviewed in FY 2019, Substance Abuse Intensive Outpatient Program (SAIOP),

demonstrated an increase of 26 percent from initial to subsequent review for applicable reassessed

providers. This was mostly due to improvement in progress note documentation being reflective of

the individual recovery plan (IRP) as SAIOP was not included on many IRPs in the initial BHQR review.

Improvement was noted within Intensive Family Intervention (IFI) Service Guidelines from the

subsequent FY 2018 review (88%) to the initial FY 2019 review (94%). However, the score reduced by


106 | P a g e

the subsequent FY 2019 review to 91 percent. The most significant reduction was with the indicator

“The Team Leader is meeting with families at least 2x/month” as scoring for this indicator dropped

45 percentage points from 93 percent to 48 percent by the FY 2019 subsequent review. Of note,

changes to scoring methodology or questions reviewed did not change for IFI during FY 2019.

Three providers of Intensive Case Management (ICM) services were reassessed in FY 2019 with the

Service Guidelines score for ICM dropping from 88 percent (initial review) to 86 percent (subsequent

review). Three ICM Service Guidelines indicators decreased by 18 percentage points or more from

initial to subsequent review:

“Documentation consists of interventions that provide environmental supports and care

coordination that promotes recovery as identified in the Individual Recovery Plan (IRP).” This

indicator scored 92 percent for the initial FY 2019 review, and dropped to 70 percent at the

subsequent review.

“Documentation reflects referral and linkage to services and resources identified on the

Individual Recovery Plan (IRP) including housing, social supports, family/natural supports,

entitlements (SSI/SSDI, Food Stamps, VA), income, transportation, etc.” scored 100 percent

for the initial FY 2019 review, but dropped to 80 percent by the subsequent review.

The lowest-scoring indicator within ICM was, “There is a joint development of a crisis plan to

include the provider and individual; the provider is listed as primarily responsible,” scored 50

percent for the initial FY 2019 review and dropped to 32 percent by the subsequent review.

Two Assertive Community Treatment (ACT) providers were reassessed in FY 2019. While there was

no change in the Overall ACT score from initial to subsequent review, two indicators had a more than

15 percentage point difference. “The ACT Team is working with the individual towards educational or

vocational needs, interests, per IRP” increased from 83 percent to 100 percent; while “There is

documented evidence that the ACT Team is working with informal support systems/collateral

contacts at least 2-4 times per month” decreased 22 points from 33 percent to 11 percent.

Focused Outcome Areas: As in FY 2018, providers reviewed for a second time in FY 2019

demonstrated improvement in FOA scores, from 93 percent to 94. Seventeen providers (61%)


107 | P a g e

demonstrated either no change or improvement in this area and 11 providers had a decline in score

at an average of five points. Of all FOAs reviewed during a reassessment in FY 2019, Safety was the

lowest-scoring category both at first review (76%) and at second review (80%). The Safety FOA

continued as an area for growth and improvement from FY 2018 to FY 2019 among not only

reassessed providers but also the entire network.

CSU Reassessment Review Findings

Three of the 19 unique CSU providers had a reassessment during FY 2019 due to either a low Overall

score in FY 2018 (< 90%) or a low score early in FY 2019. Figure 32 provides a graphic representation

of each of the CSU’s progress from the FY 2018 result to the subsequent FY 2019 reassessment

review. One of the three CSU providers (CSU #2) was also on the twice a year frequency in FY 2018.

This CSU provider (CSU #2) was the only provider to consistently improve from review to review

since FY 2018. The other two CSU’s, CSU #1 and CSU #2, ended the 2019 fiscal year having declined

in Overall scores to 75 percent and 78 percent, respectively.

Figure 32. Overall Score Results by Reassessed CSU

87%

78%

84%

75%

87%

87%

79%

82%

88%

85%

72%

96%

50% 60% 70% 80% 90% 100%

Statewide Average

CSU #3

CSU #2

CSU #1

FY 2018 Initial FY 2019 Subsequent FY 2019


108 | P a g e

Conclusion

Eighty-five percent of BHQR reassessed providers experienced an increase in Overall score from

initial to subsequent FY 2018 review, but only 36 percent of providers increased the Overall score

during FY 2019. Additionally, two of the three CSU’s reassessed in FY 2019 did not improve from

initial to subsequent review. Distributions of reassessed providers (based on each scored category)

can be found in Appendix B. Those providers who fell below the established threshold at either of

the two reviews during FY 2019 will be required to have an initial and subsequent review scheduled

for FY 2020 regardless of improvement in one or both categories (Overall and Billing score). Based on

all BHQR and CSUQR data and findings obtained from FY 2019 reviews, it is anticipated that the

threshold for FY 2020 BHQRs is to remain at 90 percent for both Overall and Billing scores and

CSUQRs remain at 90 percent Overall.

Technical Assistance/Exit Conference

While technical assistance occurred throughout the review process via interactions between the

Collaborative and provider staff, one measurable component of the volume of technical assistance

provided was collected from the exit conferences. Upon completion of all BHQRs and CSUQRs, the

lead assessor completed a formal exit conference. The exit conference(s) supplied providers with

tentative scores, provider strengths, and opportunities for growth including quality

recommendations made by the Collaborative. Table 20 provides details on the technical assistance/

exit conferences completed during FY 2019.


Fiscal Year 2019 Quarter 1 Quarter 2 Quarter 3 Quarter 4 Total

Year End

Total BHQR Exits 50 29 45 33 157

Total Minutes 2,766 1,574 2,729 2,008 9,077

Total Attendees 405 254 418 383 1,460

Total CSUQR Exit 2 5 5 10 22

Total Minutes 65 270 315 660 1,310

Total Attendees 12 64 76 215 367


109 | P a g e


Fiscal Year 2019 Quarter 1 Quarter 2 Quarter 3 Quarter 4 Total

Year End

Total Exits 52 34 50 43 179

Through the 179 exit interviews conducted in 2019, technical assistance was provided to 1,827

provider staff members for an approximate total of 173 hours. In FY 2019, the total number of

attendees increased by more than 200 from 2018. The CSUQR average attendance per exit was 17

(FY18 average attendance per exit was 15) and the BHQR average attendance was nine (exactly the

same BHQR average attendee count per exit as in FY 2018). The average exit conference length was

approximately 58 minutes. The length of exit conferences varies by type of exit conference and may

include ACT reviews, which included an additional 15 records per review and details specific to ACT

services discussed and reported. Exit conferences and technical assistance will continue throughout

FY 2020 as through the provider feedback survey, providers identified them as helpful assistance

from the Collaborative.

Summary of Findings and Recommendations for Behavioral Health Providers Quality Management continually reassesses review processes to ensure we are capturing results and

analyzing outcomes that give the best information and truest picture of service quality in Georgia.

This includes reviewing feedback received from both providers and individuals, as well as revision of

review tools as needed to ensure accurate and measurable reporting of results.

June 2019 marked the completion of the fourth year of the Collaborative’s Quality Management

program. Quality Assessors completed 157 BHQRs and 22 CSUQRs in FY 2019 with 179 exit

conferences conducted across all review types. Exit conferences may have occurred jointly across the

BHQR, CSUQR, and ACT reviews depending on whether the provider rendered more than one of

those services.


110 | P a g e

Provider Performance

Table 21. BHQR, ACT, CSUQR Overall Averages by Year

Billing Validation FOA

Assessment/ Planning

Service Guidelines Overall

BHQR FY 2017 84% 89% 77% 88% 84%

BHQR FY 2018 85% 92% 84% 90% 88%

BHQR FY 2019 86% 94% 88% 90% 90%

ACT FY 2017 90% 90% 80% 88% 87%

ACT FY 2018 91% 93% 87% 84% 89%

ACT FY 2019 88% 93% 90% 84% 89%

CSUQR FY 2017 NA 91% IRR – 80% 87% 86%

CSUQR FY 2018 NA 91% IRR – 83% 91% 88%

CSUQR FY 2019 NA 93% IRR – 79% 89% 87%

Table 21 summarizes the Overall averages by fiscal year:

While BHQRs resulted in an average improvement in Overall score from 88 percent to the

threshold of 90 percent, CSUQRs declined in Overall score from the previous year to 87

percent.

ACT remained consistent to the previous year’s results related to FOAs, Service Guidelines,

and Overall. However, ACT services improved in the area of Assessment and Planning (90%)

and declined in Billing validation by three points from the previous year (88%).

Five new providers were reviewed and analyzed in FY 2019 that had never before had a

quality review completed. When grouped together, the average Overall score for these five

providers was 64 percent. Of the five providers, three scored within the top ten lowest

scoring reviews for Overall score, with two of those providers receiving a zero percent for the

Billing category. Interesting to note, the most cited Billing discrepancies in FY 2019 for new

providers were the same as the first year of the BHQRs (FY 2016). Furthermore, new


111 | P a g e

providers’ Whole Health FOA score (60%) was similar to the baseline year (FY 2016) results of

the entire network (63%).

Twenty-eight providers were reviewed for a second time in FY 2019 due to either low scores

in Overall, Billing, or both (< 80% in FY 2018; < 90% in FY 2019). While this resulted in a

greater number of reviews completed for FY 2019, it is still less than the number of additional

reviews conducted in FY 2017 (N = 35). While it is anticipated providers whom are reassessed

are likely to obtain an elevated score when measured on the same indicators, only 36 percent

(N = 10) actually received an increased score by year end.

All category results had statistically significant (p < .01) changes, except Service Guidelines.

The Service Guidelines scores remained static from FY 2018 to FY 2019 for both BHQRs (90%)

and ACT (84%) reviews, but decreased by two percentage points in CSUQRs (89%).

Focused Outcome Areas (FOAs) were introduced for the first time to the Georgia Behavioral Health

network in June 2016. With results continuing to significantly improve (p < .01) and most providers

excelling in meeting the measured standards, Table 22 provides further details and summarizes the

FOAs by subcategory, review type, and year to further support the improvements identified.

Table 22. BHQR and CSUQR FOA Scores

Focused Outcome Areas BHQR

FY 2017 BHQR

FY 2018 BHQR

FY 2019 CSUQR FY 2017

CSUQR FY 2018

CSUQR FY 2019

Whole Health 74% 84% 91% 89% 88% 96%

Safety 83% 78% 80% 85% 82% 84%

Person Centered Practices 91% 95% 94% 88% 88% 90%

Community 93% 96% 97% 94% 92% 93%

Choice 96% 97% 97% 98% 97% 99%

Rights 93% 93% 94% 92% 97% 97%

Overall Score 89% 92% 94% 91% 91% 93%

The Overall score for the FOAs met or exceeded 93 percent in FY 2019 for both BHQR and CSUQRs.

However, five new providers reviewed for the first time in FY 2019 averaged a low score of 60


112 | P a g e

percent for this FOA, a similar finding to the FY 2016 introduction of this FOA result of 63 percent.

Minimal changes were found across the entire network in many of the FOA categories including

Person Centered Practices, Community, Choice and Rights (all of which exceeded 90 percent). Whole

Health generously improved from FY 2018 to FY 2019 for both BHQRs (91%) and CSUQRs (96%)

whereby results increased by seven points and eight points, respectively. Lastly, while Safety

demonstrated improvement in providers’ scores reaching a result of 80 percent and 84 percent, for

BHQRs and CSUQRs respectively, improvement needs are evident. This is especially the case specific

to safety/crisis plans being developed when needed with all necessary elements (BHQR 86%, p < .01;

CSUQR 88%) as well as records containing a signed consent for being educated on medication risks

and benefits when prescribed (BHQR 73%, p < .01; CSUQR 70%).

Specific to reassessed providers who fell below the minimum threshold for Billing and Overall scores,

Table 23 reflects the number of providers who improved from the first FY 2019 review to the second

review within the year. Moreover, while 14 reassessed providers met or exceeded the 90 percent

statewide Overall score at first review, only 12 providers exceeded the 90 percent threshold after the

second review. Providers who obtained low FY 2018 Overall or Billing scores were scheduled for

review and reassessment in FY 2019. While many of the reassessed providers demonstrated

improvements on an individual level from first FY 2019 review to second FY 2019 review, those that

continued to fall below the established thresholds are to be scheduled for an additional two reviews

in FY 2020. Additionally, providers were supplied with additional technical assistance during reviews

to assist with performance improvement.

Table 23. FY 2019 Reassessment Review Results Number (%) of Providers Who Increased in Score

from FY 2019 Review #1 to FY 2019 Review #2

Category BHQR

(N = 28) CSU

(N = 3)

Billing 12 (43%) NA

Individual Record Review NA 2 (67%)

Service Guidelines 17 (61%) 0 (0%)

Focused Outcome Areas 12 (43%) 1 (33%)


113 | P a g e

Table 23. FY 2019 Reassessment Review Results Number (%) of Providers Who Increased in Score

from FY 2019 Review #1 to FY 2019 Review #2

Category BHQR

(N = 28) CSU

(N = 3)

Assessment & Planning 17 (61%) NA

Overall Scores 10 (36%) 1 (33%)

FY 2019 BH Accomplishments

Throughout the year, the Quality Management Department partners with DBHDD Quality

Improvement, DBHDD subject matter experts, and others to discuss and review findings, concerns,

and areas of need across the provider network based on monthly, quarterly, and annual review

results. It is through this partnership the Collaborative derives recommendations and revisions to

the review process.

Below is a listing of accomplishments achieved by the Collaborative that have occurred throughout

FY 2019.

BHQR and CSUQR tools were revised for clarity, alignment, and formalized criteria including:

o Addition of Substance Abuse Intensive Outpatient Program as a measurable service

o A new “non-scored” Programmatic review category was piloted to collect, monitor,

and report various issues to DBHDD specific to programmatic standards of specific

service types.

o Differentiation between Peer Support Whole Health and Wellness - Individual versus

Group services.

o Inclusion of a billing discrepancy “No valid, verified diagnosis on date of service

provided.”

FY 2019 Quality Training Series

o Safe Storage, Handling and Administration of Medication

o Making the Most of Your Crisis Stabilization Unit Quality Review


114 | P a g e

o Making the Most of Your Behavioral Health Quality Review

o Medicaid Assisted Treatment (MAT) Provider Training

o Documentation for Paraprofessionals – A Guiding Light

o Person-Centered Approach to Crisis and Safety Planning

o Recovery and Rehabilitation – The Ultimate Selfie

o CSU Transition/Discharge Planning & Overview of Care Coordination

o Tier 1 & 2 Information Session “BH Provider Standards & KPIs

Successful collaboration with the Collaborative’s IDD department in developing Final and Exit

summary and assessments specific to the review process, while also assisting in alignment of

IDD tool revision changes and needs with categories reviewed within the BHQRs/CSUQRs.

Annual review of all BHQR and CSUQR tools in preparation for the new fiscal year.

Completion of the requested “Office of Deaf Services Specialty Review.”

A forty percent increase in reassessments were completed in FY 2019 as compared to FY

2018, likely as a direct result of the increased threshold for Overall and Billing scores to 90

percent.

Quality Management met all review reporting contractual requirements as set forth by

DBHDD including but not limited to quarterly provider score reporting, semi-annual review

data and score presentation, Medicaid discrepancy reporting, as well as any ad hoc request

made by DBHDD.

Quality Management conducted various ad hoc reporting and data analysis which was

provided to its client either as a direct result of previous year’s analysis or at the request of

DBHDD and included:

o Sample size Power Analysis (BHQR/CSUQR/ACT/ICM)

o Supported Employment Study (IDD specific but included BH collaboration)

o Intensive Case Management Analysis

o Individual/Staff Interview Analysis

o Yale RSA-R Report

o Intensive Family Intervention Review/Claim Analysis


115 | P a g e

o Office of Deaf Services Review pilot

o Suicide Study proposal and sampling methodology

BH Systems Strengths and Recommendations for Improvement

In FY 2019, both BHQR and CSUQR scores fluctuated either positively or negatively dependent upon

review and category type. While BHQRs Overall score increased to 90 percent, CSUQR overall scores

declined by one point to 87 percent. Differences (p < .01) were found in each of the review type

categories following scoring as based on statewide average results compared to the previous year,

except for Service Guidelines. Thus, significant differences were further evaluated at the indicator

level to aid in deciphering strengths and improvement needs. These included the following:

The BHQR Assessment and Planning category resulted in significant improvements related to

the three indicators with the lowest scores in the past, co-occurring health conditions being

addressed/included in the IRP/NCP (FY18 60%; FY19: 70%), discharge plan defining criteria

(FY18 66%; FY19 76%), and interventions/objectives being goal-linked and service-consistent

(FY18 92%; FY19 96%). The exact opposite was found in the case of CSUQRs by which the

same exact indicators declined (p < .01) to a result of 36 percent (FY18 54%), 48 percent

(FY18 69%), and 90 percent (FY18 96%).

While ten of the 18 services reviewed with the BHQR Service Guidelines demonstrated

improvement from FY 2018 to FY 2019, Nursing continuously progressed (p < .01) year to

year since FY 2017. Nursing documentation reflected education to individuals related to

identified health issues, individualized goals, individuals’ progress (or lack thereof), and

response to treatment/interventions.

For the second year in a row, of the three BHQR FOA Safety indicators, the one most often

scored “no” was “Individuals (or their legal guardians) signed medication consent forms along

with the prescriber”. However, results of this indicator have continued to trend upward yet

again by five points to a result of 73 percent for FY 2019.

The BHQR Choice FOA indicators all exceeded 90 percent in FY 2019, so too was the case for

the FOA, Community Life. However, while four of the five Community Life indicators declined


116 | P a g e

by one point, documentation across the network with the individual having age-appropriate

responsibilities/valued roles in the community improved significantly (p < .01) to 98 percent.

Specific to the CSUQR IRR, only Admission/Initial Evaluation/Initial Screening and Course of

Stay had categorical improvement. Course of Stay (90%) had a six percent (p < .01) increase

for FY 2019 compared to FY 2018 with 90 percent of records having documented the

individual’s participation in training/therapy with even more records demonstrating the

individual being offered groups as needed (95%, p < .01). One hundred percent of C&A CSU

records (ages 17 and under) had clear documents to support the individual participating in

training/therapy as well as being offered groups as needed.

Although part of a non-scored element within CSUQR IRR category, follow-up and connection

to continuing care statistically decreased (p < .01) by 12 points to a low result of 39 percent.

Moreover, documentation of vital signs (non-scored indicator) improved (p < .01) from 62

percent in FY 2018 to 72 percent in FY 2019. Fifty-seven percent of records for individuals

aged 17 or under met this standard compared to those aged 18 and over (73%).

While no significant improvements were identified specifically in CSUQR Service Guidelines,

several measured standards or indicators related to staff access, staff ratios, and staff

availability were met at 100 percent.

All CSUQR Whole Health FOA indicators improved significantly; demonstrating documented

safeguards for utilized medications known to have substantial risks or side effects (93%),

ongoing assessment to determine external referral needs (99%), as well as communication

with external referral sources and providers as needed (99%).

While several providers remained consistent or demonstrated improvements in their results from FY

2018 to FY 2019, indicator level declines provide for opportunities for improvement, especially in the

case of targeted BHQR services as well as CSUQR categories. Several targeted training efforts have

occurred over FY 2019 to address low scoring, downward-trending indicators and or service specific

trainings resulting from BHQR or CSUQRs scoring as well as via interactive technical assistance

interactions that occurred throughout the year with providers directly. Additionally, Quality


117 | P a g e

Management has devised several recommendations identified below based on results supplied

throughout the FY 2019 annual report.

Recommendation I: Due to fifty-four percent of reassessed providers scoring lower on the

subsequent review in FY 2019, a recommendation is being made to adhere to the existing threshold

requirements to remain at 90 percent for Overall and Billing scores at the provider level.

Recommendation II: Throughout the year, Quality Management monitors various provider findings,

specific service details, and indicator level review results to evaluate continuous upward/downward

trends. Additionally, provider reassessment (providers reviewed for a second time for not meeting

the annual threshold) data and findings are analyzed and reported. With these findings the

Collaborative has identified and recommended changes to both the BHQR and CSUQR tools to

further supplement the quality review process and reporting. Therefore, an added section entitled

“Quality Risk Items” is being recommended for FY 2020 to include an Overall score point reduction (2

percentage points) for each identified item. Quality Risk Items may result in a provider review

reduction with up to a 10-point maximum reduction for items that continue to be recommended at

each review for the specific provider it applies. Reductions also may be applied in cases of high-level

concerns including, but not limited to:

Columbia Suicide Severity Rating Score not in three or more records

Duplicated documents in three or more records

Provider lacks proof of background check on employee, staff, or contractor

Required staffing is incomplete for more than 90 days in a program

Repeat quality improvement recommendations on subsequent reviews (due to low

reassessment scores of similar indicators)

For a complete listing of Quality Risk items for FY 2020, access the BHQR/CSUQR Tool – Quality Risk

Items at the Collaborative’s website.

Recommendation III: The Collaborative recommends the introduction of additional reasons that

claims are not justified. These additional reasons would better align the Collaborative’s scoring

policies with other entities such as Department of Community Health. These additional discrepancy

https://www.georgiacollaborative.com/providers/behavioral-health-providers/


118 | P a g e

reasons are also intended to increase the quality of documentation within individuals’ records by

asking providers to align their documentation more fully with existing standards. These discrepancy

areas include:

High utilization without clinical justification

Intervention unrelated to IRP w/o clinical justification

Monthly contacts not met per DBHDD Service Guidelines

No overall progress documented

Note does not include response to intervention

Credentials not submitted within required timeframe

Progress note not filed within seven calendar days

Recommendation IV: Continue annual review and evaluation of existing BHQR and CSUQR tools and

requirements, especially for recently added indicators to align with current state requirements and

DBHDD recommendations in accordance with the DBHDD Provider Manual.

Recommendation V: Based on analysis of FY 2019 review and reassessment data, continue the

Quality Training Series in FY 2020 and participate in the 2020 Behavioral Health Symposium, for all

BH and CSU agencies to include, but not limited to, the following:

Partnering with Individuals for Effective Recovery Planning

Behavioral Health Assessment: It’s More Than You May Think

Professional Self-Care and Boundaries

New Provider Trainings

o Overview of Collaborative/Intro to Quality Management (QM) /DBHDD

Provider Manual

o Clinical Care Coordination/QM

Targeted training based on service, category, or QM recommendations:

o Billing and Documentation

o Medication Assisted Treatment (MAT) Provider Training

o Certified Peer Specialist (CPS) Documentation Training

Recommendation VI: In FY 2019, DBHDD released its FY 2017 BH Suicide Mortality Report. Through

review of this document as well as various provider review findings and DBHDD collaborative

discussions, the Collaborative has been approved to begin data collection utilizing a tracer


119 | P a g e

methodology to follow individuals who have been readmitted to a CSU and have some form of

suicidality at the time of first admission, known as the Quality Management Suicidal Ideation Quality

Improvement Study. Additionally, a specific question is to be included as a scored element beginning

with Tier 1 and Tier 2+ providers specific to suicide risk screening, and the Columbia Suicide Severity

Risk Scale (CSSRS) in the Assessment and Planning (BHQR) or IRR (CSUQR) sections as appropriate.

Recommendation VII: Findings of ad hoc reporting throughout the fiscal year, led to findings in

which it was identified some services may be underrepresented compared to utilization reports (e.g.

Payment Service Mix Report, internal claims review/analysis). Furthermore, as some services

become more familiar or available to individuals, changes to BHQR/CSUQR tools to include these

services may be required. Thus, in conjunction with DBHDD, services or changes to services

reviewed may occur on an annual basis and it is being recommended that Quality Management

begin reviewing the following service types for FY 2020:

Intensive Case Management subsample (15 additional records)

Office of Deaf Service Review

Crisis Respite Apartment

Supported Employment

Recommendation VIII: As the Programmatic category pilot proved successful through FY 2019,

supplying supportive supplemental information related to certain services, it is recommended that

Quality Management at the request of DBHDD will continue to collect data specific to these

indicators and requirements as a non-scored element.

Recommendation IX: It is notable that all Interview, individual and staff, areas scored in the 90th

percentile since contract inception, indicating a high level of satisfaction for both individuals served

by the providers and the services received, as well as staff knowledge in providing service to

individuals. Furthermore, little difference was identified between scores of similar indicators asked

of both individuals and staff. As a direct result of such four year findings related to the individual and

staff interviews, it has been recommended by the Collaborative and approved by DBHDD to


120 | P a g e

eliminate staff interviews beginning in FY 2020, and reduce the individual interview to a set of ten

questions.

Recommendation X: Three of the five new providers, never before having been reviewed by Quality

Management, scored within the 10 lowest of all 157 reviews. Furthermore, analysis at the indicator

level provided an additional level of detail by which it became evident that new providers require an

additional level of training or technical assistance prior to their initial review occurring. Therefore,

Quality Management is recommending for FY 2020 a collaboration and inclusion of a training model

specifically tailored to new providers.

Recommendation XI: Inclusion of Immediate Actions/Recommendations at BHQR and CSUQR exit

conferences will continue in FY 2020. Furthermore, Quality Management will tailor the formal Exit

and Final Assessment documents to include any Quality Risk Items identified documented within the

first two pages for accessibility and visibility.


121 | P a g e

Section 4: Intellectual and Developmental Disabilities

Background Person Centered Reviews (PCR) and Quality Enhancement Provider Reviews (QEPR) are used to

monitor the portion of the Quality Management system that offers services to individuals with

intellectual and developmental disabilities (IDD). Assessors evaluate providers’ service delivery

systems to determine the quality of services offered and the quality of life of people receiving those

services.

The purpose of the PCR is to assess the extent to which individuals with IDD are supported by the

services they receive, are satisfied with those services, and achieve outcomes important to them.

The purpose of the QEPR is to review providers’ systems and practices to ensure they meet

requirements set forth by the Medicaid waiver and DBHDD, and to evaluate the effectiveness of their

service delivery system. Follow-up review activities offered through the Quality Technical Assistance

Consultation (QTAC) provide targeted technical assistance to help providers improve service delivery

systems and address any issues specific to individuals served.

Quality Assessors use various tools to collect data from interviews, observations, and record reviews

to compile a well-rounded picture of individuals receiving services: how supports and services are

provided by their circle of supports, how involved individuals are in the decisions and plans

developed for them, as well as the use of person centered practices and quality of services provided.

In this section of the report, results are presented for each review tool for both the PCR and QEPR.

Aggregate scores of IDD quality reviews are a weighted average, based on the total number of

standards scored [total met / (total met + total not met)]. Except for the Administrative Staff

Qualifications and Training (Q&T) and Developmental Disability Service Specific (DDSS) tools,

indicators within each tool are grouped into six Focused Outcome Areas (FOAs) important to achieve

and maintain:

Whole Health—individuals receiving services are healthy, aware of their health-related

needs, and direct their own health care regimen


122 | P a g e

Safety— individuals receiving services are safe in their home and work environments and in

their communities; they understand or are learning how to self-preserve in all environments

Person Centered Practices—supports and services are provided based on personal

preferences and direction given by the individual receiving services

Community Life—individuals receiving services are actively participating and developing

social roles in their communities as desired

Choice—information needed to make informed choices on life decisions, such as where to

live, where to work, and which supports, services and providers to use, is available and

individuals receiving services are included in these decisions

Rights—rights are upheld and information and education is provided to ensure

understanding of rights

Sampling Method Person Centered Review (PCR) The PCR uses a random sample of 484 eligible adults, age 18 and over, who had not received a PCR

during the previous year and were receiving services reviewed through this contract. The PCR sample

was stratified and sampled proportionate to each of the six DBHDD regions; therefore, the number

of PCRs per region is proportionate to the number of individuals receiving services within the region.

Table 24 shows the number and percent for the eligible population (sampling frame), as well as the

number and percent of PCRs completed within each region.


123 | P a g e

Table 24. FY 2019 PCR Sample by Region

Population PCR Sample

Region N Percent N Percent

1 2,456 20.2% 92 19.0%

2 2,076 17.0% 88 18.2%

3 3,034 24.9% 119 24.6%

4 1,324 10.9% 56 11.6%

5 1,628 13.4% 62 12.8%

6 1,665 13.7% 67 13.8%

Total 12,183 100% 484 100%

Quality Enhancement Provider Review (QEPR)

Providers offering waiver or state funded services are eligible for a QEPR, a total of 100 reviewed

each year. The QEPR sample for FY 2019 was selected as follows:

Forty-four providers who had not participated in a QEPR during the first three years of the

contract (FY 2016 – FY 2018) were selected for review.

An additional 48 providers were randomly selected from active providers who had been

reviewed previously, but not since the first year of the contract (FY 2016), for a total of 92

service providers.

Three Support Coordination agencies were randomly selected from agencies that had not

been reviewed in the previous year (FY 2018).

Five providers were selected by DBHDD, generating a total sample of 100 providers for the

QEPR.

Three of the 100 selected providers offered crisis services. Because these services are very different,

they are monitored with a separate QEPR tailored to crisis services. Therefore, QEPR scores are

presented separately for 97 service providers and three crisis providers.


124 | P a g e

Based on the number of eligible individuals served, providers were stratified into three categories by

size: “Small,” “Medium,” and “Large.” The number within each size category is shown by year in

Table 25.

Table 25. QEPR Provider Sample by Size and by Year

Provider Size FY 2017 FY 2018 FY 2019

Small (caseload ≤ 30) 52 72 47

Medium (30 < caseload < 100) 19 12 22

Large (caseload ≥ 100) 24 13 25

Support Coordination Agency 1 2 3

Crisis Stabilization Unit 4 1 3

Total 100 100 100

As part of the QEPR, a sample of staff working directly with people receiving services is selected to

complete the review of staff qualifications and training. The selection is random, stratified to ensure

all services the provider offers are included.

Review Processes

PCR and QEPR The focus of the PCR is on the quality of life people receiving services have and the quality of

services received, from the perspective of the individual receiving services. The focus of the QEPR is

on the provider’s overall practices, quality of services offered to all individuals served by the

provider, and level of compliance with Medicaid waiver and state requirements. Both the PCR and

QEPR use the Individual Observation Staff Assessment (IOSA), which includes an Individual Interview

(II) with individuals receiving services, Staff Interview with their support staff (SI), and on-site

observation(s) (OBS) at residential and day programs, as applicable.

In addition to the IOSA, the PCR and QEPR include an evaluation of the Individual Service Plan using

a quality assurance checklist (ISP QA), a Provider Record Review (PPR) of the provider’s record

maintained for individuals sampled, and review of documentation specific to service delivery

requirements using the Developmental Disabilities Service Specific (DDSS) tool. The number of PRR


125 | P a g e

and DDSS reviews completed for each process depends upon the number of services received by the

individuals receiving services.

While the PCR and QEPR share most of the same tools, there are a few exceptions. During the PCR,

the Support Coordinator Record Review (SCRR) and Support Coordinator Interview (SCI) tools are

completed to assess services provided by the Support Coordinator (SC) working with the individual

receiving services. The NCI Adult In-Person Survey is also conducted as part of the PCR. Data

collected for the survey are entered in the ODESA system (Online Data Entry Survey Application)

developed and maintained by Human Services Research Institute (HSRI)). HSRI analyzes and

generates annual reports comparing Georgia’s results to other participating states and the NCI

average.11

During the QEPR, each provider organization receives one administrative review to monitor

compliance with requirements through the Qualifications and Training (Q&T) component of the

review. This includes a review of a random sample of staff records to determine if staff has the

required qualifications for specific services rendered (e.g., background screenings, level of

education), and whether a required training was completed within specified timeframes. The total

number of records reviewed or interviews completed in FY 2019 for the PCR and QEPR is listed in

Table 26 by review tool.

11 Go to https://www.nationalcoreindicators.org/ for more information.

https://www.nationalcoreindicators.org/


126 | P a g e

Table 26. FY 2019 Number of Reviews by Tool and Review Type

Review Tool PCR

(N = 484) QEPR12 (N = 97)

Total (N = 581)

IOSA - Individual Interview (II) 484 744 1,228

IOSA – Observation (OBS) 356 364 720

IOSA - Staff Interview (SI) 475 359 834

ISPQA Checklist13 479 739 1,218

Support Coordinator Interview (SCI) 484 11 495

Support Coordinator Record Review (SCRR) 484 59 543

Provider Record Review (PRR) 844 1,006 1,850

Staff Qualifications and Training (Q&T) N/A 914 914

DD Service Specific 1,004 1,382 2,386

When a PCR is completed, a report is given to the individual’s Support Coordinator that includes an

evaluation of the supports and services provided, the strengths of the support team (including the

Support Coordinator, provider, and family), and recommendations for the individual interviewed and

the support team. The providers who participated in the PCR receive a report of record review

results specific to the services they provide. A provider who participates in a QEPR receives a

comprehensive report that identifies strengths of the service delivery systems, recommendations for

improvement, record review results and several performance scores: the Overall score, Q&T score,

and DDSS score.

Quality Technical Assistance Consultation (QTAC)

The QTAC is conducted approximately 90 days after completion of the QEPR, based on any service

concerns identified during the PCR or QEPR, or if the provider requests technical assistance.

However, if there is a health, safety or rights Quality of Care (QOC) concern identified during the

QEPR, the provider is notified and a QTAC is conducted approximately 30 days after completion of

the QEPR. During the QOC QTAC, the Quality Assessor will specifically address the concern to

ensure it has been addressed by the provider and whether actions are being taken to prevent further

12 Numbers do not include Crisis providers, who were analyzed separately. 13 Five individuals who were interviewed for the PCR and five for the QEPR had an Employment Express Support Plan and therefore, did not have an ISP QA Checklist completed.


127 | P a g e

issue. This provider will also receive anther QTAC approximately 90 days after completion of the

QEPR to follow up on all of the other recommendations provided during the QEPR. Technical

assistance is provided to offer suggestions and guidance to improve service delivery systems for a

specific individual receiving services or to address overall provider systems. The process uses a

consultative approach to address specific issues and concerns related to someone receiving services

or systems and practices that need improvement. The QTAC supplements the PCR and QEPR

processes by affording providers the opportunity to solicit technical assistance for specific needs

within their service delivery system. During FY 2019, Quality Management completed 170 QTACs.

Person Centered Review

PCR Scores by Tool

Figure 33 shows the average score for each tool used during the PCR, comparing FY 2016 through FY

2019. Findings within each year show a similar pattern, with scores for the Individual Interview, Staff

Interview, and Support Coordinator Interview higher than scores for provider or Support Coordinator

documentation (record reviews). Scores across all tools decreased from FY 2016 to FY 2017, with

results remaining approximately the same in FY 2018 and FY 2019, with one exception. Interviews

with the Support Coordinator showed a decrease from 83.5 percent in FY 2018 to 79.4 percent in FY

2019. However, a revised Support Coordinator Interview tool was implemented July 1, 2018, and

direct comparisons are not appropriate. Analysis of the new tool will be provided in the Indicator

Analysis by FOA and Opportunities for Growth section.


128 | P a g e

Figure 33. PCR Scores by Tool and Year

PCR Scores by Focused Outcome Area (FOA)

Indicators in the PCR tools (II, SI, OBS, PRR, SCI, and SCRR) are organized around the six FOAs. Results

for each FOA by fiscal year are shown in Figure 34. While scores for the FOAs had decreased from the

first to the second year of the contract, in FY 2018 (third contract year) scores increased for each

FOA. In the fourth year (FY 2019), FOA scores remained similar to the FY 2018 results.

79.4%

94.3%

93.0%

83.5%

95.6%

92.2%

83.3%

94.3%

91.9%

50% 60% 70% 80% 90% 100%

SupportCoordinator Interview

Staff Interview

IndividualInterview

FY 2017 (N = 481) FY 2018 (N = 484) FY 2019 (N = 484)

76.8%

74.9%

98.6%

77.4%

74.8%

98.2%

73.7%

70.2%

96.8%

50% 60% 70% 80% 90% 100%

SupportCoordinator

RecordReview

ProviderRecordReview

Observation


129 | P a g e

Figure 34. PCR Scores by FOA and Year

Low-Scoring Indicators by FOA (PCR) and Opportunities for Improvement

Each FOA uses several standards, or indicators, to measure how well the area is being addressed.

Most FY 2019 findings showed relatively high scores across all perspectives of the PCR, particularly

for Individual and Staff Interviews, and on-site observations, averaging over 88 percent for each.

However, there are specific areas within each FOA where scoring was relatively low, has remained

low over the past two or three years, or has decreased significantly since FY 2018. These are

presented in this section by FOA in Tables 27 through 33. All p values, based on a two-tailed test, are

shown for the difference of proportions between FY 2018 and FY 2019 (p value FY18/19). DBHDD has

determined statistical significance for this report to be p < .01 (highlighted in blue).

Whole Health The average Whole Health score in FY 2019 was 86.3 percent, close to the same score as in FY 2018

(87.4%). Support Coordinators scored over 95 percent in documenting how they monitor the health

and welfare of individuals receiving services, how the frequency of contacts increases when health

issues are identified, and how supports and services are provided based on the documented needs

85.2%

92.5%

86.3%

84.4%

91.6%

87.4%

82.6%

90.3%

84.7%

50% 60% 70% 80% 90% 100%


Safety

Whole Health

FY 2017 (N = 481) FY 2018 (N = 484) FY 2019 (N = 484)

93.4%

80.9%

77.3%

92.0%

82.3%

76.9%

90.2%

78.7%

71.7%

50% 60% 70% 80% 90% 100%

Rights

Choice

CommunityLife


130 | P a g e

of the person. Most individuals (99% or more) who were interviewed had access to supports and

services to address whole health needs, access to needed medications, and indicated staff followed

up on their specific needs; over 99 percent also agreed with the medications they were taking and

felt those medications were effective.

For the previous three years, (FY 2016 – FY 2018), individuals receiving services had reported a

continued decline in several areas related to medication use. Each year, results indicated individuals

were less likely to be aware of what medications they were taking and why, the side effects of the

medications, and what the medication should look like. Individuals receiving services were also less

likely each year to be aware of their diagnosis. While remaining the lowest-scoring areas as reported

through the individual interviews, information in Table 27 indicates some improvement in FY 2019,

particularly in ensuring an awareness of the diagnosis.

Staff support for the individual to help them learn about medications and their side effects remained

relatively low in FY 2019. Almost 43 percent of staff working with individuals had not helped them

learn about medication they were taking (57.3% met), down over 11 points since FY 2018. The

proportion of staff unable to describe the medications people were taking or their advanced

directives has also decreased significantly since FY 2018. These findings appear to be integrated, as it

is difficult to help someone learn about medications if not familiar with them (unable to describe

them). Many Support Coordinators, when interviewed, were unable to describe how needs related

to specialty health assessments are addressed (59.0% Met).

Of the 79 indicators in the PRR used to measure Whole Health, the greatest gain was shown in

maintaining documentation that a wheelchair evaluation had been completed, up 10 points since FY

2018 to 71.7 percent (p = 0.087). However, in both years close to 14 percent (n = 11) of the 79

indicators showed a score of less than 70 percent. Approximately 40 percent to 70 percent of

providers did not have documentation to support if the following preventative or specialty

assessments had been completed: Dental (54.9%), Pap/Pelvic exam (30.8%), Mammogram (43.9%),


131 | P a g e

PSA (44.4%), Bone Density (49.3%), Vision (50.7%) or Hearing (36.9%) Evaluation, and AIMS testing

(69.7%).14

Table 27 displays PRR indicators with low scores that reflected a decrease of more than nine

percentage points since FY 2018. Education on all medications and the risk/benefits of medication

were offered significantly less often in FY 2019; and providers were less likely to maintain

documentation of PSA tests, bone density tests, or evidence of the person’s ability to manage

personal healthcare. While providers are likely to document how medications were properly

prescribed by a psychiatrist or psychiatric nurse practitioner (86.9%), this had decreased significantly

by close to 10 points since FY 2018.


FY 2017 FY 2018 FY 2019 p value

FY18- FY1915 Individual Interview Individual receiving services was aware of:

The side effects of the medications 52.4%

(n = 410) 43.6%

(n = 433) 47.2%

(n = 422) p = .291

Why medications are prescribed 68.5%

(n = 410) 62.5%

(n = 435) 67.8%

(n= 425) p = .103

What medications he/she is taking 68.3%

(n = 435) 63.5%

(n = 458) 69.3%

(n = 453) p = .060

His or her diagnoses 71.9%

(n = 473) 64.1%

(n = 482) 71.4%

(n = 482) p = .015

The medication's color, shape, when it is taken 80.4%

(n = 408) 72.6%

(n = 431) 77.1%

(n = 423) p = .130

Staff Interview Staff could describe:

How to support the individual to learn about medications

67.7% (n = 195)

68.4% (n = 307)

57.3% (n = 335)

p = .004

The side effects of medications taken 78.7%

(n = 211) 75.4%

(n = 353) 68.8%

(n = 375) p = .048

Advanced health directives 80.6%

(n = 268) 89.3%

(n = 365) 79.3%

(n = 416) p = .000

14 PSA is Prostrate-Specific Antigen and AIMS is the Abnormal Involuntary Movement Scale 15 Shows the difference in percentages from FY 2018 to FY 2019.


132 | P a g e


FY 2017 FY 2018 FY 2019 p value

FY18- FY1915 Provider Record Review Documentation demonstrated how/included:

Individuals and families are provided education on all prescribed medications

25.7% (n = 723)

48.3% (n = 621)

27.9% (n = 380)

p = .000

Education is provided to the individual on the risks and side effects of the medication

33.9% (n = 610)

52.1% (n = 511)

37.5% (n = 651)

p = .000

Copies of the individual’s current preventative healthcare reports for a PSA test

31.3% (n = 147)

55.6% (n = 166)

44.4% (n = 153)

p = .046

Copies of the individual’s current preventative healthcare reports for a bone density test

36.5% (n = 74)

60.7% (n = 84)

49.3% (n = 67)

p =.161

The person’s ability to maintain personal health including any necessary supports

58.0% (n = 531)

59.9% (n = 690)

49.3% (n = 801)

p = .000

Antipsychotic medications were prescribed by a psychiatrist or psychiatric nurse practitioner

96.0% (n = 373)

96.5% (n = 342)

86.9% (n = 297)

p = .000

Support Coordinator Interview:

Needs related to additional specialty health assessments are addressed by the Support Coordinator.

NA NA 59.0%

(n = 144) NA

Safety The average FOA score for Safety in FY 2019 was 96.5 percent, compared to 91.6 percent in FY 2018.

Almost every individual interviewed (> 98.8%) felt safe in all environments and free from all types of

abuse. Most providers (> 99%) and Support Coordinators (> 96%) reviewed had emergency

preparedness plans in place and had documented specific emergency contact information for the

individual receiving services. Support Coordinators advocated to ensure follow-up occurred for

identified safety concerns (97.0%) and reported critical incidents as per the policy (95.2%).

Table 28 includes the lower scoring Safety indicators for FY 2019, indicating some had changed

significantly since FY 2018. Appropriate communication and training are critical components to

helping ensure individuals with IDD understand and are safe from exploitation and neglect, and how

to report abuse or ask for assistance in the event any type of abuse occurs. Findings suggest many


133 | P a g e

individuals who received services did not understand what constitutes exploitation or neglect. At the

same time, significantly fewer staff provided abuse, neglect and exploitation training according to

the person’s learning style. The crisis hotline is available if there is a behavioral crisis or event that

produces a risk of hospitalization; however, staff were significantly less likely to provide education to

individuals receiving services on how to use the crisis hotline.

Support Coordinator Record Review standards all showed scores above 90 percent; however,

interviews with coordinators using the newly implemented interview tool indicated most had not

evaluated the effectiveness of behavioral or safety/crisis plans and many had not coordinated

services to prevent the recurrence of a previously identified crisis or safety concern.

While most provider documentation did not show evidence of education on self-preservation or

effective resiliency skills was offered according to the person’s learning style, this standard showed

some improvement since FY 2018, up 4.6 percentage points. The greatest increase for providers was

in documenting evidence of ongoing efforts to identify, address, and seek prevention of abuse,

neglect, and exploitation, up 16.4 percentage points to 55.9 percent.


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19


Individual is aware of or recognizes what constitutes exploitation

76.6% (n = 480)

70.5% (n = 482)

64.1% (n = 482)

p = .034

Individual is aware of or recognizes what constitutes neglect

81.7% (n = 480)

77.0% (n = 483)

71.8% (n = 483)

p = .064

Individual is aware of how to respond in an emergency/safety situation if supports are incapacitated

78.7% (n = 468)

71.5% (n = 481)

74.5% (n = 483)

p = .294

Individual is aware of how to call 911 in an emergency/safety situation

79.8% (n = 470)

75.0% (n = 472)

75.3% (n = 478)

p = .915

Staff Interview

Staff is providing education on how to use the crisis hotline

52.8% (n = 91)

59.9% (n = 352)

45.4% (n= 443)

p = .000


134 | P a g e


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19

Staff is not training the person according to a person’s learning style, on abuse, neglect, and exploitation

83.4% (n = 302)

85.3% (n = 441)

74.0% (n = 473)

p = .000

Provider Record Review Documentation demonstrated:

How education is offered to the person on how to self-preserve or develop effective resiliency skills according to the person’s learning style

41.1% (n = 893)

41.5% (n = 843)

46.1% (n = 840)

p = .057

Ongoing evidence of identifying, addressing, and seeking prevention of abuse, neglect, and exploitation

22.9% (n = 893)

39.5% (n = 838)

55.9% (n = 832)

p = .000

Evidence of a written order by the physician including the rationale and instructions for the use of adaptive supportive devices or medical protective equipment

72.3% (n = 101)

66.1% (n = 165)

59.0% (n = 166)

p = .182

Monitoring and analysis of Safety/Risk assessments16

69.6% (n = 332)

71.5% (n = 522)

66.6% (n = 580)

p = .079

Monitoring and analysis of the Positive Behavioral Support Plan

72.9% (n = 85)

93.5% (n = 92)

84.3% (n = 83)

p = .051

Support Coordinator Interview

Support Coordinator is aware of safety/crisis or behavioral plan and evaluates effectiveness of the plan

NA NA 44.7%

(n = 107) NA

When informed of a crisis or concern, the Support Coordinator coordinates supports and services to prevent recurrence of a similar behavioral crisis or safety concern

NA NA 55.9%

(n = 127) NA


Person Centered Practices showed an average score of 85.2 percent, similar to FY 2018 (84.8%).

Almost all individuals interviewed (> 99%) felt their preferences and cultural beliefs were solicited

and their preferred communication styles were offered to them. All but five individuals indicated

16 In FY 2017, the indicator was written to only include safety assessments and did not include safety/risk assessments.


135 | P a g e

they were involved in the development of their Individual Service Plan (ISP). Support Coordinators

almost always had the ISP in the record (98.1%) and almost all providers (99.3%) had evidence the

ISP was reviewed and updated as warranted and needed. These findings help support the underlying

essence of person-centered practices: knowing the person well and ensuring the person is involved

in decision making.

Person Centered Practices also mean the person has control of decisions, goals, and activities and is

provided options to support informed choice for these. Findings each year (Table 29) suggest many

individuals receiving services were unable to describe something new they had experienced or

learned in the past six months. Data also indicate many individuals did not regularly review progress

toward their own goals or determine when a goal had been achieved. At the same time, findings

suggest most providers and many Support Coordinators were not regularly reviewing goals with

individuals they served, 41.8 percent and 43.2 percent met respectively, and approximately 20

percent of Support Coordinators could not describe how they evaluate progress on goals. Compared

to FY 2018, providers and Support Coordinators were significantly less likely to consistently include

documentation of the individual’s responses to care, services and treatment (61.7% and 53.1%

respectively).


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19


In the past 6 months, the Individual was able to identify something:

New or experienced 60.3%

(n = 471) 62.6%

(n = 479) 62.9%

(n = 477) p = .924

Had been learned 65.8%

(n = 474) 67.2%

(n = 478) 71.1%

(n = 478) p = .192

Person is involved in regular review of progress toward all goals

72.7% (480)

73.6% (n = 467)

70.1% (n = 469)

p = .234

Individual determines when progress on goals is achieved or goals are met

81.8% (n = 479)

75.3% (n = 482)

79.4% (n = 480)

P = .129

Staff Interview

Staff formally reviews progress on goals/objectives with the individual

72.2% (n = 302)

68.4% (n = 434)

66.5% (n = 465)

p = .545


136 | P a g e


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19 Provider Record Review

Documentation reflected:

The individual’s hopes and dreams 42.0%

(n = 893) 31.5%

(n = 847) 40.6%

(n = 834) p = .000

Review of progress and benefit of goals occurs regularly with the individual

40.4% (n = 891)

44.0% (n = 848)

41.8% (n = 837)

p = .362

The individual’s talents/strengths17 NA 44.2%

(n = 847) 44.8%

(n = 841) p = .804

The individual’s response to the services, supports, care and treatment as a consistent theme

63.3% (n = 894)

68.7% (n = 853)

61.7% (n = 841)

p = .003

Support Coordinator Record Review

Documentation demonstrated:

The individual's talents 28.8%

(n = 479) 35.3%

(n = 484) 38.9%

(n = 483) p = .247

Review of progress and benefit of goals occurs regularly with the individual

56.1% (n = 472)

49.8% (n = 484)

43.2% (n = 482)

p = .040

The individual's strengths 41.0%

(n = 480) 44.2%

(n = 484) 47.0%

(n = 483) p = .32

The individual’s response to the services, supports, care and treatment as a consistent theme

61.6% (n = 479)

66.7% (n = 484)

53.1% (n = 481)

p = .000

Progress notes describe progress toward goals including response to the intervention or activity, based on data

57.8% (n = 481)

62.7% (n = 483)

56.7% (n = 483)

p = .057

Support Coordinator Interview:

Support Coordinator evaluates progress of person’s goals and objectives

NA NA 79.4%

(n = 480) NA

Community Life Community Life, interaction with and integration in the surrounding community, showed the lowest

average score among the FOAs each year of the ASO contract, with 77.3 percent in FY 2019;

however, during interviews with individuals, most indicated they go out in (99.0%) and actively

17 Two separate indicators were combined into one for FY 2018.


137 | P a g e

participate in (97.7%) the community. They chose their community activities (98.1%) and indicated

they have supports needed when participating in the community (98.8%). Compared to FY 2018, a

significantly higher proportion of individuals felt they had opportunities to learn about social roles

(82.5%) (Table 30). Providers showed evidence (PRR) community services were actually provided in

the community as intended (98.6%) and they were aware of people’s interests and preferences for

community activities (98.9%).

Additional information from the PCRs regarding integration into the community is shown in Table 30.

Individuals indicated they were often not getting exposure to new activities in the community or

pursuing preferences related to employment goals, 67.6 percent and 79.9 percent respectively.

As noted above, many individuals receiving services felt they had opportunities to learn about social

roles; however, fewer were able to identify a social role that is important to them. In addition, many

staff members were not supporting individuals to develop new social roles, and Support

Coordinators were often not coordinating supports to develop or maintain social roles.


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19

Individual Interview The individual was:

Exposed to new community activities ( in the past 6 months)

63.2% (n = 473)

70.1% (n = 478)

67.6% (n = 463)

p = .408

Actively pursuing preferences related to goals of employment (not a readiness model)

70.4% (n = 338)

71.5% (n = 312)

79.9% (n = 364)

p = .011

Able to identify a valued social role 75.5%

(n = 478) 76.7%

(n = 480) 73.9%

(n = 482) p = .314

Provided opportunities to learn about social roles in the community

71.5% (n = 478)

75.8% (n = 476)

82.5% (n = 479)

p = .011

Staff Interview

Staff was able to describe how the individual is provided opportunities to:

Develop community employment 58.3%

(n = 235) 60.7%

(n = 338) 62.6%

(n = 396) p = .598

Develop new social roles 60.4%

(n = 298) 70.6%

(n = 429) 71.4%

(n = 447) p = .794


138 | P a g e


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19

Support Coordinator Interview The Support Coordinator was:

Coordinating services and supports to promote opportunities for the person to develop new valued social roles

NA NA 62.6%

(n = 478) NA

Coordinating services and supports to promote opportunities for the person to maintain valued social roles.

NA NA 67.7%

(n = 477) NA

Documentation from Support Coordinators (SCRR) and providers (PRR) reflected several consistently

low-scoring areas (Table 31). Many were not documenting how they supported people to seek

employment in competitive settings, develop social roles that reflect the person’s interests, or to

experience the community. Approximately 53 percent to 64 percent of providers and Support

Coordinators supported individuals to participate in the community the same as citizens without

disabilities, a key component in the Centers for Medicaid and Medicare Services (CMS) Settings Rule,

and have responsibilities in the community as desired. 18


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19

Documentation demonstrated:

Opportunities to seek employment in competitive integrated settings

Provider Record Review 24.9%

(n = 794) 24.3%

(n = 729) 27.9%

(n = 721) p = .107

Support Coordinator Record Review 41.9%

(n = 394) 43.7%

(n = 373) 44.1%

(n = 442) p = .990

Development of social roles and natural supports that reflect the individual’s interests


(n = 881) 28.9%

(n = 829) 36.4%

(n = 782) p = .001


(n = 481 52.9%

(n = 482) 49.3%

(n = 483) p = .263

18 See the Medicaid website for more information on the CMS Settings Rule: https://www.medicaid.gov/medicaid/hcbs/guidance/hcbs-final-regulation/index.html

https://www.medicaid.gov/medicaid/hcbs/guidance/hcbs-final-regulation/index.html


139 | P a g e


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19

How the individual is supported to learn about, explore and experience the community


(n = 891) 44.6%

(n = 827) 51.1%

(n = 788) p = .009


(n = 480) 45.7%

(n = 481) 47.1%

(n = 482) p = .663

How the individual is supported to have or has responsibilities in the community as desired


(n = 883) 50.7%

(n = 825) 53.2%

(n = 785) p = .316


(n = 477) 63.5%

(n = 479) 60.9%

(n = 483) p = .406

How the individual is supported to/able to participate in community activities and employment the same as individuals without disabilities


(n = 854) 55.6%

(n = 806) 53.4%

(n = 776) p = .380


(n = 458) 62.6%

(n = 462) 64.8%

(n = 477) p = .483

Choice Choice is the second, lowest-scoring FOA, showing an average score of 80.9 percent for FY 2019,

similar to FY 2018 (82.3%). Of the 22 standards measured through interviews with individuals

receiving services, 20 reflected a score of over 94 percent or higher, indicating people felt they were

generally satisfied with services, involved in life’s decisions, and offered choices. Of note is a

significant positive increase (p < .001) of 11 points since FY 2018 to 85.6 percent (Table 32), showing

individuals felt they were much more often involved in decisions about work and supported

employment.

Choice is intimately linked to ensuring services are person-centered, based on preferences and

controlled by the individual receiving services, specifically informed choice where viable options are

provided. As indicated in Table 32, staff was often not able to describe how options for competitive

employment were provided (51.2% met) or for integrated living settings, the latter showing a

significant decrease of 11 points since FY 2018, to 53.8 percent. Findings from record reviews often


140 | P a g e

indicated a lack of documentation for how individuals receiving services were making meaningful

and informed choices in different aspects of their lives, including living environments (e.g., how

personal the environment is), living situations (e.g., where, roommates), competitive employment

and education options, and community participation.

Education, exploration, and experiences (the “3Es”) are critical aspects of enhancing people’s

awareness of options available to them. These activities were often not documented by providers

and Support Coordinators. In addition, Support Coordinator documentation showed a decrease in 11

of the 14 SCRR Choice indicators, and coordinators who were interviewed were often not able to

describe how the individual receiving services expresses dissatisfaction with those services. Not

understanding when someone is unhappy with services may inhibit processes of seeking out other

options and opportunities to better serve individuals receiving services.


FY 2017 FY 2018 FY 2019 p value

FY18 - FY19 Individual Interview

The individual is involved in decisions about work and supported employment

73.7% (n = 327)

74.5% (n = 310)

85.6% (n = 365)

p = .000

Staff Interview Staff is presenting options of:

Competitive/supported employment 50.5%

(n = 220) 49.2%

(n = 313) 51.2%

(n = 383) p = .560

Living situations are integrated into local community (i.e., full continuum of housing options, roommate)

65.5% (n = 905)

65.5% (n = 284)

53.8% (n = 356)

p = .003

Provider Record Review Providers demonstrated through documentation how individuals:

Exercise meaningful choices about living environments

21.1% (n = 715)

26.3% (n = 759)

26.5% (n = 791)

p = .929

Are provided a choice of living situations 26.4%

(n = 708) 26.0%

(n = 739) 29.1%

(n = 787) p = .176

Make informed choices about competitive or supported employment options

30.8% (n = 827)

26.5% (n = 759)

32.1% (n = 795)

p = .015


141 | P a g e


Are provided with information to make informed choices (education, exploration and experiences)

27.4% (n = 891)

27.8% (n = 845)

36.6% (n = 835)

p = .000

Make informed choices about community participation and social interaction

42.8% (n = 888)

47.5% (n = 832)

49.3% (n = 819)

p = .464

Support Coordinator Record Review Support Coordinators demonstrated through documentation how individuals:

Are provided with education, exploration and experiences

31.2% (n = 481)

35.8% (n = 483)

31.0% (n = 484)

p = .114

Are offered employment or educational options

49.3% (n = 423)

52.7% (n = 438)

51.2% (n = 463)

p = .652

Exercise meaningful choices about the living environment

50.3% (n = 441)

53.6% (n = 476)

50.7% (n = 477)

p = .370

Are provided choices of living situations 47.1%

(n = 433) 55.6%

(n = 466) 50.1%

(n = 475) p = .091

Exercise meaningful choices about community participation

50.3% (n = 441)

66.3% (n = 483)

67.6% (n = 482)

p = .668

Support Coordinator Interview

Support Coordinator addresses any dissatisfaction the person expresses relating to supports and services, including Support Coordination

NA NA 55.6%

(n = 135) NA

Rights Scores surrounding Rights have remained relatively high each year since FY 2016, showing an

average score of 93.4 percent in FY 2019. Individuals interviewed felt they had privacy (99.6%), did

not have rights restrictions (99.9%), and were treated with respect (99.6%). In addition, staff was

aware of individuals’ rights and preferences for exercising those rights (99.3%), providers (99.7%)

had a signed consent form to authorize the release of information, and almost all (99.5%) provided

evidence funds were not co-mingled between the individual and the provider. During the interview,

most Support Coordinators (86.3%) could describe how they advocate for individuals receiving

services to exercise their rights.


142 | P a g e

Findings in Table 33 show the lowest-scoring indicators for the Rights FOA. While staff scored quite

high in most areas of rights, staff was also significantly less likely, compared to FY 2018, to evaluate

individuals’ understanding of how to exercise their rights. On average, PRR findings were highest on

the Rights FOA; however, the lowest-scoring areas indicate many providers were not offering

ongoing rights education for people receiving services or ensuring individuals were protected, when

applicable, with a lease or other legal agreement to protect them from eviction.

Support Coordinators’ documentation showed improvement on seven of eight Rights indicators

scored through the SCRR. However, 37.3 percent showed evidence of a signature on the notification

of rights and responsibilities, and only 48.4 percent had a signed informed consent form for

psychotropic medications. While documentation often did not show evidence the coordinator

shared the complaints/grievance policy according to the individual’s learning style, this indicator

showed an 18-point increase since FY 2018.


FY 2017 FY 2018 FY 2019

p value FY18 - FY19

Staff Interview Staff is presenting options of:

Staff has a means to evaluate/assess the person’s understanding of and preferences for exercising rights.

78.3% (n = 299)

84.5% (n = 432)

76.5% (n=464)

p = .003

Provider Record Review Provider records demonstrated evidence of:

Ongoing education about rights 32.6%

(n = 891) 43.0%

(n = 849) 49.3%

(n=827) p = .010

For Provider-Owned or Controlled Residential Settings, there was documentation demonstrating the individual has a lease or other legally enforceable agreement to protect from eviction

44.3% (n = 201)

49.8% (n = 227)

53.0% (n = 185)

p = .518

Support Coordinator Record Review Support Coordinator documentation demonstrated evidence of:


143 | P a g e


FY 2017 FY 2018 FY 2019

p value FY18 - FY19

The individual or legal guardian's signature on notification the individual has been informed about rights and responsibilities, at least annually

22.7% (n = 476)

23.6% (n = 461)

37.3% (n = 458)

p = .72

Informed consent for taking psychotropic medications prescribed by a psychiatrist or psychiatric nurse practitioner

35.5% (n = 251)

41.1% (n = 248)

48.4% (n = 225)

p = .19

The complaints/grievance policy is shared in a manner accommodating the individual’s learning style

34.7% (n = 479)

42.0% (n = 483)

60.1% (n = 476)

p = .02

Support Coordinator Interview Support Coordinator advocates to ensure:

Plans are in place to reinstate any rights that have been restricted for safety or legal reasons.

NA NA 24.7%

(n = 73) NA

PCR Scores by Tool and Focused Outcome Area (FOA)

PCR results for FY 2019 are presented by FOA and tool in Figure 35. Findings for the review

components varied across each FOA, and indicated the following:

Record review results (PRR and SCRR) were at least 15 points lower than other tools for

Choice.

The Provider Record Review results reflected the lowest score in Whole Health, Person

Centered Practices and Choice.

Provider Record Review results were highest for Rights but, similar to previous years, less

than 90 percent met (86.3%).

SCRR was greater than 93 percent in areas of Whole Health and Safety; however, SC

Interviews were among the lowest scores for these FOAs.

For Rights, the Record Reviews for Support Coordinators showed the lowest score; over 10

points lower than any other tool.


144 | P a g e

PRR and SC Interview results were more than 10 points lower than other tools measuring

Safety.

Figure 35. FY 2019 PCR Scores by Tool and FOA

Low-Scoring FOAs by Perspective (Tool)

Each tool used in the PCR process represents a different perspective on quality of services provided,

and the extent to which goals and outcomes are addressed through those services. Interviews,

implemented as a conversation, offer the person receiving services, the provider and the Support

Coordinator the opportunity to describe how services are offered and received and what their

impact may be. Documentation review (PRR and SCRR) offers the opportunity for providers and

77.9%

81.8%

71.9%

79.1%

95.6%

93.8%

97.2%

99.9%

97.5%

94.7%

95.5%

91.9%

85.4%

76.1%

83.0%

89.6%

92.3%

89.1%

50% 60% 70% 80% 90% 100%


Safety

WholeHealth

Individual Interview SC Interview Staff Interviews Observation SCRR PRR

86.3%

58.5%

70.3%

67.0%

66.9%

67.0%

99.0%

99.0%

97.5%

97.6%

91.9%

88.4%

79.7%

83.4%

74.2%

98.4%

96.6%

88.0%

50% 60% 70% 80% 90% 100%

Rights

Choice

Community Life


145 | P a g e

Support Coordinators to show evidence of activities and how policies and procedures are

implemented through their service delivery systems.

The following graphic displays the lowest-scoring FOA from each tool or perspective. Conversations

with people involved in the quality management system indicated community inclusion (FOA –

Community Life) was the lowest score from data collected through all the different interviews. On

the other hand, documentation from providers and Support Coordinators was least likely to have

evidence of how choice (PRR) and rights (SCRR) are offered to individuals receiving services.

Findings may suggest that while providers and Support Coordinators document community activities,

the people receiving services do not always feel it is present in their lives to the extent desired.

Analysis of these areas at the indicator level is detailed in the Indicator Analysis by FOA (PCR) and

Opportunities for Improvement section.


146 | P a g e

FOAs by Demographics

Analysis was completed comparing FOA results across regions, residential settings, and funding

source.19 Results are presented for FOA by region, residence and funding source (Figures 36 through

39). To enhance the readability of the side-by side demographics, the start point of the axis is 50

percent. Please note some of the demographic categories have a relatively small sample. For

example, the sample for host homes is 28 and for state funded services (SFS) the sample is 47.

Findings indicate the following:

In FY 2018, Region 1 showed the lowest scores for all the FOAs; however, in FY 2019 the

region was the highest scoring in Choice and second highest scoring for Community Life and

Rights.

Individuals receiving services and living in Region 6 were significantly less likely to be

participating in their Community Life as desired (p = .000), compared to their counterparts

living in other regions.20

Individuals living in Region 2 were significantly less likely to have Choice in their lives (p

< .001), compared to their counterparts living in other regions.

Individuals receiving services and living with a parent scored significantly lower in Whole

Health than their counterparts in any other residential settings (p < .001).

Receiving services while living in a group home showed a significantly lower likelihood of

being integrated into the community as desired when compared to other residential settings

(p<.000).

Results for the COMP waiver indicated individuals were more likely to have Whole Health

indicators met compared to NOW or SFS funding sources (p < .01).

19 Analyses by age group and disability did not generate discernable differences across categories. The “Other” residential category had only three PCRs and results are not shown. 20 It should be noted the difference of proportions test was based on the number of indicators scored; therefore, the n sizes were relatively large, increasing the likelihood of statistical significance.


147 | P a g e

Figure 36. FY 2019 PCR Scores by FOA and Region

85.2%

92.5%

86.3%

83.5%

94.0%

89.7%

87.7%

94.8%

88.0%

88.8%

92.5%

86.9%

83.7%

92.0%

85.0%

83.1%

90.9%

85.9%

86.3%

92.3%

84.9%

50% 60% 70% 80% 90% 100%


Safety

Whole Health

Region 1 (n = 92) Region 2 (n = 88) Region 3 (n = 119) Region 4 (n = 56)

Region 5 (n = 62) Region 6 (n = 67) State (N = 484)

93.4%

80.9%

77.3%

93.8%

81.3%

70.9%

95.1%

83.0%

79.8%

93.7%

82.3%

83.6%

93.5%

79.6%

74.4%

91.2%

75.6%

75.4%

94.0%

85.3%

81.7%

50% 60% 70% 80% 90% 100%

Rights

Choice

CommunityLife


148 | P a g e

Figure 37. FY 2019 PCR Scores by FOA and Residential Setting

85.1%

92.5%

83.6%

86.8%

92.6%

87.8%

83.5%

91.6%

87.6%

85.2%

92.7%

88.7%

50% 60% 70% 80% 90% 100%


Safety

Whole Health

Group Home (n = 138) Host Home (n = 28)

Own Place (n = 54) With Parents (n = 261)

93.9%

81.4%

77.9%

93.7%

84.0%

83.3%

90.4%

78.1%

80.4%

93.3%

80.2%

74.0%

50% 60% 70% 80% 90% 100%

Rights

Choice

CommunityLife


149 | P a g e

Figure 38. FY 2019 PCR Scores by FOA and Funding Source

Individual Service Plan Quality Assurance Checklist (ISP QA)

During PCRs, Assessors review the content of the ISP to assess adherence to requirements and the

extent to which the plan addressed specific needs, goals, and desires.

ISP Written to Support a Meaningful Life The ISP QA checklist provides an overall rating for each service plan, based upon the degree to which

the ISP is written to provide a meaningful life for the individual receiving services. Each ISP was

identified as written to support one of the following lifestyles:

Service Life: The ISP supports a life with basic paid services and paid supports. Needs “important

for” the individual were addressed, such as health and safety. However, there is not an organized

effort to provide support in obtaining other expressed desires “important to” the individual, such as

getting a driver’s license, having a home, or acting in a play. The individual receiving services was not

connected to the community and has not developed social roles but expresses a desire to do so.

82.0%

93.1%

82.8%

86.1%

92.7%

84.1%

85.1%

92.3%

87.6%

50% 60% 70% 80% 90% 100%


Safety

Whole Health

COMP (n = 259) NOW (n = 178) SFS (n = 47)

93.5%

79.5%

77.0%

94.2%

82.1%

79.2%

93.1%

80.5%

76.4%

50% 60% 70% 80% 90% 100%

Rights

Choice

CommunityLife


150 | P a g e

Good but Paid Life: The ISP supports a life with connections to various supports and services (paid

and non-paid). Expressed goals “important to” the person are present, indicating goals and desires

were obtained beyond basic health and safety needs. Individuals may go out into the community but

with only limited integration into community activities. For example, someone may go to church but

not have the opportunity to participate in Sunday school or sing in the choir. Community connections

are lacking, and there is an indication of a desire to achieve more.

Community Life: The ISP supports a life with the desired level of integration in the community and in

various preferred settings. Friends and support beyond providers and family members are

demonstrated, as is the development of meaningful social roles, such as belonging to a book club or

having employment in a competitive rather than segregated environment. Rather than just going to

church, the individual receiving services may be an usher at the church or sing in the choir.

Relationships developed in the community are reciprocal. The ISP is written with goals that help

support moving toward a Community Life, as she or he chooses.

Overall findings from this Life Indicator are presented in Figure 39. A majority of ISPs reviewed each

year was written to support a “Good but Paid Life.” Findings in FY 2019 reflect a significant shift from

Community Life (decreasing 10 percentage points; p=.000) to a Good but Paid Life (increasing close

to 10 points; p =.001), indicating ISPs were significantly less likely to support people receiving

services.


151 | P a g e

Figure 39. PCR ISP QA Life Indicator by Year

The ISP QA Life Indicator findings by region and year are presented in Table 34. While N sizes are

relatively small across regions, each had over 50 ISPs reviewed, suggesting the following:

The overall downward trend in Community Life seen in Figure 39 appeared to be significant in

three regions (shaded in blue): Region 2 (drop of 23.5 points; p < .001), Region 3 (drop of 9.1

points; p < .018) and Region 5 (drop of 9.5 points; p < .001).

Region 3 showed a steady increase in the proportion of ISPs written to support a Service Life

and a steady decrease in the proportion written to support a Community Life.

ISPs in Region 6 were consistently most likely to represent a Service Life.


152 | P a g e

Table 34. ISP QA Life Indicator by Region and Year

Service Life Good but Paid Life Community Life

Region FY 2017 FY 2018 FY 2019 FY 2017 FY 2018 FY 2019 FY 2017 FY 2018 FY 2019

1 6.3% 15.7% 11.1% 55.8% 58.4% 74.4% 37.9% 25.8% 14.4%

2 10.0% 5.1% 14.8% 75.6% 67.9% 81.8% 14.4% 26.9% 3.4%

3 8.2% 17.1% 19.3% 74.6% 67.9% 74.8% 17.2% 15.0% 5.9%

4 9.8% 15.9% 3.6% 86.3% 77.8% 92.7% 3.9% 6.3% 3.6%

5 20.6% 8.9% 14.8% 68.3% 75.0% 78.7% 11.1% 16.1% 6.6%

6 23.7% 25.9% 27.3% 69.5% 55.2% 56.1% 6.8% 19.0% 16.7%

ISP Expectations

In December 2017, DBHDD completed modifications to the checklist template and expectations, and

the new form went into effect in January 2018. While the original form has slowly been phased out

as each ISP is renewed, 53 ISP QA Checklists were completed in FY 2019 using the original form with

426 completed using the new ISP form. All the expectations and indicators have been modified, with

the exception of the ISP QA Life Indicator Rating of how the ISP is written, shown in Table 34 for all

479 checklists completed. For the remaining results, data are presented for the original and new ISPs

separately.

Results Using Original ISP QA Checklist

Quality Assessors reviewed 12 different indicators to measure the quality of the ISP. Each indicator is

listed in Table 35 and each has four requirements that must be met. The indicator is rated 0 - 4, “0”

meaning the ISP did not meet any of the requirements and “4” meaning all four of the requirements

were addressed. The percent of ISPs at each rating on 12 different indicators is shown. For example,

56.6 percent of ISPs reviewed in FY 2019 had all four requirements present for the indicator

regarding the communication chart. Data indicate:

On average, 66.4 percent of ISPs reviewed addressed all elements in each section.

The Health and Safety Review section was most likely to have all four indicators met (83.0%).

The ISP indicators measuring how well the individual’s hopes and dreams were addressed and

whether goals were person-centered, were most likely to have two or fewer indicators addressed,


153 | P a g e

over 26 percent each with a rating of “0”, “1” or “2”. These indicators speak to the extent to which

the Support Coordinator knows and understands the specific needs and desires of people they serve.

Table 35. FY 2019 ISP QA Checklist Ratings by Expectation (N = 53)

Ratings ISP QA Checklist Description 0 1 2 3 4 Average

Person-centered Important to/for 1.9% 0.0% 3.8% 15.1% 79.2% 3.7

Health and Safety Review Section completed accurately and thoroughly. (HRST information should be noted throughout ISP.)

1.9% 3.8% 1.9% 9.4% 83.0% 3.7

SIS completed and support needs are addressed in the ISP. (SIS Information should be noted throughout ISP.)

1.9% 0.0% 0.0% 26.4% 71.7% 3.7

Meeting Minutes 3.8% 3.8% 1.9% 15.1% 75.5% 3.5

Service Summary 1.9% 3.8% 9.4% 15.1% 69.8% 3.5

Communication Chart (this includes traditional and non-traditional forms of communication)

1.9% 0.0% 11.3% 30.2% 56.6% 3.4

Training Goal Action Plan 1.9% 1.9% 3.8% 39.6% 52.8% 3.4

Relationship Map & discussion on ways to develop relationships

1.9% 0.0% 13.2% 28.3% 56.6% 3.4

Rights, Psychotropic Medications, Behavior Supports Section

13.2% 0.0% 1.9% 7.5% 77.4% 3.4

Action Plans/Objectives 1.9% 3.8% 9.4% 30.2% 54.7% 3.3

Hopes and Dreams: If you could be, do or include anything in your life what would it be?

11.3% 3.8% 11.3% 9.4% 64.2% 3.1

Goals are Person Centered. (Include the discussion/rationale, desired outcome and goal to make this determination.)

5.7% 5.7% 15.1% 18.9% 54.7% 3.1

Total 4.1% 2.2% 6.9% 20.4% 66.4% 3.4

Table 36 shows the average rating (0 - 4) by indicator for FY 2017 through FY 2019. Information in the

table indicates:

The average rating for all three years was 3.4 but showed some change by year at the

indicator level.

Indicators measuring how well the individual’s hopes and dreams were addressed and

whether goals were person-centered were the lowest rated expectations all three years.


154 | P a g e

The greatest difference between FY 2018 and FY 2019 was in scoring the Rights, Psychotropic

Medications and Behavior Supports section, a drop from an average of 3.9 to 3.4.

Table 36. Average Rating by Expectation (0 - 4) by Year

ISP QA Checklist FY 2017

(N = 481) FY 2018

(N = 397) FY 2019 (N = 53)

Rights, Psychotropic Medications, Behavior Supports Section

3.9 3.9 3.4

Health and Safety Review Section completed accurately and thoroughly (HRST information should be noted throughout ISP)

3.9 3.9 3.7

Person-centered Important To/For 3.6 3.7 3.7

Supports Intensity Scale (SIS) completed and support needs are addressed in the ISP

3.6 3.6 3.7

Communication Chart 3.4 3.4 3.4

Service Summary 3.3 3.4 3.5

Relationship Map & discussion on ways to develop relationships

3.3 3.3 3.4

Meeting Minutes 3.2 3.3 3.5

Training Goal Action Plan 3.2 3.2 3.4

Hopes and Dreams 3.0 3.1 3.1

Action Plans/Objectives 3.1 3.1 3.3

Goals are Person-centered 2.9 3.0 3.1

Overall Average 3.4 3.4 3.4

Results Using New ISP QA Checklist

The New ISP QA Checklist (effective date 1/1/2018) uses seven Likert scaled indicators with the

following scoring options: Needs Improvement, Emerging, Achieving and Exceeding. Average

statewide ratings are presented by indicator in Table 37, suggesting almost all the ISPs were

Achieving or Exceeding when documenting, in the Health and Safety section, how all current issues,

needs and/or risks are addressed (95.6%). Several indicators measuring how people are connected

to their communities and how person-centered their services are, were most likely to be scored as

Needs Improvement or Emerging (percent shown in parentheses):

Summary reflects the individual’s community life (approximately 47%)


155 | P a g e

The relationship map demonstrates the person is connected to the community (close to 42%)

Goals and objectives were person centered (31.9%)

Table 37. FY 2019 New ISP QA Checklist Ratings (N = 426)

Indicator

Needs Improvement Emerging Achieving Exceeds

Communication Chart provides meaningful information staff can use to best support the person.

1.4% 25.1% 40.8% 32.6%

Goals and objectives are person centered. 5.6% 26.3% 31.9% 36.2%

Health and Safety section includes how all current issues, needs and/or risks are addressed.

1.7% 0.0% 62.3% 36.0%

Map demonstrates the person is connected to the community.

6.3% 35.4% 20.4% 37.8%

Profile is detailed and includes person centered information.

5.9% 18.8% 35.9% 39.4%

Summary describes changes in the person’s life in the last year and the supports/plans needed to address the change with the person.

3.5% 27.7% 48.6% 20.2%

Summary reflects the person’s community life. (Maximizes opportunities for community living and receiving services in the most integrated setting.)

10.6% 36.4% 42.7% 10.3%

Average New Checklist Ratings 5.5% 24.4% 40.2% 30.3%

PCR Results by Service

The PCR includes a record review for every service received, i.e., services eligible for review by the

Collaborative. Information in Table 38 shows results for record reviews completed during the PCRs,

by service and year. The N is the number of records reviewed for each service and the percent met is

based on the total number of indicators reviewed. When reviewing the findings, it is important to

note several services had fewer than 20 records reviewed, listed in a separate section in the table.

Among services with 32 or more records, listed first in the table, Supported Employment reflected

the highest score in FY 2019 (81.2%), the same as the FY 2018 results.


156 | P a g e

Table 38. PCR PRR Results by Service and Year

FY 2017 FY 2018 FY 2019 Service N % Met N % Met N % Met

Provider Record Review 894 70.2% 856 74.8% 844 74.9%

Community Access (Group) 378 67.9% 361 74.6% 371 74.2%

Community Access (Individual) 95 72.9% 92 74.4% 87 74.7%

Community Living 72 66.8% 104 70.1% 74 72.9%

Community Residential 195 73.4% 195 77.1% 164 75.9%

Prevocational 74 64.9% 51 71.6% 71 72.5%

Supported Employment 71 77.8% 32 81.0% 48 81.2%

Behavioral Supports 2 91.7% 8 83.9% 18 81.9%

Nursing Services 1 87.8% 2 68.8% 8 78.3%

Respite 6 61.8% 8 67.6% 3 69.6%

Occupational Therapy - - 1 83.3% - -

Physical Therapy - - 1 82.8% - -

Speech/Language Therapy - - 1 82.8% - -

Support Coordination Record Review 481 73.6% 484 77.5% 484 76.8%

Table 39 displays scores by FOA for the services reviewed through record reviews (PRR and SCRR)

during PCRs competed in FY 2019.21 Among providers with more than 30 records reviewed, the first

seven listed in the table, results indicated the following:

For a majority of services, the lowest score was for Choice, ranging from 50.2 percent for

Community Residential Alternative (CRA) to 75.1 for Supported Employment.

Providers of CRA services scored lowest in Choice and Community Life.

Community Access, offered at the individual level, and Community Living Supports reflected

the lowest Safety scores.

Support Coordinators scored approximately 95 percent in documenting Whole Health and

Safety, but had the lowest average Rights score (66.8%).

21 Note these results are not comparable to FOA findings presented earlier as only the PPR or SCRR are used here and previous results included all tools in the PCR.


157 | P a g e

Table 39. FY 2019 PCR Record Review Results by FOA and Service

Service Whole Health Safety PCP

Comm Life Choice Rights Average

Community Access(Group) (N = 371) 65.6% 83.8% 79.5% 69.3% 59.0% 88.7% 74.2%

Community Access(Individual) (N = 87) 66.1% 79.2% 79.5% 77.3% 61.6% 85.3% 74.7%

Community Living Support (N = 74) 70.5% 77.2% 74.0% 70.1% 59.6% 84.6% 72.9%

Community Residential Alternative (N = 164) 86.0% 80.2% 73.3% 61.7% 50.2% 84.5% 75.9%

Prevocational (N = 71) 63.6% 82.4% 77.8% 66.9% 56.0% 86.8% 72.5%

Supported Employment (N = 48) 65.4% 85.5% 83.1% 91.2% 75.1% 88.0% 81.2%

Support Coordination (N = 484) 94.0% 95.7% 79.4% 67.2% 66.9% 66.8% 76.8%

Services with fewer than 30 records reviewed

Behavioral Supports (N = 18) 73.7% 85.9% 87.4% 87.8% 81.8% 75.4% 81.9%

Respite (N = 3) 66.7% 73.1% 70.8% 55.6% 55.6% 90.6% 69.6%

Nursing Services (N = 8) 82.0% 77.3% 79.3% 100.0% 43.5% 90.7% 78.3%

PCR Strengths and Recommendations

During the PCR, Assessors identified strengths about services offered and provided

recommendations to help improve services and overall quality of life for people receiving services.

The strengths and recommendations are not included in calculating the provider’s score but are

included in the report as information for the provider. Table 40 displays strengths identified in

approximately half or more of the PCRs completed in FY 2019 and Table 41 displays

recommendations provided during approximately 40 percent or more of PCRs completed during FY

2019.

Staff strengths most often cited included an awareness of unique safety needs, achievements, and

an understanding of how individuals they serve communicate everyday decisions. People receiving

services felt valued, possibly in part because staff promoted their independence. Challenges that

generated recommendations surrounded areas of safety education and skill building to help with

various safety situations, and developing ways to identify new experiences in the community and in

daily activities.


158 | P a g e

Table 40. FY 2019 Strengths Most Often Identified During a PCR

Strength Instances Percent of PCRs

Staff is aware of the unique safety needs of the person. 328 67.8%

Staff acknowledges the person's achievements. 276 57.0%

Staff has a clear understanding how the person communicates choice making in everyday decisions.

271 56.0%

Person feels valued. 270 55.8%

Staff consistently promotes independence. 245 50.6%

Table 41. FY 2019 Recommendations Most Often Provided During a PCR

Recommendation Instances Percent of PCRs

Provide education, according to the appropriate learning style, on the different types of exploitation and how to keep safe, or what to do in these situations.

208 43.0%

Identify ways to expose the individual to new experiences in the community.

202 41.7%

Conduct "what if" scenarios to determine the individual's skills in various safety situations.

201 41.5%

Ensure daily schedules and activities promote exposure to new things (new places and new faces) and are not stagnant in nature.

193 39.9%

Quality Enhancement Provider Review (QEPR)

QEPR Scores by Size

The QEPR Overall score is based on findings from the Individual Interview, Staff Interview,

Observation and Provider Record Review (PRR). In addition, the Administrative Qualifications and

Training (Q&T), based on a sample of staff rendering services, is used to determine if staff has all the

required education, certifications and training as per services rendered. The Developmental


159 | P a g e

Disability Service Specific (DDSS) component is used to monitor compliance specific to each service.

Scores from these review components are given to the provider but not included in the provider’s

Overall score. Three of the providers offer crisis services and received a separate crisis specific QEPR

for those services. Because crisis services are very different than typical services, results are

presented separately.

In FY 2019, 97 providers were selected for a QEPR: 47 Small, 22 Medium, and 28 Large. Three

additional QEPRs completed for crisis services providers resulted in a total of 100 QEPRs.

Comparisons by year are not appropriate as the QEPR samples were not selected to be

representative of the state – the proportion of Small providers increased each year through FY 2018.

See Figure 40 for distributions of QEPRs by provider size for FY 2016, FY 2017, FY 2018 and FY 2019.

Figure 40. FY 2016 - FY 2019 QEPR Samples by Provider Size and Type

Overall QEPR Score by Tool and Year

Figure 41 shows the average Overall scores for providers reviewed from FY 2017 to FY 2019 and

scores for each tool used in the review process. Several characteristics of the QEPR samples impact

the ability to track trends and comparisons across years must be made with caution:

47%

72%

52%

47%

22%

12%

19%

36%

28%

15%

25%

16%

3%

1%

4%

1%

0% 20% 40% 60% 80% 100%

FY 2019

FY 2018

FY 2017

FY 2016

Small (caseload ≤ 30) Medium (30 < caseload < 100)

Large (caseload ≥ 100) Crisis Provider


160 | P a g e

The samples each year were not selected to be representative of all providers.

Some providers were reviewed more than once, particularly for QEPRs completed in FY 2019.

As shown in Figure 40, the proportion of reviews completed by size varied each year.

The Overall score for FY 2019 was 86.2 percent, with Observations and Staff Interviews showing the

highest scores of 97.2 percent and 95.7 percent respectively. As in previous years, the PRR reflected

the lowest-scoring area (73.2%). The Q&T score (83.2%) is similar to providers reviewed during FY

2018 (84.6%).

Figure 41. Overall QEPR Scores by Tool and Year

Overall Score

FY 17: 83.7% FY18: 84.8%FY19: 86.2%


FY17: 90.7%FY18: 89.6%FY19: 92.3%

Staff Interview

FY17: 95.2%FY18: 95.3%FY19: 95.7%

Observation

FY17: 95.7%FY18: 96.0%FY19: 97.2%

Record Review

FY17: 69.6%FY18: 72.4%FY19: 73.2%


FY17: 82.6%

FY18: 84.6%

FY19: 83.2%

DD Service Specific

FY17: 99.8%

FY18: 99.8% FY19: 99.8%


161 | P a g e

Overall Crisis Provider Scores

Nine crisis provider QEPRs have been completed, one in FY 2016, four in FY 2017, one in FY 2018,

and three in FY 2019. Figure 42 shows the Overall score and scores by tool and year. Compared to

other providers reviewed with the QEPR, crisis providers showed similar scores across all tools, with

the exception of the PRR on which crisis providers scored higher than other providers, 88.9 percent

and 73.2 percent respectively.

Figure 42. Overall Crisis Provider Scores by Tool and Year

Qualifications and Training (Q & T) and Service Specific (DDSS)

Every QEPR includes a review of staff qualifications and training (Q&T) and service specific reviews

(DDSS). In FY 2019, a sample of 914 staff records was reviewed, stratified by service to ensure all

services offered by each provider were included. A total of 33 staff records were reviewed for the

three providers offering crisis services. The primary purpose of the Q&T record review is to confirm

Overall Score

FY17: 86.8% FY18: 93.1%FY19: 92.3%


FY17: 86.0% FY18: 90.2%FY19: 91.1%

Staff Interview

FY17: 89.1% FY18: 97.8%FY19: 97.8%

Observation

FY17: 92.5% FY18: 97.9%FY19: 96.9%

PRR

FY17: 84.1% FY18: 91.1%FY19: 88.9%

FY17: N = 4 FY18: N = 1 FY19: N = 3


162 | P a g e

relevant staff information is accurate and up to date (e.g., driver’s license, performance evaluations,

background screening) and staff had received all required training specific to services provided. DDSS

reviews are completed to ensure services are provided as specified by DBHDD (Figure 43).

Scores for the DDSS component of the QEPR have remained close to 100 percent each year.

Qualification and Training scores varied little for providers reviewed each year, from 82.6 percent to

84.6 percent, with an average in FY 2019 of 83.2 percent. Providers of crisis services showed higher

Q&T scores each year, with an average in FY 2019 of 96.0 percent for the three providers reviewed.

Figure 43. QEPR Qualifications and Training and DDSS Results by Year

As noted previously, the Q&T review component uses a sample of employees from each provider to

determine compliance with standards. The number of employee records varies based upon the total

number of staff and services rendered. Rates for each indicator measured can be based on the

number of staff records reviewed or the number of providers who have ensured each staff record

reviewed as part of their QEPR has the requirement met. For example, most staff followed DBHDD’s

policy 04-104 to ensure a DBHDD approval letter identifying criminal check screening results are

Providers (Non-Crisis)

Q & T

• FY 2017: 82.6% (N = 947)

• FY 2018: 84.6% (N = 780)

• FY 2019: 83.2% (N = 914)

DDSS

• FY 2017: 99.8% (N = 1,667)

• FY 2018: 99.8% (N = 1,101)

• FY 2019: 99.8% (N = 1,382)

Providers (Crisis)

Q & T

• FY 2017: 91.8% (N = 51)

• FY 2018: 94.5% (N = 5)

• FY 2019: 96.0% (N = 33)

DDSS

• FY 2017: 100% (N = 48)

• FY 2018: 100% (N = 6)

• FY 2019: 100% (N = 37)


163 | P a g e

available in the personnel record (98.5%); however, only 88.0 percent of providers had the letter

present in all staff records that were reviewed. To help guide possible quality improvement

initiatives, it is important to understand how many providers have the requirements in place for all

staff. In the following table, Table 47, low-scoring Q&T indicators are shown with the percent of staff

records for which the indicator was met and the percent of providers for which all staff records

reviewed for that provider on the indicator were met.

Table 42. FY 2019 Qualifications and Training Low-Scoring Indicators

Required training in the following areas has been completed: % Met

(Staff Records) % Met

( Providers)

Skill development to work with individuals who have co-occurring diagnoses

50.8% 27.8%

Proper body mechanics (lifting/transferring/positioning) needed to assist with activities of daily living within the first 60 days of hire;

59.0% 33.3%

or annually thereafter 52.4% 21.3%

Proper body mechanics (lifting/transferring/positioning) specific to the individual’s care plan prior to working with each individual

63.3% 35.5%

or annually thereafter 68.0% 34.5%

Utilization of Communication Skills; Behavioral Support and Crisis Intervention techniques to de-escalate challenging and unsafe behaviors and/or Nationally benchmarked techniques for safe utilization of emergency interventions of last resort, and the Georgia Crisis Response System (GCRS) to access crisis services

69.7% 47.1%

Utilization of suicide prevention skills 61.6% 36.0%

Specific medications and their side effects 71.7% 49.4%

Fire Safety 87.8% 40.0%

QEPR Scores by Provider Size22

Figure 44 displays the QEPR Overall and Q&T scores by size of the organization and year.23 In FY

2019, Medium sized providers showed the lowest score for both review components, an Overall

score of 84.6 percent and a Q&T score of 80.4 percent, but differences among the provider sizes

were relatively small.

22 Crisis provider scores were excluded because these services are different than typical services. 23 See Figure 40 for details regarding provider size categories.


164 | P a g e

Figure 44. Overall Score and Qualifications and Training Score

by Provider Size and Year

QEPR Scores by Focused Outcome Area (FOA)

Each year, the Overall score is divided into six FOAs and FOA scores are calculated with the combined

results from the Individual Interview, Staff Interview, Observation (IOSA), and Provider Record

Review.24 Results for QEPRs (except for crisis providers) are shown by FOA and year in Figure 45.

Scores reflecting community integration were lowest among all the FOAs each year.

24 FOAs calculated and presented in the PCR section also included the Support Coordinator Interview and SCRR, which are not part of the QEPR process. Comparisons between the PCR and QPER are not appropriate.

87.2%

84.6%

85.6%

84.4%

82.0%

86.3%

82.7%

83.2%

86.1%

0% 25% 50% 75% 100%

Large

Medium

Small

Overall Score

FY 2017 (N = 96) FY 2018 (N = 99) FY 2019 (N = 97)

84.0%

80.4%

84.5%

85.8%

81.1%

85.3%

86.4%

76.0%

81.5%

0% 25% 50% 75% 100%

Large

Medium

Small



165 | P a g e

Figure 45. Overall QEPR Scores by FOA and Year

QEPR Scores by FOA and Tool

Results by FOA are presented in Figure 46 for each tool used in calculating the Overall score for the

QEPR process – Individual Interview, Observation, Staff Interview and PRR.25 Findings for the

providers reviewed this year indicate the following:

Provider documentation (PRR) was the lowest-scoring tool across all the FOAs, ranging from a

low of 55.0 percent for measures surrounding Choice to 84.2 percent for Rights. This pattern

is similar to results for providers reviewed in FY 2018.

In the Choice FOA provider documentation was approximately 40 percentage points lower

than results for other tools.

25 See Table 26 for the number of interviews and records completed for each QEPR component.

FY17: 83.7%(N = 96)

FY18: 84.8%(N = 99)

FY19: 86.2%(N = 97)

Whole Health

FY17: 83.2%FY18: 84.9%FY19: 84.8% Safety

FY17: 90.9%FY18: 91.0%FY19: 91.8%

PCP

FY17: 81.6%FY18: 80.4%FY19: 83.3.%Community

FY17: 70.4%FY18: 72.0%FY19: 75.4%

Choice

FY17: 75.9%FY18: 78.2%FY19: 80.2%

Rights

FY17: 91.2%FY18: 92.4%FY19: 93.7%


166 | P a g e

Figure 46. FY 2019 QEPR Scores by Focus Outcome Area and Tool

Provider Record Review results were lowest for Choice (55.0%), Community Life (66.0%) and Whole

Health (71.5%). The lowest-scoring indicators from each of these FOAs are listed in Figure 47. Over

half the records reviewed this year had not documented, as applicable, various preventative health

reports or the risk and benefits of medications (Whole Health); opportunities for competitive

employment or support to develop social roles, natural supports, or to explore and experience the

community (Community Life); or informed choices for living situations, employment options,

community participation and social interactions (Choice).

75.7%

81.4%

71.5%

93.2%

99.1%

96.3%

94.7%

96.8%

95.5%

88.2%

91.9%

89.3%

0% 25% 50% 75% 100%


Safety

Whole Health

Individual Interview (N = 744) Staff Interview (N = 359)

Observation (N = 364) PRR (N = 1,006)

84.2%

55.0%

66.0%

96.7%

98.1%

94.9%

97.6%

94.6%

88.8%

97.9%

95.7%

85.1%

0% 25% 50% 75% 100%

Rights

Choice

CommunityLife


167 | P a g e

Figure 47. FY 2019 PRR Low-Scoring Indicators - QEPR

QEPR Scores by FOA and Provider Size

Overall scores by FOA and size are presented in Figure 48. Large provider organizations reviewed

during FY 2019, compared to smaller organizations, were somewhat more likely to have indicators

met that measure Person Centered Practices and Community Life. Most other differences across the

FOAs and provider size were relatively small.

• Education provided on risks/benefits of medications (32.1%) or on all prescribed medications (29.1%)

• Preventative test reports are in the record for: Pap (25.6%), Mammogram (41.3%), Bone Density (46.2%), vision (48.6%) or hearing (35.9%) evaluations

Whole Health

• Opportunities to seek employment in competitive settings (24.9%)

• Supporting development of social roles/natural supports reflective of person's interests (29.3%)

• Supporting people to learn about, explore, and experience community (43.3%)

• Participation in community as citizens without disabilities (48.0%)

Community Life

• Choice of living situations (24.9%) or environments (23.3%)

• How the person is making informed choices in general (education, exploration and experiences) (29.8%), about competitive or supported employment options (25.2%) or about community participation and social interactions (39.7%)

Choice


168 | P a g e

Figure 48. FY 2019 QEPR Provider Scores by FOA and Size

Quality and Technical Assistance Consultation (QTAC) A total of 170 QTACs were completed, of which 137 were at the provider level and 33 at the

individual level. A provider level QTAC focuses on the overall provider practices and processes versus

an individual level QTAC that focuses on services and supports for one specific person. Table 43

shows the number of QTACs by referral source and type for FY 2019. Because more than one referral

type may be used, totals do not sum to 170. Percentages are calculated from the total for each type

of QTAC (137 or 33). Most QTACs were completed at the provider level (80.6%) and the majority

(63.5%) for a QEPR follow-up. These are initiated internally for the QEPR follow up but once onsite,

the provider may request additional technical assistance.

86.3%

92.7%

84.1%

78.7%

91.7%

85.0%

80.7%

90.0%

86.9%

0% 25% 50% 75% 100%

Person CenteredPractices

Safety

Whole Health

Small (n = 47) Medium (n = 22) Large (n = 28)

95.0%

82.1%

77.6%

92.5%

76.0%

73.7%

92.2%

80.5%

71.8%

0% 25% 50% 75% 100%

Rights

Choice

Community Life


169 | P a g e

Table 43. FY 2019 QTACs by Referral Source and Type

Provider (n = 137) Individual (n = 33)

Referral Source Number Percent Number Percent

Internal 122 89.1% 33 100%

Provider 15 10.9% 0 0.0%

Referral Type:

Quality of Care (PCR/QEPR) 27 19.7% 33 100%

Provider Training 11 8.0% - -

Provider Request 17 12.4% - -

QEPR Follow Up 87 63.5% - -

Table 44 provides a list of detailed reasons for the QTACs completed in FY 2019. More than one

reason can be identified for a QTAC and percentages are based on the total number of QTACs (137 or

33). The most frequently cited reasons at the provider level were related to the QEPR and follow-up

to a Quality of Care Concern. Individual level QTACs were most likely due to a Quality of Care

Concern follow-up, with an additional eight related to a concern about a provider’s safety practices.

Table 44. QTAC Referral Reasons: FY 2019

Provider (n = 137) Individual (n = 33)

Reason Number Percent Number Percent

QEPR Follow Up 88 64.2% - -

Quality of Care Concern follow up 26 19.0% 23 69.7%

Review of person centered documentation 19 13.9% - -

Safety practice concern(s) for the provider 0 0.0% 8 24.2%

Request for 2 day Person Centered Thinking Training 5 3.6% - -

Human rights concern for the person(s) 2 1.5% 2 6.1%

Follow up to previous QTAC 1 0.7% 1 3.0%

Assistance with criminal background checks 1 0.7% - -

Person centered training needed 1 0.7% - -

Total 143 34

Technical assistance (TA) is provided at every QTAC. Approximately 23 to 26 percent of provider level

QTACs offered brainstorming or group discussion. The most common type of technical assistance

offered at the individual level was individual discussion (50.0%) with staff (Figure 49). Discussions


170 | P a g e

typically cover the problems or barriers facing the provider and tactics tried, with further discussion

on other ways to address the issue. Ideas were presented to improve the quality of service delivery

or documentation. Other types of TA provided included different formats for training, brainstorming,

and resources to help address areas that needed improvement.

Figure 49. FY 2018 QTAC Technical Assistance Provided

Intellectual and Developmental Disability Summary of Findings and Recommendations June 2019 marked the completion of the fourth year of the Collaborative contract. Quality Assessors

completed 100 QEPRs, 484 PCRs, and 170 QTACs. Findings throughout this report were used to

address a wide range of requirements and promising practices.

FY 2019 IDD Accomplishments

Several opportunities were provided throughout the year for the IDD Quality Management team to

work in partnership with DBHDD. This collaboration focused on three key initiatives: training for

10.9%

50.0%

10.9%

17.4%

10.9%

4.5%

18.7%

14.9%

9.0%

25.7%

22.8%

3.4%

0% 20% 40% 60% 80% 100%

Role Play

Resources

Individual Discussion

Group Training

Group Discussion

Brainstorming

1:1Training

Provider(N =137)

Individual(N = 33)


171 | P a g e

stakeholders, the new IDD Case Management system, as well as tool and process changes for the

upcoming fiscal year.

Training The Collaborative’s Quality Management team and DBHDD’s Quality Department met bi-weekly and

with DBHDD’s Provider Relations and IDD Leadership monthly to discuss review findings from the

current fiscal year and other identified areas of need across the provider network. Based on

discussion of review findings, training sessions on the six Focused Outcome Areas were developed

and presented throughout the year. The following is a list of topics of trainings provided for

stakeholders in the 2019 fiscal year:

Whole Health

How to Identify Potential Health Risks

Monitoring Health Risks and Protocols

Person Centered

CMS Expectations for the Support Planning Process

Creating a Person Centered Culture in Any Setting

Community Networking with the Community to Build Relationships and Natural Supports

Community Life: How to support community relationships and natural supports

Supporting Individuals to Gain Competitive Employment

Rights How to Identify Rights Violations

How to Support Self Advocacy

Choice What is Informed Choice?

Safety Educating Individuals on Abuse, Neglect & Exploitation


172 | P a g e

Human Services Research Institute (HSRI), a sub-contractor of Qlarant, developed a presentation

from the FY 2016 to 2018 National Core Indicators (NCI) data collected by Collaborative Assessors.26

The focus was on strategies for quality improvement, based on findings from the analysis. The

presentation was provided to DBHDD leadership, with examples of how the NCI data could be

analyzed and used to develop quality improvement initiatives.

IDD Case Management System For two years the Collaborative has assisted in developing a new web-based IDD case management

system (IDD Connects). In preparation for the implementation of the new system, Qlarant was asked

to develop the training to be used to train all external stakeholders who will be utilizing the system.

During this fiscal year, in collaboration with DBHDD, sixteen separate training modules and user

guides were developed for the new IDD Case Management System project, called IDD Connects. The

training modules included PowerPoint presentations and guides, and were developed based upon

different user roles for the new IDD Connects system. Qlarant also trained Beacon staff on the

training modules to help support training efforts across the state. The Collaborative and DBHDD

conducted the first training at the end of the fiscal year for Regional Field Office staff.

Quality Tool and Process Revisions During the year, Qlarant and DBHDD conducted analysis of the tools used for the review processes.

Analysis were conducted for the PRR and included creating a discrimination index, to help determine

which indicators may best predict high vs low scoring providers, and Point-Biserial Correlation to

correlate each indicator with the overall score on the tool.27 Results from these analyses were used

by DBHDD to guide modifications to the tools and processes. On March 29, DBHDD requested the

review processes be modified to meet the following criteria:

Increase reliability and validity of the tools and eliminate redundancy between tools.

26 National Core Indicator website ( https://www.nationalcoreindicators.org/states/GA/) includes specific state reports including Georgia’s survey data since FY 2015. 27 Findings were sent to DBHDD in the excel spreadsheet (Discrimination Analysis revised 021219) and information from this analysis was appended to the Factor Analysis report provided to DBHDD in FY 2018, completed by Qlarant: Discrimination Analysis Intellectual and Developmental Disabilities (IDD) Provider Record Reviews: Quality Enhancement Provider Reviews, July 2016 - June 2018.

https://www.nationalcoreindicators.org/states/GA/


173 | P a g e

Focus on Provider Record Review as the primary source of information while keeping the

voice of the individual through participation in National Core Indicators.

Removal of several review activities (interviews and observations) to decrease administrative

burden on providers.

Apply weights to the eight main components of the new review process when calculating an

overall score for the provider, weighting areas of safety, whole health and person-centered

practices heavier than community life, rights, choice, qualifications and training, and service

guidelines.

Consequently, the Person Centered Review process will be discontinued in FY 2020 as well as several

review processes from the QEPR: the Individual Interview, Staff Interview, Support Coordinator

Interview, and Observations. Workgroups were developed during the last quarter of FY 2019 to

ensure implementation of new tools and processes by July 1, 2019. The workgroups included

representatives of the DBHDD Quality Improvement Office, DBHDD IDD program staff and the

Collaborative and were used to modify record review tools, review procedures, the web-based

application and QEPR reports. Tool revisions included:

Qlarant presented these changes to the provider network during three separate face-to-face

provider meetings around the state and through one webinar.

IDD System Strengths and Recommendations

Findings in this report indicate people receiving IDD services through the DBHDD system were

satisfied with those services, felt safe, and had many health needs met. They (99% or more) had

Tool Updates • Provider Record Review

• Staff Qualifications and Training

Tool Redesign• Support Coordinator Record Review

• DDSS (renamed to Service Guidelines)

New Tool Creation

• Administrative Review


174 | P a g e

access to supports and services to address whole health needs, access to needed medications, and

were free from all types of abuse. Staff providing services appeared to be aware of how individuals

self-preserve and if individuals served felt safe in their work and living environments (99.6%).

The service delivery system appears to promote an atmosphere to enhance aspects of person-

centered practices. Individuals participated in life’s decisions, with all but five of 484 individuals

indicating they were involved in the development of their ISP. They felt they were offered choices (>

94%) and most individuals interviewed indicated they had privacy (99.6%), did not have any rights

restrictions (99.9%), and were treated with respect (99.6%).

Findings on many of the tools for both the PCR and QEPR showed scores of over 90 percent (Table

45). Similar to previous years, documentation from the provider and Support Coordinator record

reviews was the lowest scoring area, during both the PCR and QEPR.

Table 45. FY 2019 Summary by Tool and Review Type

Tool PCR

(N = 484) QEPR 28 (N = 97)

Crisis29 (N = 3)

IOSA - Individual Interview 93.0% 92.3% 91.1%

IOSA - Observation 98.6% 97.2% 96.9%

IOSA - Staff Interview 94.3% 95.7% 97.8%

Provider Record Review 74.9% 73.2% 88.9%

SC Record Review 76.8% 74.4% NA

SC Interview 79.4% 99.7% NA

Admin Q&T NA 83.2% 96.0%

DDSS 100% 100% 100%

FOA scores from both review types (PCR and QEPR) suggest the service delivery system appears to

do relatively well across the six critical areas, showing scores of 75 percent and higher. Since FY 2017,

Safety and Rights have generated the highest scores - over 90 percent each year - suggesting

providers have systems and practices in place to help individuals be safe in their environments and

28 QEPRs completed for the one Support Coordinator agency included 59 SCRRs and 11 Support Coordinator Interviews. 29 Crisis provider QEPRs do not include the SCRR or SCI.


175 | P a g e

exercise and learn about their rights. Individuals receiving services have indicated they do feel safe

(92.4%) and their rights are upheld (98.4%).

Opportunities for Improvement

While overall FOA scores remained relatively high, “drill down” to the standards/indicators

measuring each FOA provides insight into opportunities for developing quality improvement

initiatives or training programs.

Whole Health

While some improvements have been seen in FY 2019, knowledge and education surrounding

medications have remained among the lower scoring areas in Whole Health. Close to half of

individuals interviewed did not understand the side effects of the medications they were taking, and

approximately 31 percent did not know what medications they were taking or why. In addition,

providers were significantly less likely (compared to FY 2018) to document how they offer education

to individuals receiving services on their prescribed medications (27.9%), including their risks and

side effects (37.5%), and were significantly less likely to be able to describe how individuals they

serve were supported to learn about medications (57.3%) or their side effects (68.8%). At the same

time, approximately half of providers had at least one staff member who had not completed training

within required timeframes, specific to medications and their side effects.

Documentation demonstrated Individuals receiving services were also significantly less likely to be

able to maintain personal health, including any necessary supports (49.3%). A lack of knowledge and

education surrounding medications, in combination with an inability to understand how to maintain

health, could lead to serious consequences.

Recommendation XII: The DBHDD Provider Manual for Community Developmental Disability

Providers requires education be provided to individuals on the risks and benefits of anti-psychotic

medications and all other types of prescribed medication and this must be documented in the

clinical record. Training in this area was conducted by the Collaborative in June of 2018 and it is

recommended the training be conducted again and specifically target nursing staff of providers.


176 | P a g e

Recommendation XIII: It is recommended that training sessions for providers include techniques to

effectively offer health education for individuals with all types of communication styles, particularly

on how to maintain their own health. The sessions should include ways to determine if individuals

receiving services understand, to the best level possible, how to maintain personal health and

understand different aspects of the medications they are taking.

Recommendation XIV: It is recommended the Collaborative work with the DBHDD Office of Health

and Wellness to identify initiatives around independence in maintaining health and staff training

around individuals’ medications.

Safety

Without effective training, it is difficult for individuals being served to recognize abusive situations

and without an understanding of how to report them to someone, the abusive activities could go

unreported and cause harm. Information from interviews indicates approximately 35 percent of

individuals receiving services were not aware of what constitutes exploitation or neglect. In addition,

approximately 54 percent of providers had not offered education on how to self-preserve or develop

effective resiliency skills according to the person’s learning style. Over 43 percent of PCRs included a

recommendation related to offering education, according to the person's learning style, on different

types of exploitation, on how to keep safe, and possible actions to take in these situations. In

addition, only 42 percent of PCRs included a recommendation to help individuals receiving services

learn how to manage safety situations by conducting “what if” scenarios.

Support Coordinators are an important advocate for individuals receiving services. If there is a crisis

or behavioral plan in place, it is essential for the coordinator to evaluate the effectiveness of the

plan. If a crisis occurs, it is critical to coordinate supports to prevent a recurrence. Interviews with

Support Coordinators indicated these activities were often not occurring, 44.7 percent and 55.9

percent, respectively.


177 | P a g e

Recommendation XV: The Collaborative and DBHDD could consider developing and providing

training specific to how Support Coordinators ensure systems are in place to evaluate crisis and

behavioral plans, as well as how to evaluate those plans to ensure they are effectively preventing

crisis or if there is a need for additional supports.


Person centered practices should be an essential part of each provider’s service delivery system. To

the extent possible, the individual receiving services should be at the center of all decisions, plans,

and goals. It is important for providers to understand who each individual is and what he or she may

want, hope for, and dream of. What is actually important to anyone receiving services should be

known and incorporated into supports as much as possible. Unfortunately, approximately 32 percent

of ISPs scored in the lowest two rating categories, Needs Improvement or Emerging, demonstrating

they did not ensure all the goals and objectives were person centered or the supports to help with

changes in a person’s life were secured. In over a quarter of the ISPs reviewed, the Communication

Chart did not provide personal information that could be used to best support the individual

receiving services.

As noted, to offer person centered services it is essential to understand what someone wants, strives

for, is particularly good at doing, or finds important. As key advocates for people receiving services,

Support Coordinators are often on the “front line” helping ensure needs and desires are known and

understood. Unfortunately, fewer than half of Support Coordinators interviewed demonstrated a

knowledge of talents or strengths of people they served, or regularly reviewed goals with individuals

receiving services. In addition, providers and Support Coordinators both demonstrated a significant

decrease from FY 2018, documenting how individuals respond to services or supports. A lack of

knowledge about individualized responses to supports will also impede the ability of providing truly

person-centered services.

Recommendation XVI: DBHDD and the Collaborative will be providing training on the changes to the

ISP being implemented in FY 2020. It is recommended additional training be conducted for

providers, specifically on the goals section of the new ISP and how they can help support individuals


178 | P a g e

in developing goals and action plans that are person centered and updated based upon the review of

progress with the person.

Recommendation XVII: One of the key components to implementing a person-centered approach to

planning is to ensure individuals receiving services have the opportunity to review their goals and

make changes as deemed necessary. With implementation of the new IDD Connects system (new

electronic record and ISP) in FY 2020, it will be possible to monitor how and when the person’s ISP

goals are reviewed and changed as needed. A report could be generated from the system to track

how this process is being implemented by region. Findings from this report could guide training the

Collaborative provides, specific to each region as needed.

Recommendation XVIII: FY 2020 will be the first year all ISPs have been transitioned to the “new”

format. Some of the low scoring areas demonstrated in this report may be partially due to a learning

curve for Support Coordinators, individuals and families based upon the new ISP format

implemented in November 2018. If there is no improvement shown in the FY 2020 annual report,

further analysis conducted by the Collaborative is recommended to help determine where quality

appears to be lacking, particularly in ensuring person centered planning is used, ensuring the person

truly has a voice during planning meetings and community engagement is optimized.

Community Life

According to CMS expectations, individuals with intellectual and developmental disabilities should be

able to participate in their communities in the same manner as individuals who do not have a

disability. Community Life remains the lowest scoring area in the PCR and for providers reviewed

through the QEPR. Developing new social roles, such as being a member of a swim team, dance

group or book club, and maintaining those valued roles are instrumental ways to increase

community involvement and integration. Information from individual interviews suggests

approximately 30 percent to 40 percent of individuals receiving services had not been given the

opportunity to learn about or develop new social roles in the community, or had support to maintain

valued social roles.


179 | P a g e

When we first meet someone often the first question we ask is, “what do you do” or “where do you

work?” Many people not only identify themselves and others through their employment but also

expand social connections through contacts at work. Individuals receiving IDD services can also

expand social connections through integrated community employment, enhancing natural or paid

supports. On average, over a three-year time period (FY 2017 – FY 2019) approximately 10 percent of

individuals received Supported Employment services. Data from PCR reviews indicates provider and

Support Coordinator documentation often does not show support for individuals to seek

employment in integrated settings (27.9% and 44.1% respectively).

The ISP should be written to help ensure integration into the community in various settings, as

desired. However, only 8.4 percent of ISPs were written to reflect full participation in the community.

The Likert Scale indicators demonstrated close to 47 percent of ISPs scored in the lower two rating

categories, Emerging or Needs Improvement, showing summaries often did not reflect the person’s

community life. In addition, close to 42 percent of the relationship maps scored in the lowest two

ratings, lacking demonstration of how the person is connected to the community.

Recommendation XIX: DBHDD began contracting with an agency called Wise as an initiative to

increase the number of individuals with IDD who have integrated employment. It is recommended

DBHDD share progress on this initiative through its monthly newsletter and ensure the actions and

plans being made are shared with all stakeholders. This move toward increased awareness could

hopefully lead to increased support from the provider network.

Recommendation XX: There is a new emphasis on employment in the new ISP in IDD Connects, with

additional questions required to be answered. It is recommended the Collaborative ensure data

related to this section are analyzed and reported to DBHDD during regular quality management

meetings. This will help determine if there is improvement in the number of people who are

interested in work who also have goals and action plans in place to address that interest.

Choice

Informed choice is the cornerstone of helping anyone understand and achieve meaningful goals and

direct supports and services. Information from documentation indicates between approximately 50


180 | P a g e

percent and 70 percent of provider and Support Coordinator records lacked evidence informed

choice was provided for competitive or supported employment, living situations, environments,

community participation/social interactions, or for opportunities for education, exploration or

experiences in the community. Close to half of staff interviewed had not provided options for

competitive employment. Individuals interviewed often indicated they were involved in

employment decisions (85.6%); however, provider and Support Coordinator documentation appears

to be what was primarily affecting the low Choice FOA score.

Recommendation XXI: It is recommended the Collaborative’s Quality Management team develop a

training to address how to document choices offered and how providers can support individuals to

make informed decisions. This training would target support staff and Support Coordinators.

Rights

The proper use of restrictive interventions is strictly monitored and enforced by DBHDD. Individuals

interviewed during the review processes did not indicate any violations regarding the improper use

of any type of restrictive intervention or any unauthorized restrictions in their home, community,

work, or day program. They generally feel their rights are upheld. However, records maintained by

providers and Support Coordinators indicated there may be some issues regarding proper

documentation of addressing rights, including: offering ongoing rights education; signed informed

consent forms for medications; and a signed form of notification about rights and responsibilities.

All of these were scored met on fewer than half the records reviewed.

Recommendation XXII: DBHDD could consider requiring Support Coordinators to provide education

to individuals and families on the importance of giving and receiving copies of signed consent forms

for psychotropic medications. Through this, Support Coordinators could ensure consent forms are

being completed and subsequently document this in their own support notes. This should include

education about each individual’s rights and responsibilities.

Recommendation XXIII: DBHDD may want to consider requiring the informed medication consent

document/form be uploaded into the IDD Connects system. If the person receives psychotropic

medication(s) and the form is missing, the system could trigger a “Task” be sent to the Support


181 | P a g e

Coordinator which would be monitored by the Support Coordination agency and DBHDD for

completion.

Additional Recommendations

Provider Documentation

Throughout the findings in this report, documentation by providers and Support Coordinators has

shown consistently lower scores when compared to other information gathering techniques

(interviews or direct observations). This pattern has persisted for several years. For example,

individuals receiving services:

Indicated they had seen a doctor for an annual physical but records often did not include

corresponding documentation

Were free from abuse, neglect and exploitation (ANE) but many providers had not documented

how they identify or address ANE

Indicated they had choice across most areas of their lives but records did not include

corresponding documentation

Seemed to understand their own talents, strengths and goals but most providers did not

document those talents/strengths or how they review progress toward achieving goals with each

individual

Recommendation XXIV: The Collaborative’s Quality Management leadership could develop a

training to specifically address how Support Coordinators can improve support notes and

documentation specific to areas identified throughout the intellectual and developmental disability

section of this report.

Recommendation XXV: Documentation training sessions were developed several years ago and

presented across the state. Over time, however, new providers have joined the DBHDD service

delivery team, new staff have joined existing providers, and new requirements for providers may

have impacted the quality of provider documentation. The Collaborative’s Quality Management

team could identify key areas from this report where documentation is clearly falling behind


182 | P a g e

interview findings and target these specific to training for providers and training for Support

Coordinators.

Quality Enhancement Provider Reviews: Qualifications and Training

Since FY 2017, the average Qualifications and Training score has remained between approximately

83 percent and 84 percent. Some analysis at the indicator level shows that fewer than half of

providers reviewed this year ensured all staff (for the staff records reviewed) had required training

for fire safety, specific medications and their side effects, utilization of suicide prevention skills, and

skill development to work with individuals who have co-occurring diagnoses. In addition, fewer than

half of the providers had received training on proper body mechanics for lifting, transferring or

positioning individuals. Knowledge from these sessions can help ensure the health and safety of

individuals receiving services, and also help protect staff from potential injury.

Recommendation XXVI: It is recommended the Collaborative continue to monitor these results in FY

2020 for future training needs. The Collaborative should continue to provide technical assistance

during QEPRs related to staff training.


183 | P a g e

Section 5: Feedback Survey Results

Following completion of quality reviews for both behavioral health and intellectual and

developmental disabilities, providers are offered the opportunity to complete a feedback survey. For

intellectual and developmental disability services, individuals who participate in the interview are

also offered the opportunity to provide feedback about the process. Surveys are optional and may

be completed by the same agency more than once in a fiscal year. In addition, it is not known who

completes the survey, as individuals and staff are not required to submit their name or provider

agency. A five-point Likert scale is used: strongly agree, somewhat agree, neither agree nor disagree,

somewhat disagree, or strongly disagree.

Table 46 provides information from providers for data entered into the system during FY 2019.

Response rates were generally low, but findings were, again, overwhelmingly positive across all the

review types and from both providers and individuals when compared to previous years. The percent

was calculated as follows: (Strongly Agree + Somewhat Agree) / Total Responses. Results of

indicators for each of the two systems are identified below for annual comparison.

Table 46. FY 2019 Collaborative Provider Feedback Surveys Percent: Strongly Agree + Somewhat Agree/Total Responses

Survey Indicators Result

(N = 221)

Overall, you are satisfied with the review/consultation process. 97%

The Collaborative staff interacted with you and your staff in a professional manner. 98%

The Collaborative staff interacted with the individuals you support in a professional manner.

98%

The Collaborative staff answered your questions and concerns clearly and consistent with DBHDD manual.

98%

The Collaborative staff facilitated an environment which was collaborative and positive. 100%

You would contact the Collaborative staff for technical assistance, training, and resource support, if needed.

99%

The process provided constructive feedback. 98%


184 | P a g e

Table 46. FY 2019 Collaborative Provider Feedback Surveys Percent: Strongly Agree + Somewhat Agree/Total Responses

Survey Indicators Result

(N = 221)

The process helped identify the strengths of your supports and services. 98%

The feedback you received will help provide supports and services that meet the desired outcomes of the individuals you support.

98%

The recommendations generated from this process can be used to make a positive contribution to the individuals served.

99%

The recommendations generated from this process can be used to make a positive contribution to your organization overall.

99%

The feedback provided will assist your organization with making quality improvements to systems and practices.

99%

Overall Survey Result 98%

Only seven responses were received specific to the individual survey completed following an IDD

quality review, all responses resulted in “strongly agree,” a 100 percent score. Comments from

individuals include:

Overall, I am pleased with how the interview went.

The purpose of the interview was explained to me.

I was treated with respect.

The person who interviewed me seemed interested in what I said.

The person who interviewed me was pleasant.

My questions were answered.

The length of the interview was good.

Feedback obtained from the satisfaction surveys provides a qualitative approach to individual and

staff perceptions related to review processes and yielded positive results in FY 2019. Quality

Management will continue to use feedback from the surveys to review processes and provide

professional, effective, and constructive approaches to ensure providers are equipped with the

necessary tools and opportunities they need to best support individuals.


185 | P a g e

Section 6: Conclusion

FY 2019 marked the fourth complete year of the Collaborative Quality Management review process.

During FY 2019, Quality Management completed 3,116 reviews for behavioral health and intellectual

and developmental disability providers. The BHQR Overall score for FY 2019 was 90 percent, a two-

point increase from FY 2018. Additionally, all four BHQR categories increased from the previous year

with Assessment and Planning and Focused Outcome Areas continuing an upward trend since

contract inception.

While there were demonstrated successes noted in all of the different BHQR categories,

opportunities for growth similar to past years continued, especially related to addressing all assessed

needs, co-occurring issues being evaluated and addressed, as well as appropriate discharge planning

criteria documentation. Furthermore, the ACT Service Guidelines score remained steady at 84

percent as a subset of the BHQR reviews with ACT reviews resulting in an average Overall score of 89

percent. Several declines in ACT indicators specific to Billing (FY18 91%; FY19 88%) counteracted the

improvement of the Assessment and Planning score (FY18 87%; FY19 90%), rendering an ACT Overall

score average equivocal to the previous year’s result of 89 percent.

Additionally, FY 2019 marked the third year reassessments were completed for providers who fell

below the revised threshold criteria (90%). The threshold had been raised because of high statewide

averages in FY 2018 and progress at the specific provider level. Twenty-eight (22%) providers were

reviewed for a second BHQR during the year compared to 22 (15%) in the year prior. Only three of

the 19 CSU’s required a reassessment in FY 2019. In review of BHQR scores for these specific

providers at the time of their first FY 2019 review compared to their second FY 2019 review, the

Overall score fell slightly from 89 percent to 88 percent. However, all category scores of the

reassessed BHQRs increased slightly except for Billing, which declined from the time of initial FY

2019 review to the subsequent FY 2019 second review.

While the reassessment of providers is expected to lead to elevated results from year-to-year as a

direct result of ongoing feedback and technical assistance provided, some providers continued to

decline in scoring from first to subsequent review. This is evident in the case of CSUQR


186 | P a g e

reassessments in which two of the three CSUs fell to less than 80 percent by year-end. Moreover, 85

percent of reassessed BHQR providers experienced an increase in the Overall score from initial to

subsequent FY 2018 review, but only 36 percent of providers increased the Overall score during FY

2019. As FY 2019 resulted in fewer providers obtaining improve scores from initial to subsequent

review, it is recommended that the threshold for FY 2020 remain unchanged from FY 2019.

IDD Quality Assessors completed 100 QEPRs, 484 PCRs, and 170 QTACs. On average, findings in this

report reflected the greatest strengths in the areas of rights and safety, and people appeared to be

satisfied with those services. PCR and QEPR interviews with individuals and staff showed scores of

over 90 percent; however, like previous years, provider and Support Coordinator documentation

reflected through record reviews remained on average 76 percent or less. Interviews with the

Support Coordinator showed a decrease from 83.5 percent in FY 2018 to 79.4 percent in FY 2019.

However, a revised Support Coordinator Interview tool was implemented July 1, 2018, and direct

comparisons are not appropriate.

Average statewide PCR FOA scores remained fairly consistent since FY 2017, with the greatest gains

shown in Community Life, up to 77.3 percent. Record review results (PRR and SCRR) were at least 10

points or more lower than other tools for Community Life, Choice and Person Centered Practices and

PRR and SC Interview results were 10 points or more lower than other tools measuring Safety.

Results for FOA by region, residence and funding source produced the following findings:

Individuals living in Region 6 were significantly less likely to be participating their Community

Life as desired (p< .001), compared to their counterparts living in other regions.30

Individuals living in Region 2 were significantly less likely to have Choice in their lives

(p< .001), compared to their counterparts living in other regions.

Individuals receiving services and living with a parent scored significantly lower in Whole

Health than their counterparts in any other residential setting (p < .001).

30 It should be noted the difference of proportions test was based on the number of indicators scored; therefore, the “n” sizes were relatively large, increasing the likelihood of statistical significance.


187 | P a g e

Receiving services while living in a group home, showed a significantly lower likelihood of

being integrated into the community as desired when compared to other residential settings

(p<.001).

Results for the COMP waiver indicated individuals were more likely to have Whole Health

indicators met compared to NOW or SFS funding sources (p < .01).

The overall QEPR scores were also fairly consistent across the same timeframe, 83.7 percent to 86.2

percent. The Qualifications and Training component of the QEPR has reflected scores of 83 percent

to 84 percent since FY 2017.

To reduce administrative burden to providers, Quality Management conducted joint BHQR, CSUQR

and QEPRs as often as possible. Fourteen joint reviews were completed during FY 2019, four more

than the previous year, with 11 conducted at Community Service Board locations. By combining

BHQR and IDD activities, the Collaborative has promoted additional education for the network and

also among Assessors, further strengthening expertise and collaboration. Through the joint

discussions offered at exit conferences, that include both BHQR and IDD providers and Assessors,

joint reviews enhanced valuable integration of information and feedback and therefore this

approach will continue throughout FY 2020.

Provider trainings in FY 2020 will continue to occur both in person as well as via webinar through the

formalized Quality Training Program based on collaboration and partnership with DBHDD. Quality

Management shall use findings from the behavioral health, CSU, and intellectual and developmental

disability reviews to recommend topics for trainings. Such trainings will be prioritized based on

identified system needs.

Throughout the revisions to the IDD tools, the Collaborative explored alignment of both the BH and

IDD review processes. The revision to the FY 2020 IDD tools will further align indicators between BH

and IDD. Previously, similarity between the two review types (BHQR/IDD) included only Focused

Outcome Areas categories, but item-level indicators remained different between the populations.

An addition of an IDD Service Guidelines section is anticipated for FY 2020. The Service Guideline


188 | P a g e

section of the IDD tool will extrapolate data similar to BHQRs/CSUQRs whereby a set of

indicators/standards are to be measured only once per service.

Annual analyses and recommendations discussed throughout this report drive statewide BH and IDD

quality initiatives for the department’s internal and external review processes. Analysis also drives

efforts specific to targeted provider training and technical assistance/support to those who serve and

treat the individuals in need. Therefore, Quality Management will continue to collaborate and work

diligently with both its client, DBHDD, as well as the Georgia provider network to continue facilitating

the delivery of whole-health, person-centered, and quality supports and services to individuals and

their families throughout the state.


189 | P a g e

Appendix A: Abbreviations and Acronyms

Acronym Definition ACT Assertive Community Treatment

AD Addictive Diseases

ADSS Addictive Disease Support Services

AIMS Abnormal Involuntary Movement Scale

ANE Abuse, neglect, and exploitation

BH Behavioral Health

BHQR Behavioral Health Quality Review

C&A Child and Adolescent

CIS Consumer Information System

CMS Centers for Medicaid and Medicare Services

CPRP Certified Peer Recovery Specialist

CPS Certified Peer Specialist

CPS-WH Certified Peer Specialist – Whole Health

C-SSRS Columbia Suicide Severity Rating Scale

CST Community Support Team

CSU Crisis Stabilization Unit

CSUQR Crisis Stabilization Unit Quality Review

DBHDD Department of Behavioral Health and Developmental Disabilities

DDSS Developmental Disability Service Specific

FOA Focused Outcome Area(s)

FY Fiscal Year

GLC Georgia Learning Community

HSRI Human Services Research Institute

II Individual Interview

IDD Intellectual and Developmental Disability

IFI Intensive Family Intervention

IOP Intensive Outpatient

IOSA Individual Observation Staff Assessment

IRR Individual Record Review

IRP Individual Recovery / Resiliency Plan

ISP QA Individual Service Plan Quality Assurance Checklist

ISP Individual Service Plan

KPI Key Performance Indicator

MAR Medication Administration Record

MAT Medication-Assisted Treatment

MH Mental Health


190 | P a g e

N “n” Number in sample (number in subset of sample)

NA Not Applicable

NCI National Core Indicators

NCP Nursing Care Plan

NIOP Non-Intensive Outpatient

NR Not Reviewed

ODESA Online Data Entry Survey Application

OBS On-site Observations

ODS Office of Deaf Services

PCP Person Centered Practices

PCR Person Centered Review

PHI Protected Health Information

PRR Provider Record Review

QEPR Quality Enhancement Provider Review

QM Quality Management

Q&T Qualifications and Training

QTAC Quality Technical Assistance Consultation

R “r” Number of reviews

RN Registered Nurse

SC Support Coordinator

SCI Support Coordinator Interview

SCRR Support Coordinator Record Review

SFS State Funded Services

SI Staff Interview

SIS Supports Intensity Scale

SU Substance Use

TA Technical Assistance


191 | P a g e

Appendix B: Score Distributions

Graphical distributions for each category show the percent of providers who scored within the specified range of scores,

as indicated on the horizontal/x-axis. With each distribution, the median, mean, and mode may be provided. The median

represents the score that falls in the middle of distribution (50th percentile) and the mean represents the average score.

Standard deviation refers to the amount of variation or dispersion there is in a distribution of scores, or how much scores

tend to spread-out from the mean. Standard deviations are sometimes used to determine significant differences between

scores within a distribution; however, this application requires a distribution to be normally distributed (similar to a bell

curve). Most of the distributions presented in this report do not meet the requirements of a normal distribution, meaning

they do not look like a normal bell curve; therefore, standard deviations have been excluded from the report at this time.

Table of Contents (Links available by clicking on Figure Name)

Behavioral Health Quality Review Distributions

BHQR Overall Score Distribution

BHQR Tier 1 Overall Score Distribution


BHQR Tier 2+ Overall Score Distribution


BHQR Billing Score Distribution

BHQR Assessment & Treatment Score Planning Distribution

BHQR Service Guidelines Score Distribution

BHQR Focused Outcome Areas Score Distribution

Assertive Community Treatment (ACT) Quality Review Distributions

ACT Overall Score Distribution

ACT Billing Score Distribution

ACT Assessment & Planning Distribution

ACT Service Guidelines Score Distribution

ACT Focused Outcome Areas Score Distribution


192 | P a g e

CSU Quality Review Distributions

CSU Overall Score Distribution

CSU Service Guidelines Score Distribution

CSU Focused Outcome Areas Score Distribution

CSU Individual Record Review Score Distribution

Reassessment BHQR Distributions

BHQR Overall Score Distribution of Reassessed Providers

BHQR Billing Score Distribution of Reassessed Providers

BHQR Assessment & Treatment Planning Distribution of Reassessed Providers

BHQR Service Guidelines Score Distribution of Reassessed Providers

BHQR Focused Outcome Areas Score Distribution of Reassessed Providers

PCR Distributions: Scores by Tool and Year

PCR Individual Interview Scores by Year

PCR Observation Scores by Year

PCR Staff Interview Scores by Year

PCR Provider Record Review Scores by Year

PCR Support Coordinator Review Scores by Year

PCR Support Coordinator Interview Scores by Year

QEPR Distributions: Scores by Tool and Year

FY 2019 Overall QEPR Score

FY 2019 QEPR Individual Interview Scores

FY 2019 QEPR Observation Scores

FY 2019 QEPR Staff Interview Scores

FY 2019 QEPR Provider Record Review Scores

FY 2019 QEPR Qualifications and Training Scores

FY 2019 QEPR Service Discrepancy Scores

FY 2019 QEPR Overall Scores by Provider Size

FY 2019 QEPR Qualifications and Training Scores by Provider Size


193 | P a g e

BHQR Overall Score Distribution by Year

BHQR Tier 1 Overall Score Distribution by Year

0% 1% 2%5%

19%

49%

25%

1% 2% 3%6%

42%47%

1% 1%3% 3%

32%

61%

0%

10%

20%

30%

40%

50%

60%

70%

80%

FY 2017 (N = 167; Mean = 84%; Median = 86%)

FY 2018 (N = 156; Mean = 88%; Median = 90%)

FY 2019 (N = 157; Mean = 90%; Median = 92%)

21%

58%

25%

3%

48% 48%

4%

59%

37%

0%

10%

20%

30%

40%

50%

60%

70%

80%

FY 2017 (N = 28; Mean = 85%; Median = 85%)

FY 2018 (N = 29; Mean = 88%; Median = 90%)

FY 2019 (N = 27; Mean = 88%; Median = 89%)


194 | P a g e


BHQR Tier 2+ Overall Score Distribution by Year

2%5%

19%

50%

24%

1% 2%6%

41%

49%

1% 1% 4%

25%

70%

0%

10%

20%

30%

40%

50%

60%

70%

80%

FY 2017 (N = 115; Mean = 85%; Median = 86%)

FY 2018 (N = 93; Mean = 89%; Median = 90%)

FY 2019 (N = 102; Mean = 91%; Median = 93%)

83%

17%

83%

17%

33%

67%

0%

20%

40%

60%

80%

100%

FY 2017 (N = 6; Mean = 86%; Median = 85%)

FY 2018 (N = 6; Mean = 87%; Median = 86%)

FY 2019 (N = 6; Mean = 92%; Median = 92%)


195 | P a g e


BHQR Billing Score Distribution by Year

6% 6%11% 11% 11%

56%

7% 7% 7% 7%

29%

43%

9%13%

30%

43%

0%

20%

40%

60%

80%

100%

FY 2017 (N = 18; Mean = 83%; Median = 91%)

FY 2018 (N = 28; Mean = 83%; Median = 89%)

FY 2019 (N = 22; Mean = 84%; Median = 91%)

1% 2% 1% 1% 2% 4% 5%8%

28%

48%

1% 2% 1% 1% 1% 1% 3%

11%

26%

53%

1% 1% 3% 1% 2% 3%8%

29%

54%

0%

10%

20%

30%

40%

50%

60%

70%

FY 2017 (N = 167; Mean = 84%; Median = 90%)

FY 2018 (N = 156; Mean = 85%; Median = 92%)

FY 2019 (N = 157; Mean = 86%; Median = 92%)


196 | P a g e

BHQR Assessment & Treatment Planning Score Distribution by Year

BHQR Compliance with Service Guidelines Score Distribution by Year

1% 1%5%

23%

31% 31%

9%

1% 1% 2%

7%

21%

40%

29%

1% 1% 1%4%

8%

38%

47%

0

0.1

0.2

0.3

0.4

0.5

FY 2017 (N = 167; Mean = 77%; Median = 78%)

FY 2018 (N = 156; Mean = 84%; Median = 86%)

FY 2019 (N = 157; Mean = 88%; Median = 90%)

1% 1%7% 7%

31%

53%

1% 1%4% 3%

29%

62%

1% 1% 1% 1% 3% 4%

27%

64%

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

FY 2017 (N = 167; Mean = 88%; Median = 91%)

FY 2018 (N = 156; Mean = 90%; Median = 93%)

FY 2019 (N = 157; Mean = 90%; Median = 92%)


197 | P a g e

BHQR Focused Outcome Areas Score Distribution by Year

ACT Overall Score Distribution by Year

2%8%

41%50%

2% 3%

28%

67%

1%5%

11%

83%

0%

20%

40%

60%

80%

100%

FY 2017 (N = 167; Mean = 89%; Median = 90%)

FY 2018 (N = 156; Mean = 92%; Median = 93%)

FY 2019 (N = 157; Mean = 94%; Median = 95%)

26% 25%

50%

5%

48% 48%

10%

40%

50%

0%

10%

20%

30%

40%

50%

60%

FY 2017 (N = 19; Mean = 87%; Median = 91%)

FY 2018 (N = 21; Mean = 89%; Median = 90%)

FY 2019 (N = 20; Mean = 89%; Median = 90%)


198 | P a g e

ACT Billing Score Distribution by Year

ACT Assessment & Planning Score Distribution by Year

5%

15%20%

60%

5% 10%

19%

67%

5%

20%25%

50%

0%

10%

20%

30%

40%

50%

60%

70%

80%

FY 2017 (N = 19; Mean = 90%; Median = 96%)FY 2018 (N = 21; Mean = 91%; Median = 95%)FY 2019 (N = 20; Mean = 88%; Median = 90%)

20%

35%40%

5%

19%

43%38%

10%

0%

30%

60%

0%

10%

20%

30%

40%

50%

60%

70%

FY 2017 (N = 19; Mean = 80%; Median = 80%)

FY 2018 (N = 21; Mean = 87%; Median = 87%)

FY 2019 (N = 20; Mean = 90%; Median = 92%)


199 | P a g e

ACT Service Guidelines

Score Distribution by Year

ACT FOAs Score Distribution by Year

5%10%

50%

35%33%38%

29%

10%15%

55%

20%

0%

10%

20%

30%

40%

50%

60%

FY 2017 (N = 19; Mean = 88%; Median = 89%)

FY 2018 (N = 21; Mean = 84%; Median = 86%)

FY 2019 (N = 20; Mean = 84%; Median = 87%)

10%

35%

55%

19%

81%

30%

70%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

FY 2017 (N = 19; Mean = 90%; Median = 91%)

FY 2018 (N = 21; Mean = 93%; Median = 95%)

FY 2019 (N = 20; Mean = 93%; Median = 93%)


200 | P a g e

CSU Overall Score Distribution by Year

CSU Compliance with Service Guidelines Score Distribution by Year

4%

17%

43%

35%

5% 5%

48%43%

18%

55%

27%

0%

10%

20%

30%

40%

50%

60%

FY 2017 (N = 23; Mean = 86%; Median = 88%)

FY 2018 (N = 21; Mean = 88%; Median = 90%)

FY 2019 (N = 22; Mean = 87%; Median = 87%)

4%

17%

43%

35%

5% 5%

48%

43%

14%

23%

18%

45%

0%

10%

20%

30%

40%

50%

60%

FY 2017 (N = 23; Mean = 87%; Median = 91%)

FY 2018 (N = 21; Mean = 91%; Median = 100%)

FY 2019 (N = 22; Mean = 89%; Median = 90%)


201 | P a g e

CSU Focused Outcome Areas Overall Distribution by Year

CSU Individual Record Review Score Distribution by Year

9%

35%

57%

48%52%

23%

77%

0%

10%

20%

30%

40%

50%

60%

70%

80%

FY 2017 (N = 23; Mean = 91%; Median = 92%)FY 2018 (N = 21; Mean = 91%; Median = 91%)FY 2019 (N = 22; Mean = 93%; Median = 95%)

4% 4%

26%

65%

5% 5%

29%33%

29%

18%

27%

50%

5%

0%

10%

20%

30%

40%

50%

60%

70%

FY 2017 (N = 23; Mean = 82%; Median = 82%)

FY 2018 (N = 21; Mean = 83%; Median = 83%)

FY 2019 (N = 22; Mean = 79%; Median = 83%)


202 | P a g e

FY 2019 BHQR Overall Score Distribution of Reassessed Providers

FY 2019 BHQR Billing Score Distribution of Reassessed Providers

4% 4%

43%50%

7% 4%

39%

50%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



4% 7%

50%

32%

11%7%

11%

43%

29%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1st BHQR (N = 28; Mean = 87%; Median 89%)



203 | P a g e

FY 2019 BHQR Assessment & Planning Score Distribution of Reassessed Providers

FY 2019 BHQR Service Guidelines Score Distribution of Reassessed Providers

4% 7% 7% 11%

32%39%

7% 7%

39%46%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



4% 4% 4%

32%

57%

7%4%

21%

68%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




204 | P a g e

FY 2019 BHQR FOAs Score Distribution of Reassessed Providers

PCR Individual Interview Scores by Year

4%

18%

79%

4%

18%

79%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



1%6%

22%

66%

5%2% 5%

20%

62%

11%

1% 5%

17%

66%

11%

0%

10%

20%

30%

40%

50%

60%

70%

FY 2017 (N = 481; Mean = 91.9%; Median = 93.7%)

FY 2018 (N = 484; Mean = 92.2%; Median = 95.1%)

FY 2019 (N = 484; Mean = 93.0%; Median = 94.9%)


205 | P a g e

PCR Observation Scores by Year

PCR Staff Interview Scores by Year

4% 6%

52%

38%

<1% 1%

42%

57%

3%

41%

57%

0%

10%

20%

30%

40%

50%

60%

70%

FY 2017 (N = 181; Mean =96.8%; Median = 98.7%)

FY 2018 (N = 276; Mean =98.2%; Median = 100%)

FY 2019 (N = 274; Mean = 98.6%; Median = 100.0%)

<1% 1% 2%

11%

69%

16%

3%6%

67%

24%

1%

12%

66%

21%

0%

10%

20%

30%

40%

50%

60%

70%

FY 2017 (N = 224; Mean = 94.3%; Median = 96.0%)

FY 2018 (N = 312; Mean = 95.6%; Median = 97.1%)

FY 2019 (N = 323; Mean = 94.3%; Median = 96.2%)


206 | P a g e

PCR Provider Record Review Scores by Year

PCR Support Coordinator Record Review Scores by Year

<1%1%

5%

14%

27%29%

18%

6%

<1%1%

4%

9%

16%

28% 29%

11%

2%<1%

2%

8%

24%

28%25%

11%

2%

0%

5%

10%

15%

20%

25%

30%

FY 2017 (N = 478; Mean = 70.2%; Median = 70.5%)

FY 2018 (N = 483; Mean = 74.8%; Median = 75.9%)

FY 2019 (N = 478; Mean = 74.9%; Median = 74.3%)

1%1% 3% 5%

10%

16%

20%

24%

17%

2%1%

<1%1% 2% 2%

9%

13%

19%

25% 24%

3%

1% <1%2%

3%

7%

12%

25%

27%

19%

2%

0%

5%

10%

15%

20%

25%

30%

FY 2017 (N = 481; Mean = 73.7%; Median = 76.4%)

FY 2018 (N = 484; Mean = 77.4%; Median = 75.9%)

FY 2019 (N = 484; Mean = 76.8%; Median = 78.9%)


207 | P a g e

PCR Support Coordinator Interview Scores by Year

FY 2019 Overall QEPR Score

11%

1% 2%7%

45%

33%

12%

2%5%

39%42%

13%

1% 1% 1%6%

12%18%

48%

0%

10%

20%

30%

40%

50%

60%

FY 2017 (N = 479; Mean = 83.3%; Median = 96.7%)

FY 2018 (N = 484; Mean = 83.5%; Median = 98.4%)

FY 2019 (N = 484; Mean = 79.4%; Median = 96.4%)

12%

60%

28%

0%

20%

40%

60%

80%

100%

(N = 97; Mean = 86.2%; Median = 86.8%)


208 | P a g e

FY 2019 QEPR Individual Interview Scores

FY 2019 QEPR Observation Scores

2%

30%

66%

2%0%

20%

40%

60%

80%

100%

(N = 96; Mean = 92.3%; Median = 92.4%)

1% 1%5%

83%

10%

0%

25%

50%

75%

100%

(N = 82; Mean = 97.2%; Median = 97.6%)


209 | P a g e

FY 2019 QEPR Staff Interview Scores

FY 2019 QEPR Record Review Scores

1%8%

80%

10%

0%

20%

40%

60%

80%

100%

(N = 97; Mean = 95.3%; Median = 96.3%)

1% 1%4%

12%

28%34%

14%

5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

(N = 97; Mean = 73.2%; Median = 70.1%)


210 | P a g e

FY 2019 QEPR Qualifications and Training Scores

FY 2019 QEPR Service Discrepancy Scores

1% 2% 3%10%

16%

28%33%

6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

(N = 97; Mean = 83.2%; Median = 85.9%)

8%

92%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

(N = 97; Mean = 99.8%; Median = 100.0%)


211 | P a g e

FY 2019 QEPR Overall Scores by Provider Size

17%

60%

23%

18%

73%

9%

4%

68%

29%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Small (n = 47; Mean = 85.6%; Median = 86.2%)

Medium (n = 22; Mean = 84.6%; Median = 84.7%)

Large (n = 28; Mean = 87.2%; Median = 87.4%)


212 | P a g e

FY 2019 QEPR Qualifications and Training Scores by Provider Size

2% 2%

4%

9%

15%

26%

32%

11%

23% 23%

32%

23%

4% 4% 4%

18%

29%

39%

4%

0%

10%

20%

30%

40%

50%

Small (n = 47; Mean = 84.5%; Median = 86.9%)

Medium (n = 22; Mean = 80.4%; Median = 83.2%)

Large (n = 28; Mean = 84.0%; Median = 84.7%)

Date post:	03-Aug-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

FY 2019 Quality Management Annual Report · 2020-01-27 · Quality Management Annual Report FY 2019...

Documents