Page 1
FY 16-17
Medi-Cal Specialty Mental Health
External Quality Review
MHP Final Report
Prepared by:
Behavioral Health Concepts, Inc.
5901 Christie Avenue, Suite 502
Emeryville, CA 94608
www.caleqro.com
Colusa County Conducted on
August 4, 2016
Page 2
TABLE OF CONTENTS
COLUSA MENTAL HEALTH PLAN SUMMARY FINDINGS ............................................................................................. 4
INTRODUCTION ..................................................................................................................................................................... 6
PRIOR YEAR REVIEW FINDINGS, FY15-16 ..................................................................................................................... 8
STATUS OF FY15-16 REVIEW RECOMMENDATIONS ................................................................................................................................ 8 Assignment of Ratings ............................................................................................................................................................................... 8 Key Recommendations from FY15-16 ................................................................................................................................................ 8
CHANGES IN THE MHP ENVIRONMENT AND WITHIN THE MHP—IMPACT AND IMPLICATIONS .................................................... 11
PERFORMANCE MEASUREMENT .................................................................................................................................... 13
TOTAL BENEFICIARIES SERVED................................................................................................................................................................... 13 PENETRATION RATES AND APPROVED CLAIM DOLLARS PER BENEFICIARY ....................................................................................... 14
PERFORMANCE IMPROVEMENT PROJECT VALIDATION ....................................................................................... 22
COLUSA MHP PIPS IDENTIFIED FOR VALIDATION .................................................................................................................................. 22 CLINICAL PIP—RECOVERY .......................................................................................................................................................................... 24 NON-CLINICAL PIP—PARENT INTERVENTION ........................................................................................................................................ 26 PERFORMANCE IMPROVEMENT PROJECT FINDINGS—IMPACT AND IMPLICATIONS .......................................................................... 27
PERFORMANCE & QUALITY MANAGEMENT KEY COMPONENTS ......................................................................... 28
Access to Care .............................................................................................................................................................................................. 28 Timeliness of Services ............................................................................................................................................................................... 29 Quality of Care ............................................................................................................................................................................................. 31
KEY COMPONENTS FINDINGS—IMPACT AND IMPLICATIONS ................................................................................................................ 35
CONSUMER AND FAMILY MEMBER FOCUS GROUP(S) ............................................................................................ 37
CONSUMER/FAMILY MEMBER FOCUS GROUP 1 ...................................................................................................................................... 37 CONSUMER/FAMILY MEMBER FOCUS GROUP FINDINGS—IMPLICATIONS ......................................................................................... 38
INFORMATION SYSTEMS REVIEW ................................................................................................................................. 39
KEY ISCA INFORMATION PROVIDED BY THE MHP ................................................................................................................................. 39 CURRENT OPERATIONS ................................................................................................................................................................................. 40 PLANS FOR INFORMATION SYSTEMS CHANGE .......................................................................................................................................... 41 ELECTRONIC HEALTH RECORD STATUS ..................................................................................................................................................... 41 MAJOR CHANGES SINCE LAST YEAR ........................................................................................................................................................... 42 PRIORITIES FOR THE COMING YEAR ........................................................................................................................................................... 42 OTHER SIGNIFICANT ISSUES ........................................................................................................................................................................ 43 MEDI-CAL CLAIMS PROCESSING ................................................................................................................................................................... 43 INFORMATION SYSTEMS REVIEW FINDINGS—IMPLICATIONS ............................................................................................................... 43
SITE REVIEW PROCESS BARRIERS ................................................................................................................................ 45
CONCLUSIONS ...................................................................................................................................................................... 46
STRENGTHS AND OPPORTUNITIES .............................................................................................................................................................. 46 Access to Care .............................................................................................................................................................................................. 46 Timeliness of Services ............................................................................................................................................................................... 46 Quality of Care ............................................................................................................................................................................................. 47 Consumer Outcomes ................................................................................................................................................................................. 47
Page 3
RECOMMENDATIONS ..................................................................................................................................................................................... 48
ATTACHMENTS ................................................................................................................................................................... 49
ATTACHMENT A—REVIEW AGENDA .............................................................................................................................................................. ATTACHMENT B—REVIEW PARTICIPANTS ................................................................................................................................................... ATTACHMENT C—APPROVED CLAIMS SOURCE DATA ................................................................................................................................ ATTACHMENT D—PIP VALIDATION TOOL ...................................................................................................................................................
Page 4
COLUSA MENTAL HEALTH PLAN SUMMARY FINDINGS
Beneficiaries served in CY15—523
MHP Threshold Language(s)—Spanish
MHP Size—Small Rural
MHP Region—Superior
MHP Location— Colusa
MHP County Seat—Colusa, CA
MHP-Reported Significant Changes
The MHP hired a new Quality Improvement (QI) Coordinator who will be responsible
for data analysis and trending.
In the spring of 2016, a county administrative officer was hired, with MHP leadership as
a direct report.
The MHP has experienced difficulties with crisis protocols since April 2016 with the
closure of the only hospital in the county.
The MHP has had great difficulty in working with Anthem for Affordable Care Act (ACA)
referrals. Anthem does not seem to have any providers that have openings, and they are
not very responsive to consumers and staff calls.
Performance Measurement Findings from CY15 Claims Data
The MHP’s percentage of high-cost beneficiaries was similar to statewide and showed
an increase from CY14. Its percentage of HCB claim dollars was higher than statewide
and doubled from CY14 to CY15.
Information Systems Findings
The MHP plans to install and begin implementation of a Telechart Patient Portal
following system upgrade in September 2016.
The MHP continues to pursue Meaningful Use, Stage I qualification through
Medicare/Medicaid’s EHR incentive program with a projected target date of October
2016. The MHP is working towards electronic capture of its clinical data and the use of
health IT for continuous quality improvement at the point of care and the exchange of
information.
Page 5
MHP has no current plans to expand EHR functionality with the addition of eLabs, Level
of Care/Level of Service assessments.
Strengths and Recommendations Findings
Investigate the feasibility to provide more funding or resources to support crisis staff.
Evaluate contributing factors for the increase in hospitalization rates from FY 14/15 to
FY15/16.
Re-establish and formalize the Cultural Competence Committee with regularly
scheduled meetings.
Track timeliness for psychiatry for both children and adults, to determine the impact of
the .5 FTE elimination of child psychiatry and the implementation of the policy
requiring conjoint services for medication child clients.
Expand EHR functionality through consultation with Kings View and nearby counties.
Quantifiably show improvement and expansion through the addition of applications,
such as eLabs, Level of Care/Level of Service, Alerts, etc.
Page 6
INTRODUCTION
The United States Department of Health and Human Services (DHHS), Centers for Medicare and
Medicaid Services (CMS) requires an annual, independent external evaluation of State Medicaid
Managed Care programs by an External Quality Review Organization (EQRO). External Quality
Review (EQR) is the analysis and evaluation by an approved EQRO of aggregate information on
quality, timeliness, and access to health care services furnished by Prepaid Inpatient Health Plans
(PIHPs) and their contractors to recipients of Managed Care services. The CMS (42 CFR §438;
Medicaid Program, External Quality Review of Medicaid Managed Care Organizations) rules specify
the requirements for evaluation of Medicaid Managed Care programs. These rules require an on-
site review or a desk review of each Medi-Cal Mental Health Plan (MHP).
The State of California Department of Health Care Services (DHCS) contracts with fifty-six (56)
county Medi-Cal MHPs to provide Medi-Cal covered specialty mental health services to Medi-Cal
beneficiaries under the provisions of Title XIX of the federal Social Security Act.
This report presents the fiscal year 2016-2017 (FY 16-17) findings of an EQR of the Colusa
MHP by the California External Quality Review Organization (CalEQRO), Behavioral Health
Concepts, Inc. (BHC).
The EQR technical report analyzes and aggregates data from the EQR activities as described below:
(1) VALIDATING PERFORMANCE MEASURES1
This report contains the results of the EQRO’s validation of eight (8) Mandatory Performance
Measures (PM) as defined by DHCS. The eight performance measures include:
Total Beneficiaries Served by each county MHP
Total Costs per Beneficiary Served by each county MHP
Penetration Rates in each county MHP
Count of TBS Beneficiaries Served Compared to the four percent (4%) Emily Q.
Benchmark (not included in MHP reports; this information is included in the Annual
Statewide Report submitted to DHCS).
Total Psychiatric Inpatient Hospital Episodes, Costs, and Average Length of Stay
Psychiatric Inpatient Hospital 7-Day and 30-Day Rehospitalization Rates
Post-Psychiatric Inpatient Hospital 7-Day and 30-Day Specialty Mental Health Services
(SMHS) Follow-Up Service Rates
1 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). Validation
of Performance Measures Reported by the MCO: A Mandatory Protocol for External Quality Review (EQR),
Protocol 2, Version 2.0, September, 2012. Washington, DC: Author.
Page 7
High Cost Beneficiaries ($30,000 or higher)
(2) VALIDATING PERFORMANCE IMPROVEMENT PROJECTS2
Each MHP is required to conduct two Performance Improvement Projects (PIPs) during the 12
months preceding the review; Colusa MHP submitted two PIPs for validation through the EQRO
review. The PIP(s) are discussed in detail later in this report.
(3) MHP HEALTH INFORMATION SYSTEM CAPABILITIES3
Utilizing the Information Systems Capabilities Assessment (ISCA) protocol, the EQRO reviewed and
analyzed the extent to which the MHP meets federal data integrity requirement for Health
Information Systems (HIS), as identified in 42 CFR §438.242. This evaluation included review of
the MHP’s reporting systems and methodologies for calculating Performance Measures (PM).
(4) VALIDATION OF STATE AND COUNTY CONSUMER SATISFACTION SURVEYS
The EQRO examined available consumer satisfaction surveys conducted by DHCS, the MHP or its
subcontractors.
CalEQRO also conducted one 90-minute focus group with beneficiaries and family members to
obtain direct qualitative evidence from beneficiaries.
(5) KEY COMPONENTS, SIGNIFICANT CHANGES, ASSESSMENT OF STRENGTHS,
OPPORTUNITIES FOR IMPROVEMENT, RECOMMENDATIONS
The CalEQRO review draws upon prior year’s findings, including sustained strengths, opportunities
for improvement, and actions in response to recommendations. Other findings in this report
include:
Changes, progress, or milestones in the MHP’s approach to performance management—
emphasizing utilization of data, specific reports, and activities designed to manage and
improve quality.
Ratings for Key Components associated with the following three domains: access,
timeliness, and quality. Submitted documentation as well as interviews with a variety of
key staff, contracted providers, advisory groups, beneficiaries, and other stakeholders
serve to inform the evaluation of MHP’s performance within these domains. Detailed
definitions for each of the review criteria can be found on the CalEQRO Website
www.caleqro.com.
2 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). Validating
Performance Improvement Projects: Mandatory Protocol for External Quality Review (EQR), Protocol 3,
Version 2.0, September 2012. Washington, DC: Author. 3 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). EQR
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations: A Mandatory Protocol for
External Quality Review (EQR), Protocol 1, Version 2.0, September 1, 2012. Washington, DC: Author.
Page 8
PRIOR YEAR REVIEW FINDINGS, FY15-16
In this section we discuss the status of last year’s (FY15-16) recommendations, as well as changes
within the MHP’s environment since its last review.
STATUS OF FY15-16 REVIEW RECOMMENDATIONS
In the FY15-16 site review report, the CalEQRO made a number of recommendations for
improvements in the MHP’s programmatic and/or operational areas. During the FY16-17 site visit,
CalEQRO and MHP staff discussed the status of those FY15-16 recommendations, which are
summarized below.
Assignment of Ratings
Fully addressed—
o resolved the identified issue
Partially addressed—Though not fully addressed, this rating reflects that the MHP has
either:
o made clear plans and is in the early stages of initiating activities to address the
recommendation
o addressed some but not all aspects of the recommendation or related issues
Not addressed—The MHP performed no meaningful activities to address the
recommendation or associated issues.
Key Recommendations from FY15-16
Recommendation #1: As recommended last year, collaborate with County
Administration to recruit the unfilled Data Coordinator position in order to continue to
move forward in the development of a data driven system of care.
☐ Fully addressed ☒ Partially addressed ☐ Not addressed
o This MHP has not filled its data analyst and/or an equivalent position within
their county system. The MHP has attempted to obtain this position through the
budget process without success.
o Alternatively, the MHP has recruited a Quality Improvement Coordinator who is
a licensed clinician to begin in August 2016. The QI coordinator may perform
analysis of data such as penetration rates, billing, access, demographics, etc.
Recommendation #2: Make staffing and space priorities for the southern portion of the
county. Track all timeliness measures reflected in the Timeliness Self-Assessment form,
for all regions for adult and children’s services. Allocate staffing and fiscal resources
Page 9
that are commensurate with the capacity demands, prioritizing the amelioration of wait
lists and timeliness issues in South County.
☐ Fully addressed ☐ Partially addressed ☒ Not addressed
o This Item was not addressed as this recommendation was made in error and not
corrected before the report was finalized last year.
Recommendation #3: Reinvigorate the Cultural Competence Committee and
reestablish routine cultural competence activities and robust outreach efforts that
involve consumers/family members on both a larger scale (community feedback at
meetings, surveys) and also on a smaller scale (i.e., committee level planning and
implementation of outreach). Evaluate the effectiveness of the activities. Establish a
diverse leadership of cultural competency and outreach levels to ensure continuous
efforts.
☐ Fully addressed ☒ Partially addressed ☐ Not addressed
o Although the MHP does not formally have a Cultural Competence Committee, the
MHP reformed its approach to carrying out the cultural competency goals. This
occurred due to the departure of the ethnic services manager the prior year.
o The MHP implemented a significant outreach effort to the Hispanic community
titled Cultura Es Vida in October 2015 with the Hispanic Heritage Program. The
MHP collaborated with the English Learner Advisory Committees to provide
twelve presentations to the preschools and schools addressing the
understanding of mental health issues. One hundred consumer surveys were
completed at the last presentation.
o The MHP has negotiated a Memorandum of Understanding (MOU) to increase
direct services to the Native American Community.
o The MHP has added bilingual Spanish speaking staff: Two added interns, a
clinician, with extra help (transportation) in January 2016. Additionally, the QI
Coordinator hired in August 2016 speaks Punjabi.
Recommendation #4: Evaluate the effectiveness of new crisis protocols through
quarterly data collection and analysis including comparison to baseline data, over time,
with demographics information as well as provider information to determine best
practices. Consider addressing this through a PIP, with consultation and technical
assistance provided by the EQRO.
☐ Fully addressed ☒ Partially addressed ☐ Not addressed
o The MHP has been experiencing difficulties with its crisis protocols since April
2016 with the bankruptcy and closure of Colusa Regional Medical Center, the
only hospital in the county.
Page 10
o The hospital closure has complicated the lives of consumers who in the past
could receive care locally after hours at the hospital 24/7. Since the closure
there is no available emergency room, the MHP crisis staff is impacted.
o The MHP is able to access all statistical data through April 2016. The MHP now
relies on information from Glenn and Sutter-Yuba counties. Glenn County
provides medical clearance for those assessed, while Sutter-Yuba accepts Colusa
MHP 5150s with reimbursement by Colusa County.
Recommendation #5: Create opportunities for line staff to encourage their involvement
and offer cross-training to expand skill sets which can support the system. Establish an
ongoing bi-directional feedback system between leadership, management and staff to
evaluate effectiveness and consumer/staff satisfaction.
☐ Fully addressed ☒ Partially addressed ☐ Not addressed
o In the fall of 2015, the MHP formed a staff advisory committee (SAC) comprised
of leadership and other system wide staff. The SAC meets quarterly to identify
challenges in providing effective services and to evaluate consumer/staff
satisfaction.
o The staff has advisory committee meetings monthly.
o A staff survey from 2015 indicated that communication could be improved
between leadership and staff particularly around receiving feedback and
communicating changes. The MHP states that they have made some changes
based on the suggestions from staff, however, feedback from staff indicate that
there are still issues remaining.
Recommendation #6: Identify and incorporate a system wide consumer outcome tool
for the children’s system of care, such as the child behavior checklist, into the EHR and
establish monthly reporting of consumer functioning similar to the MHP’s
implementation of the MORs for adults. Use data to develop an evidence based quality
driven system.
☒ Fully addressed ☐ Partially addressed ☐ Not addressed
o The MHP installed the Child Behavior Check List (CBCL) in December 2015,
followed by staff training. System implementation occurred in January 2016.
o CBCL scoring is incorporated into the discharge summary in the EHR and
Medical Necessity form.
o The MHP should be able to perform data analyses with the newly hired QI
Coordinator who commences employment in August 2016.
Recommendation #7: Continue to create and implement ways to increase the peer
employee presence in programs. Increase consumer and family member involvement in
service related committees, including policy making committees. Ensure that there are
methods or venues for Spanish speaking stakeholders to participate.
Page 11
☒ Fully addressed ☐ Partially addressed ☐ Not addressed
o In January 2016, two peer positions were added and were classified as extra help.
These are paid part time positions with no benefits.
o Three peers have attended certification training and the Department has
participated in regional training for peer certification. Certification involved a
13-month training on Peer Core Competency through the Northern CA WET
Alliance.
o There is peer participation in the QI Committee, the BH Advisory Board, and in
Consumer Leadership for Safe Haven Wellness and Recovery Center.
o The MHP identified bicultural staff within its department for the Cultura de la Vida
outreach program and from there, collaborated with the district learner advisory
committee (DLAC), English learner advisory committee (ELAC), and migrant
education programs. Through these efforts, the MHP is involving peers and
identifying natural leaders to develop a Promotores program.
CHANGES IN THE MHP ENVIRONMENT AND WITHIN THE MHP—IMPACT AND IMPLICATIONS
Changes since the last CalEQRO review, identified as having a significant effect on service provision
or management of those services are discussed below. This section emphasizes systemic changes
that affect access, timeliness, and quality, including those changes that provide context to areas
discussed later in this report.
Access to Care
o The MHP established a Cultura De La Vida program which outreaches to the
Hispanic community. Beginning in October 2015, the program has provided 12
presentations in the community on services, and various mental health topics.
o The local hospital went bankrupt and closed abruptly. There are no emergency
room services. Glenn County provides medical clearance for those assessed.
Sutter Yuba will accept 5150s with reimbursement by Colusa County.
o The wellness center was expanded to be the Safe Haven Wellness and Recovery
Center. The new larger site has more services/programing for consumers
including staff leads groups, as well as peer lead groups.
Timeliness of Services
o The closure of the hospital impacts the timeliness of consumers obtaining
medical clearance for psychiatric hospitalization.
Quality
o The MHP has an in-house psychiatrist full time. The MHP revamped its policy
and procedure regarding the medication of children. They no longer have
Page 12
medication only child clients. If receiving medication, the child is required to be
in services.
o The MHP plans to install and begin implementation of a Telechart Patient Portal
following system upgrade in September 2016.
o The MHP now has a county administrative officer who started in the spring of
2016. Historically the MHP had only a board of supervisors.
o Over the last year, the MHP reorganized its prevention programs to
accommodate staff changes. The previous prevention coordinator left, so the
MHP reorganized the duties and structure of the AOD prevention program. The
MHP’s Spanish speaking clinical intern became a permanent county position. In
addition, the MHP added case management to AOD and added another intern
who also speaks Spanish.
o The MHP hired/promoted several staff members including bringing on QI
Coordinator, who will start August 22, 2016, adding a clinician for adult services
(currently awaiting clearance), and case manager for children’s services.
o The MHP expanded its relationship with the Office of Education by assigning one
of the clinicians to provide services to children on probation that are attending
the Community School. The MHP has a presence in all schools in Colusa County.
Consumer Outcomes
o The MHP hired a new QI Coordinator who will be responsible for data analysis
and trending, including reports based on data from the Client Service
Information (CSI), California Outcomes Measurements System (CalOMS),
penetration rates, billing, walk in appointments, demographics, crisis reports
regarding contacts, outcomes, and hospitalization.
Page 13
PERFORMANCE MEASUREMENT
CalEQRO is required to validate the following PMs as defined by DHCS:
Total Beneficiaries Served by each county MHP
Total Costs per Beneficiary Served by each county MHP
Penetration Rates in each county MHP
Count of TBS Beneficiaries Served Compared to the four percent (4%) Emily Q.
Benchmark (not included in MHP reports; this information is included in the Annual
Statewide Report submitted to DHCS)
Total Psychiatric Inpatient Hospital Episodes, Costs, and Average Length of Stay
Psychiatric Inpatient Hospital 7-Day and 30-Day Rehospitalization Rates
Post-Psychiatric Inpatient Hospital 7-Day and 30-Day SMHS Follow-Up Service Rates
High Cost Beneficiaries ($30,000 or higher)
TOTAL BENEFICIARIES SERVED
Table 1 provides detail on beneficiaries served by race/ethnicity.
Table 1—Colusa County MHP Medi-Cal Enrollees and Beneficiaries Served in CY15 by Race/Ethnicity
Race/Ethnicity Average Monthly Unduplicated
Medi-Cal Enrollees* Unduplicated Annual Count of
Beneficiaries Served
White 1,258 202
Hispanic 5,377 248
African-American 36 4
Asian/Pacific Islander 103 10
Native American 78 12
Other 671 47
Total 7,521 523
*The total is not a direct sum of the averages above it. The averages are calculated separately.
Page 14
PENETRATION RATES AND APPROVED CLAIM DOLLARS PER BENEFICIARY
The penetration rate is calculated by dividing the number of unduplicated beneficiaries served by
the monthly average enrollee count. The average approved claims per beneficiary served per year
is calculated by dividing the total annual dollar amount of Medi-Cal approved claims by the
unduplicated number of Medi-Cal beneficiaries served per year.
Regarding calculation of penetration rates, the Colusa County MHP:
☒ Uses the same method as used by the EQRO
☐ Uses a different method
☐ Does not calculate its penetration rate
Page 15
Figures 1A and 1B show 3-year trends of the MHP’s overall approved claims per beneficiary and
penetration rates, compared to both the statewide average and the average for Small Rural MHPs.
$0
$1,000
$2,000
$3,000
$4,000
$5,000
$6,000
CY13 CY14 CY15
Figure 1A. Overall Average Approved Claims per Beneficiary
Colusa Small-Rural State
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
8.00%
9.00%
10.00%
CY13 CY14 CY15
Figure 1B. Overall Penetration Rates
Colusa Small-Rural State
Page 16
Figures 2A and 2B show 3-year trends of the MHP’s foster care (FC) approved claims per
beneficiary and penetration rates, compared to both the statewide average and the average for
Small Rural MHPs.
$0
$2,000
$4,000
$6,000
$8,000
$10,000
$12,000
$14,000
$16,000
CY13 CY14 CY15
Figure 2A. FC Average Approved Claims per Beneficiary
Colusa Small-Rural State
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
CY13 CY14 CY15
Figure 2B. FC Penetration Rates
Colusa Small-Rural State
Page 17
Figures 3A and 3B show 3-year trends of the MHP’s Hispanic approved claims per beneficiary and
penetration rates, compared to both the statewide average and the average for Small Rural MHPs.
$0
$1,000
$2,000
$3,000
$4,000
$5,000
$6,000
CY13 CY14 CY15
Colusa Small-Rural State
Figure 3A. Hispanic Average Approved Claims per Beneficiary
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
CY13 CY14 CY15
Figure 3B. Hispanic Penetration Rates
Colusa Small-Rural State
Page 18
HIGH-COST BENEFICIARIES
Table 2 compares the statewide data for high-cost beneficiaries (HCB) for CY15 with the MHP’s data
for CY15, as well as the prior two years. HCB in this table are identified as those with approved
claims of more than $30,000 in a year.
Table C1 (Attachment C) shows the penetration rate and approved claims per beneficiary for the
CY15 Medi-Cal Expansion (Affordable Care Act [ACS]) Penetration Rate and Approved Claims per
Beneficiary.
Table C2 (Attachment C) show the distribution of the MHP CY15 Distribution of Beneficiaries by
Approved Claims per Beneficiary (ACB) Range for the various categories; under $20,000; $20,000
to $30,000, and those above $30,000.
MHP Year
HCB
Count
Total
Beneficiary
Count
HCB %
by
Count
Average
Approved
Claims
per HCB
HCB Total
Claims
HCB % by
Approved
Claims
Statewide CY15 13,851 483,793 2.86% $51,635 $715,196,184 26.96%
CY15 14 523 2.68% $57,859 $810,021 30.48%
CY14 8 534 1.50% $41,594 $332,754 14.97%
CY13 8 500 1.60% $44,346 $354,765 19.90%
Table 2—High-Cost Beneficiaries
Colusa
Page 19
TIMELY FOLLOW-UP AFTER PSYCHIATRIC INPATIENT DISCHARGE
Figures 4A and 4B show the statewide and MHP 7-day and 30-day outpatient follow-up and
rehospitalization rates for CY14 and CY15.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Oupatient MHP Outpatient State RehospitalizationMHP
RehospitalizationState
Figure 4A. 7-Day Outpatient Follow-up and Rehospitalization Rates, Colusa MHP and State
CY14 CY15
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Oupatient MHP Outpatient State RehospitalizationMHP
RehospitalizationState
Figure 4B. 30-Day Outpatient Follow-up and Rehospitalization Rates, Colusa MHP and State
CY14 CY15
Page 20
DIAGNOSTIC CATEGORIES
Figures 5A and 5B compare the breakdown by diagnostic category of the statewide and MHP
number of beneficiaries served and total approved claims amount, respectively, for CY15.
MHP self-reported percent of consumers served with co-
occurring (substance abuse and mental health) diagnoses:
0%
5%
10%
15%
20%
25%
30%
35%
Depression Psychosis Disruptive Bipolar Anxiety Adjustment Other Deferred
Figure 5A. Diagnostic Categories, Beneficiaries Served
Colusa CY15 State CY15
0%
5%
10%
15%
20%
25%
30%
35%
Depression Psychosis Disruptive Bipolar Anxiety Adjustment Other Deferred
Figure 5B. Diagnostic Categories, Total Approved
Colusa CY15 State CY15
20%
Page 21
PERFORMANCE MEASURES FINDINGS—IMPACT AND IMPLICATIONS
Access to Care
o The MHP’s overall penetration rate has declined each year between CY13 and
CY15 similar to the trend experienced overall and across small rural MHPs. Its
penetration rate has been higher than statewide and small rural MHPs during
the same period.
o The MHP’s foster care penetration rate is similar to small rural MHPs and lower
than the statewide.
o The MHP’s Hispanic penetration rate is the same as small rural MHPs and higher
than statewide. The MHP’s Hispanic penetration rates have declined between
CY13 and CY15 similar to the downward trend experienced statewide.
Timeliness of Services
o The MHP’s 7-day and 30-day outpatient follow-up rates after discharge from
psychiatric inpatient episodes were higher than statewide.
Quality of Care
o The MHP’s percentage of high-cost beneficiaries was similar to statewide and
showed an increase from CY14. Its percentage of HCB claim dollars was higher
than statewide and doubled from CY14 to CY15.
o The MHP’s average approved claims per beneficiary served was lower than
statewide and higher than small rural MHP’s. The MHP’s average approved
claims per beneficiary served has increased consistently for the three years
between CY13 and CY15.
o The MHP’s average approved claims per beneficiary for foster care is higher
than both statewide and small rural MHPs.
o The MHP’s average approved claims per beneficiary for Hispanics is higher than
small rural MHPs but lower than statewide.
o Like statewide, a primary diagnosis of Depressive Disorders accounted for the
largest number of beneficiaries served by the MHP. However, the percentage
was higher than statewide. The MHP had a higher percentage of beneficiaries
with primary diagnosis of anxiety than statewide.
o The MHP appears to use Deferred Diagnosis slightly lower than statewide.
Consumer Outcomes
o None noted.
Page 22
PERFORMANCE IMPROVEMENT PROJECT VALIDATION
A Performance Improvement Project (PIP) is defined by CMS as “a project designed to assess and
improve processes, and outcomes of care that is designed, conducted and reported in a
methodologically sound manner.” The Validating Performance Improvement Projects Protocol
specifies that the EQRO validate two PIPs at each MHP that have been initiated, are underway, were
completed during the reporting year, or some combination of these three stages. DHCS elected to
examine projects that were underway during the preceding calendar year 2015.
COLUSA MHP PIPS IDENTIFIED FOR VALIDATION
Each MHP is required to conduct two PIPs during the 12 months preceding the review. CalEQRO
reviewed and validated two MHP submitted PIPs as shown below.
Table 3A—PIPs Submitted
PIPs for Validation # of PIPS PIP Titles
Clinical PIP 1 Recovery
Non-Clinical PIP 1 Parent Intervention
Table 3B lists the findings for each section of the evaluation of the PIPs, as required by the PIP
Protocols: Validation of Performance Improvement Projects.4
Table 3B—PIP Validation Review
Step PIP Section Validation Item
Item Rating*
Clinical PIP
Non-Clinical
PIP
1 Selected Study Topics
1.1 Stakeholder input/multi-functional team M M
1.2 Analysis of comprehensive aspects of enrollee needs, care, and services
M M
1.3 Broad spectrum of key aspects of enrollee care and services
M M
1.4 All enrolled populations M M
2 Study Question 2.1 Clearly stated NM M
4 2012 Department of Health and Human Services, Centers for Medicare and Medicaid Service Protocol 3
Version 2.0, September 2012. EQR Protocol 3: Validating Performance Improvement Projects.
Page 23
Table 3B—PIP Validation Review
Step PIP Section Validation Item
Item Rating*
Clinical PIP
Non-Clinical
PIP
3 Study Population 3.1 Clear definition of study population M M
3.2 Inclusion of the entire study population M M
4 Study Indicators
4.1 Objective, clearly defined, measurable indicators
M M
4.2 Changes in health status, functional status, enrollee satisfaction, or processes of care
M M
5 Sampling Methods
5.1 Sampling technique specified true frequency, confidence interval and margin of error
NM NA
5.2 Valid sampling techniques that protected against bias were employed
M NA
5.3 Sample contained sufficient number of enrollees
PM NA
6 Data Collection Procedures
6.1 Clear specification of data M M
6.2 Clear specification of sources of data PM M
6.3 Systematic collection of reliable and valid data for the study population
PM PM
6.4 Plan for consistent and accurate data collection
M PM
6.5 Prospective data analysis plan including contingencies
NM NM
6.6 Qualified data collection personnel M M
7 Assess Improvement Strategies
7.1 Reasonable interventions were undertaken to address causes/barriers
UTD M
8
Review Data Analysis and Interpretation of Study Results
8.1 Analysis of findings performed according to data analysis plan
NM NM
8.2 PIP results and findings presented clearly and accurately
M NA
8.3 Threats to comparability, internal and
external validity NM NA
8.4 Interpretation of results indicating the success
of the PIP and follow-up NM NA
9 Validity of Improvement
9.1 Consistent methodology throughout the study UTD NA
9.2 Documented, quantitative improvement in
processes or outcomes of care NA NA
Page 24
Table 3B—PIP Validation Review
Step PIP Section Validation Item
Item Rating*
Clinical PIP
Non-Clinical
PIP
9.3 Improvement in performance linked to the
PIP NA NA
9.4 Statistical evidence of true improvement NA NA
9.5 Sustained improvement demonstrated
through repeated measures. NA NA
*M = Met; PM = Partially Met; NM = Not Met; NA = Not Applicable; UTD = Unable to Determine
Table 3C gives the overall rating for each PIP, based on the ratings given to the validation items.
Table 3C—PIP Validation Review Summary
Summary Totals for PIP Validation Clinical
PIP
Non-Clinical
PIP
Number Met 13 13
Number Partially Met 3 2
Number Not Met 6 2
Number Applicable (AP)
(Maximum = 28 with Sampling; 25 without Sampling)
24 17
Overall PIP Rating ((#Met*2)+(#Partially Met))/(AP*2) 60.42% 82.35%
CLINICAL PIP—RECOVERY
The MHP presented its study question for the clinical PIP as follows:
“Can the Milestones of Recovery (MORS) rating scale be used to effectively guide
treatment interventions?”
Date PIP began: January 2015
Status of PIP:
☒ Active and ongoing
☐ Completed
Page 25
☐ Inactive, developed in a prior year
☐ Concept only, not yet active
☐ Submission determined not to be a PIP
☐ No PIP submitted
The MHP, as a result of its participation in the Advancing Recovery Collaborative, collected MORS
scores for all adult consumers beginning in January 2015. Higher than expected MORS scores for
13% of the study population led to the MHP focusing on transitioning consumers to lower levels of
care. The MHP aimed to improve outcomes for adults, including “graduating” from services of the
MHP. The PIP examined which interventions assisted consumers in recovery as measured using the
Milestones of Recovery outcome tool (MORs), leading to shortened and consistent time to
transition to lower level levels of care.
The MHP collected MORS scores monthly and found that individuals at MORS 7 were able to
transition out of services, as well as those at MORS 6. The MHP then focused on individuals with a
MORS score of 5. In contrast to the MORS 6 and 7 individuals, these individuals were not able to
move quickly up the MORS scale. These individuals stayed at MORS 5 throughout the measurement
cycle. Previously successful Interventions were doubled (Strength Assessment and Group
Supervision). Even with these interventions, little movement was seen for the MORS 5 individuals.
The MHP is now testing the outcome tools that may be more sensitive to smaller changes (PHQ 9
and the GAD 7) for individuals with depression or anxiety diagnoses as they can be administered
more frequently.
The study question design did not lend itself to be answerable through the PIP process.
Relevant details of these issues and recommendations are included within the comments found in
the PIP validation tool.
Technical assistance was provided to the MHP by CalEQRO. Recommendations include a discussion
on how the study question design did not lend itself to be answerable through the PIP process.
Recommend that for next PIP, set up question so that it is quantifiable and contains indicators, i.e.
“Will ____ intervention improve ________ (symptoms/problem) as measured by ________outcome tool?”
As it stands, the PIP’s design does not allow the MHP to track the new outcome tools and does not
identify which interventions improve which symptoms. Clarification on whether MORs 6s and 7s
entered at the 5 level when they came in for services initially. This would allow for a meaningful
comparison regarding how long it took to get to the next level. For next steps, the MHP plans to
amend its question, evaluate the impact of strengths assessment on “leveling up” using the MORs, in
conjunction with the anxiety and depression inventory tools. Recommended that this PIP be
considered completed rather than going into year 3 of this PIP. Instead, develop a new PIP
evaluating clinical outcomes post intervention using the anxiety and depression assessments. As it
stands, this process didn’t make an improvement for clients as they were still discharged for the
same reason which was not a result of better implementation of the MORS. Rather, the PIP provided
more information regarding level of care and helped inform the administrative process of
discharge/level of care.
Page 26
NON-CLINICAL PIP—PARENT INTERVENTION
The MHP presented its study question for the non-clinical PIP as follows:
“Will the provision parental intervention result in shorter duration of care, higher levels
of parent satisfaction with care, and improved functioning as measured by the Child
Behavior Checklist?”
Date PIP began: August, 2015
Status of PIP:
☒ Active and ongoing
☐ Completed
☐ Inactive, developed in a prior year
☐ Concept only, not yet active
☐ Submission determined not to be a PIP
☐ No PIP submitted
This PIP is at the beginning of its second year. Year 1 focused on shortening duration of services
with the addition of conjoint family services as a standard protocol. The second year of the PIP is
aimed at evaluating the impact of conjoint (automatically assigned) family interventions,
specifically Parent-Child Interaction Therapy (PCIT) on client functioning. The overarching goal of
this PIP will be to improve child functioning in many areas relevant to the child’s progress toward
resilience. The MHP analyzed data that compared the length of treatment for children receiving
family therapy of any kind or duration beyond intake as part of their treatment with children who
had not. The MHP looked at this data for fiscal year 2014/2015 and found that 145 children had
received family therapy, where 62 children had not. When comparing time in treatment for the two
groups, the MHP found that children receiving any form of family involvement (145 children)
averaged 3.763889 months in care and those that received no form of family involvement (63
children) averaged 3.967742 months. Remeasurement has proven to be a challenge. The MHP had
difficulty rolling out the CBCL due to scoring challenges and difficulty in getting the score entered
into the EHR. Additionally, the MHP discovered that they were not getting discharge CBCL scores
because families were choosing to withdraw from care when the clinician began discussing the end
of treatment. Further, no one has been in care for a full year since the implementation of the CBCL.
The MHP experienced similar issues with the parent satisfaction survey. A way to ameliorate this
type of issue would be to collect data on a quarterly basis.
Relevant details of these issues and recommendations are included within the comments found in
the PIP validation tool.
Page 27
Technical assistance was provided to the MHP by CalEQRO. Feedback included acknowledgment
that the PIP focus seemed to naturally shift to evaluating the PCIT intervention. The study question
needed clarification so that it was measurable an objective. The interventions listed are not truly
the intervention, but rather assigning all mild to moderate clients to PCIT and all moderate to
severe clients to individual as a matter of standard procedure. It was recommended that the MHP
then compare the CBCL scores of from a baseline at start of treatment after a set period of time (i.e.
3 months). This would allow the county to see the true effectiveness of the new administrative
procedure. Data should be collected and analyzed quarterly.
PERFORMANCE IMPROVEMENT PROJECT FINDINGS—IMPACT AND IMPLICATIONS
Access to Care
o If the MHP is successful identifying and implementing interventions that move
clients along the continuum of care, and decrease the level of service provided,
more space for new clients would be available.
Timeliness of Services
o If the MHP is successful identifying and implementing interventions that move
clients along the continuum of care, and decrease the level of service provided,
timeliness for appointments for new clients would improve with the new “slots”
available.
o If client functioning improves and they are able to transition out of service, more
timely appointments are available for new clients coming in.
Quality of Care
o Interventions which are effective in helping clients to improve functioning
contribute to the quality of services provided by the MHP and convey the
message of hope.
o Engaging parents in children’s treatment improves quality of care for children.
Consumer Outcomes
o Engaging parents in children’s treatment results in better outcomes for child
clients.
o Using anxiety and depression symptom inventories would allow for more
nuanced assessment of client outcomes and would prove useful for evaluating
client functioning, involvement and perceptions of treatment.
Page 28
PERFORMANCE & QUALITY MANAGEMENT KEY COMPONENTS
CalEQRO emphasizes the MHP’s use of data to promote quality and improve performance.
Components widely recognized as critical to successful performance management include an
organizational culture with focused leadership and strong stakeholder involvement, effective use of
data to drive quality management, a comprehensive service delivery system, and workforce
development strategies that support system needs. These are discussed below.
Access to Care
As shown in Table 4, CalEQRO identifies the following components as representative of a broad
service delivery system that provides access to consumers and family members. An examination of
capacity, penetration rates, cultural competency, integration and collaboration of services with
other providers forms the foundation of access to and delivery of quality services.
Table 4—Access to Care
Component Compliant
(FC/PC/NC)* Comments
1A Service accessibility and availability are reflective of cultural competence principles and practices
FC Cultural, ethnic, racial, and linguistic needs are assessed at intake. Over the last year, the MHP prioritized hiring bilingual and bicultural staff as well as identifying current staff for resources within.
Treatment outcomes are evaluated, however, the MHP would benefit from additional staffing for analysis. The MHP developed the Cultural Vida program and provided 12 presentations within the community beginning October 2015. From there, the MHP collaborated with the district level English learner advisory committee (DLAC), and English learner advisory committee (ELAC), both migrant education programs. The Cultural Vida Program is working to identify natural leaders to develop a Promotores Program. The Cultural Vida program are working with the community and the office of education with their presentation “how mental health pertains to you.” The need for this program was identified through a survey where people said they were aware of mental health but didn’t know how it pertained to them. With this presentation, there will be a pre/post survey
The MHP is using MHSA money to supplement staff salaries so that all staff can are involved in outreach and engaging the community.
1B Manages and adapts its capacity to meet beneficiary service needs
FC The MHP tracks clients coming in for appointments and also assesses the need for bilingual services.
The MHP utilizes an office assistant who speaks Spanish and assists psychiatry. The previous prevention coordinator left. As a result, the MHP reorganized duties
Page 29
Table 4—Access to Care
Component Compliant
(FC/PC/NC)* Comments and structure of the program. A Spanish speaking intern became a permanent county position. The MHP also drew from the children’s department and added another intern for this program who also speaks Spanish.
1C Integration and/or collaboration with community based services to improve access
FC The psychiatry department works with primary care providers in the area. There is also a collaboration for suicide prevention training for medical staff. They do not have contract providers in the area. The MHP has integrated AOD services with mental health. They participate in joint trainings. Trainings were offered to public health but they did not attend.
One Stop manages the transitional housing for CalWorks. However, there is no short term or long term housing in the county.
In April 2016, the local hospital went bankrupt and closed abruptly. There is no access to emergency room services. Glenn and Sutter-Yuba counties are assisting with 5150s. Glenn County provides medical clearance for those assessed. And Sutter Yuba will accept 5150s with reimbursement by Colusa County.
The MHP continues to collaborate with law enforcement agencies for crisis services. However, the MHP’s request to California Highway Patrol (CHP) to limit 5150s to Colusa residents was unsuccessful. CHP still brings out of town folks to 5150. This impacts crisis staff which is an informal mobile crisis unit as there is no official funding stream or designated vehicle.
*FC =Fully Compliant; PC = Partially Compliant; NC = Non-Compliant
Timeliness of Services
As shown in Table 5, CalEQRO identifies the following components as necessary to support a full
service delivery system that provides timely access to mental health services. The ability to provide
timely services ensures successful engagement with consumers and family members and can
improve overall outcomes while moving beneficiaries throughout the system of care to full
recovery.
Table 5—Timeliness of Services
Component Compliant
(FC/PC/NC)* Comments
2A Tracks and trends access data from initial contact to first appointment
FC The MHP sets a standard of 10 days, with 100% of its appointments meeting this standard. The range is 0-9
Page 30
Table 5—Timeliness of Services
Component Compliant
(FC/PC/NC)* Comments
days for both adults and children with an average of 3 days.
2B Tracks and trends access data from initial contact to first psychiatric appointment
PC The MHP monitors time from assignment to psychiatry (MD) to first psychiatry appointment. The MHP sets a standard of 45 days, with 100% of its appointments meeting this standard. The range is 2-33 days for with an average of 16.6 days. Adult and children services are not tracked separately. In light of the change in policy (fall 2015) regarding medication management by children’s psychiatry and the elimination of meds only practice (all children receiving medications are required to be in services), it is recommended that this measure be tracked and trended separately.
2C Tracks and trends access data for timely appointments for urgent conditions
FC The MHP reported a goal of .125 days (3 hours) wait times for urgent appointments and that they met this goal 80% of the time, with an average of .125 days.
2D Tracks and trends timely access to follow up appointments after hospitalization
FC The MHP reported 24 hospitalizations (22 adult, 2 youth) and that all clients received a follow up appointment within 7 days of discharge. The MHP sets a goal of 7 days. The average length of time for a follow up appointment with clinical staff was 1.7 days, with 100% of the appointments meeting the standard.
Last year, the MHP reported 8 hospitalizations (7 adults, 1 child) from January through June 2015 (6 months). Hospitalization rates increased this last year, even taking into account that the data for 15/16 spanned 12 months rather than last year when the MHP reported on 6 months of data.
2E Tracks and trends data on re-hospitalizations
FC The MHP reports that of the 24 hospitalizations (22 adult, 2 youth), no clients were re-hospitalized. The MHP sets a goal of 0 rehospitalizations within 30 days of discharge.
2F Tracks and trends No Shows
NC The MHP reports that they are just beginning to extract data on this measure and that they have not yet developed a format for tracking/trending.
*FC = Fully Compliant; PC = Partially Compliant; NC = Non-Compliant
Page 31
Quality of Care
As shown in Table 6, CalEQRO identifies the following components of an organization that is
dedicated to the overall quality of care. Effective quality improvement activities and data-driven
decision making require strong collaboration among staff (including consumer/family member
staff), working in information systems, data analysis, clinical care, executive management, and
program leadership. Technology infrastructure, effective business processes, and staff skills in
extracting and utilizing data for analysis must be present in order to demonstrate that analytic
findings are used to ensure overall quality of the service delivery system and organizational
operations.
Table 6—Quality of Care
Component Compliant
(FC/PC/NC)* Comments
3A Quality management and performance improvement are organizational priorities
FC The MHP hired a new Quality Improvement
Coordinator who may perform analysis of data such as
penetration rates, billing, access, demographics, etc.
3B Data are used to inform management and guide decisions
FC Based on intake tracking data on demographics, client
numbers, language and type of service need, the MHP
changed its intake schedule from 2 clinicians to 3,
including a Spanish speaker.
The MHP assesses client outcomes using the
Milestones of Recovery scale (MORs) for all adults once
per month via reporting from the EHR. The MHP will be
developing its trending ability with its new QI and EHR
coordinator. They are also working to get a depression
inventory tool (PHQ9) and anxiety inventory tool
(GAD7) into to the EHR.
The MHP is using the Child Behavior Checklist (CBCL) for children. All children receive the CBCL with results in the EHR. The MHP is beginning to look at group data.
3C Evidence of effective communication from MHP administration
PC The MHP holds a variety of meetings for
communication both within the MHP and in the
community. These include quarterly all-staff, weekly
clinical teams, Katie A., multidisciplinary teams each
month, QIC meets every other month.
Leadership/management meets weekly.
The MHP has a monthly newsletter in Spanish and
English They also have Facebook pages for
adults/children’s services. Client leadership monthly
meets to oversee and run the drop-in center. Peers
Page 32
Table 6—Quality of Care
Component Compliant
(FC/PC/NC)* Comments Helping Peers meets twice per month to provide
information on the wellness center.
Clinical staff participation during the review was
limited and not all staff who wished to participate were
permitted.
The MHP formed a representative staff advisory
committee that meets regularly to identify challenges
in providing effective services and to evaluate
consumer/staff satisfaction. However, clinical staff
reported that while the opportunity to provide ideas
and feedback exists, they felt that they had little input
in program and system planning. The consensus
among line staff was that morale is very low, citing a
punitive system if productivity is not met (either by a
little or a lot). The feeling was that there is a big
disconnect with what upper management perceives is
happening vs. what is really going on.
The MHP did make changes to its all staff meetings and
how the meeting is run. The agenda is posted before
the meeting so that those who wish to provide
feedback can. There are now standing items at each
meeting. Most recently, there was a staff survey
regarding how acknowledgements are made.
3D Evidence of stakeholder input and involvement in system planning and implementation
PC CFM participants are actively involved in running the
wellness center.
Because of the low attendance the prior year at the
MHSA stakeholder planning meeting, this year, the
MHP alternatively held meetings within each
community – Maxwell, Arbuckle, Williams and Colusa.
Spring 2016. They were able to update the plan. 10-15
people participated.
During Mental Health Awareness Month, May 2016,
the MHP held 4 events targeting specific groups most
of which had low turnout. However, the most
successful event was a Safe Haven Peer Walk In – with
Veterans centered on a car show with 33 cars. The
MHP outreached and raised $900. There were 100-150
participants.
Page 33
Table 6—Quality of Care
Component Compliant
(FC/PC/NC)* Comments A staff survey from 2015 provided feedback that
communication was poor regarding feedback and
subsequent changes. The MHP states that they have
made some changes due to suggestions from staff,
however, feedback from staff indicate that there are
still issues regarding how feedback is received and
whether it is acted upon effectively by leadership.
3E Evidence of strong collaborative partnerships with other agencies and community based services
FC As stated above, the MHP has strong collaborative
relationships with community agencies such as law
enforcement (Colusa County Sheriff, Colusa Police
Department., Williams Police Department, and
California Highway Patrol) as well as probation, and
health and human services. The MHP also has a robust
youth program through the school district. The county
has partnered with Glenn and Sutter-Yuba counties to
cover the gap in mental health services previously
provided by the local hospital, which closed abruptly in
April, 2016.
3F Evidence of a systematic clinical Continuum of Care
FC The MHP uses the MORS as an outcome tool in adult
services. This instrument is scored monthly on “MORS
Monday” which is the last Monday of the month. As
noted in the PIP write up, this tool is used to measure
consumer recovery status, and to assist with treatment
planning. The PHQ 9 was tested in the adult division as
a quick measurement tool for depression; and the GAD
7 as a quick measurement tool for generalized anxiety.
The MHP recently added the use of the Child Behavior
Check List (CBCL) in the Child division. This tool is
administered at intake, annually and at discharge. To
date no discharge CBCL’s have been recorded.
Regarding, medication monitoring, the QI plan
referenced that the MHP had not been compliant in
having prescribing practices reviewed since the
departure of psychiatrist Dr. Samson. The MHP added
this as an action item to the annual plan.
3G Evidence of individualized, client-driven treatment and recovery
PC The MHP’s drop-in center moved in June 2016 to a
larger space. The MHP wanted to maintain the peer
drop in center but within the context of wellness and
recovery. The center was renamed Safe Haven
Page 34
Table 6—Quality of Care
Component Compliant
(FC/PC/NC)* Comments Wellness & Recovery Center. In addition to drop-in
services, the center now has groups – nutrition,
mindfulness, and walking. Staff leads groups, as well as
peers. Wellness Recovery Action Planning (WRAP) will
be starting as some staff are trained WRAP leaders.
While the MHP utilizes depression and anxiety
inventories with clients in session, information on how
consumers are engaged in their own treatment
planning and how a consumer uses level of function
scales/tools, etc. themselves was not provided.
3H Evidence of consumer and family member employment in key roles throughout the system
FC Peer Specialist (a full time benefited position) manages
greeter positions at Safe Haven. Greeters receive
stipends.
The peer specialist recently completed a 13 month
training on Peer Core Competency through the
Northern California Workforce Education and Training
(WET) alliance and will be implementing those
competencies at the wellness center.
There are 5 part time drivers and safety support staff.
They are on call, work no more than 20 hours per
week.
3I Consumer run and/or consumer driven programs exist to enhance wellness and recovery
FC The MHP’s wellness center is the Safe Haven Wellness & Recovery Center. During the site visit, there were at least 3 consumers there; a full-time CFM, and 2 part-time CFM’s. The center is open to whomever comes through the doors. Service and program information were available at the desk. The operating hours are Monday through Saturday 8:30-4:30 pm. Clients receive a calendar of activities in their intake packet at assessment.
3J Measures clinical and/or functional outcomes of consumers served
FC The MHP assesses client outcomes using Milestones of
Recovery Scale (MORs) for all adults once per month
via reporting from the EHR. The MHP will developing
its trending ability with its new QI and EHR
coordinator. They are also working to get a depression
inventory tool (PHQ9) and anxiety inventory tool
(GAD7) into the EHR. The MHP is using the Child
Behavior Checklist (CBCL) for children. All children
receive the CBCL with results in the EHR. This was
Page 35
Table 6—Quality of Care
Component Compliant
(FC/PC/NC)* Comments implemented August 2015 as part of the medical
necessity form. The MHP is beginning to look at
grouped data.
3K Utilizes information from Consumer Satisfaction Surveys
PC The MHP does provide the perception and outcomes
survey but does not analyze the data prior to State
submission. The State sends raw data back for review.
The MHP does retain the comments. They added
parenting classes as a result of POQI feedback.
*FC = Fully Compliant; PC = Partially Compliant; NC = Non-Compliant
KEY COMPONENTS FINDINGS—IMPACT AND IMPLICATIONS
Access to Care
o The MHP has added bilingual Spanish speaking staff: Two added interns, a
clinician and a QI Coordinator in August 2016, with extra help transportation
positions added in January 2016.
o In April 2016, the local hospital went bankrupt and closed abruptly. There is no
access to emergency room services. The MHP collaborates with Glenn County to
provide medical clearance for those needing assessment while Sutter Yuba MHP
accepts Colusa County 5150s with reimbursement by Colusa County.
o The MHP is also collaborating with law enforcement to assist consumers in
crisis.
o The MHP has an informal mobile crisis unit as there is no official funding stream
or designated vehicle.
Timeliness of Services
o The MHP has reported no change for length of time from first request for service
to first clinical assessment from FY15-16 to FY16-17 – average length of time is
3 days surpassing their goal of 10 days.
o The MHP has reduced their average length of time from first request of service
to first psychiatric appointment from 24.69 days for FY15-16 to 16.6 days for
FY16-17 for all consumers.
o The MHP does not track no shows stating they only recently have had the
capability to track electronically.
Page 36
Quality of Care
o Based on intake tracking data on demographics, language, and type of service
needed, the MHP changed its intake schedule team from 2 clinicians to 3
including a Spanish speaker.
o They are also working to get a depression inventory tool (PHQ9) and anxiety
inventory tool (GAD7) into the EHR.
o The MHP recently hired a new QI Coordinator who will be responsible for data
analysis and trending, including reports based on data from the Client Service
Information (CSI), California Outcomes Measurements System (CalOMS),
penetration rates, billing, walk in appointments, demographics, crisis reports
regarding contacts, outcomes, and hospitalization.
Consumer Outcomes
o The MHP is using the Child Behavior Checklist (CBCL) for children. All children
receive the CBCL with results in the EHR. The MHP is beginning to look at group
data.
o The MHP assesses client outcomes using Milestone of Recovery (MORs) for all
adults once per month via reporting from the EHR. The MHP plans to develop its
trending ability with its new QI and EHR coordinator.
o The MHP did a Clinical PIP utilizing the MORs to assess how interventions assist
consumers in recovery.
Page 37
CONSUMER AND FAMILY MEMBER FOCUS GROUP(S)
CalEQRO conducted one (1) 90-minute focus group with consumers and family members during the
site review of the MHP. As part of the pre-site planning process, CalEQRO requested one (1) focus
group with 8 to 10 participants each, the details of which can be found in each section below.
The Consumer/Family Member Focus Group is an important component of the CalEQRO Site
Review process. Obtaining feedback from those who are receiving services provides significant
information regarding quality, access, timeliness, and outcomes. The focus group questions specific
to the MHP reviewed and emphasized the availability of timely access to care, recovery, peer
support, cultural competence, improved outcomes, and consumer and family member involvement.
CalEQRO provided gift certificates to thank the consumers and family members for their
participation.
CONSUMER/FAMILY MEMBER FOCUS GROUP 1
CalEQRO requested a culturally diverse group of adult beneficiaries and parents/caregivers of
child/youth beneficiaries including a mix of existing and new clients who have initiated/utilized
services within the past 12 months. The focus group was held at the MHP offices in Willows, located
at 162 E Carson Street, Colusa, CA 95932.
The group was comprised of 7 women and 3 men, 6 of which received services in the last year.
Number of participants: 10
For the six (6) participants who entered services within the past year, they described their
experience as the following:
Regarding timeliness of first contact to assessment, the responses varied from the next
day, one week, or within two weeks.
Regarding timeliness of assignment to and meeting with a therapist, responses also
varied, from one week, a couple of weeks, to a few weeks.
General comments regarding service delivery that were mentioned included the following:
The consensus of the group was then even if in crisis, it takes two to three weeks to get
help and by that time, the crisis is over.
Impacted by staff turnover or leave, reassignment of therapists takes a long time.
Public transportation is unreliable.
Recommendations for improving care included the following:
Page 38
Timely response in rescheduling missed appointments or assignment to therapist after
in-take.
Focus groups for veterans.
Provide intakes more than once-a-week.
Provide transportation to outer areas.
Interpreter used for focus group 1: ☐ No ☒ Yes Language(s): Spanish
CONSUMER/FAMILY MEMBER FOCUS GROUP FINDINGS—IMPLICATIONS
Access to Care
o Clients’ experiences with accessing initial care were positive. However, there is
inconsistencies in the process for being assigned a therapist and for ongoing
services.
o Transportation from outer areas contributes to service access issues.
Timeliness of Services
o Services impacted by staff shortages make it challenging for clients to
reschedule appointments in a timely manner.
o Clients reported that access to crisis services was not timely.
Quality of Care
o When receiving services, overall, focus group members reported satisfaction
with the services, and felt they were getting better.
Consumer Outcomes
o No information provided.
Page 39
INFORMATION SYSTEMS REVIEW
Knowledge of the capabilities of an MHP’s information system is essential to evaluate the MHP’s
capacity to manage the health care of its beneficiaries. CalEQRO used the written response to
standard questions posed in the California-specific ISCA, additional documents submitted by the
MHP, and information gathered in interviews to complete the information systems evaluation.
KEY ISCA INFORMATION PROVIDED BY THE MHP
The following information is self-reported by the MHP in the ISCA and/or the site review.
Table 8 shows the percentage of services provided by type of service provider:
Table 8—Distribution of Services by Type of Provider
Type of Provider Distribution
County-operated/staffed clinics 99.5%
Contract providers 0.5%
Network providers 0%
Total 100%
Percentage of total annual MHP budget is dedicated to support information technology
operations: (includes hardware, network, software license, IT staff)
7.2%
Consumers have on-line access to their health records either through a Personal Health
Record (PHR) feature provided within EHR or a consumer portal or a third-party PHR:
☐ Yes ☐ In Test/Pilot Phase ☒ No
MHP currently provide services to consumers using an tele-psychiatry application:
☐ Yes ☐ In Test/Pilot Phase ☒ No
o If yes, the number of remote sites currently operational:
0
Page 40
MHP self-reported technology staff changes since the previous CalEQRO review (FTE):
Table 9 – Summary of Technology Staff Changes
Number IS
Staff
Number of New
Hires
Number of Staff Retired,
Transferred, Terminated
Current Number of
Unfilled Positions
3 0 0 0
MHP self-reported data analytical staff changes since the previous CalEQRO review
(FTE):
Table 10 – Summary of Data Analytical Staff Changes
Number
Data Analytical
Staff
Number of New
Hires
Number of Staff Retired,
Transferred, Terminated
Current Number of
Unfilled Positions
0 0 0 0
The following should be noted with regard to the above information:
The MHP has recruited a QI Coordinator who is a licensed clinician to begin in August
2016. Their primary job responsibilities may encompass more intensive analytical
tasks.
CURRENT OPERATIONS
The MHP continues to implement the Cerner Behavioral Health System (CCBH) via an
Application Service Provider (ASP) contract with Kings View Behavioral Health.
Table 11 lists the primary systems and applications the MHP uses to conduct business and manage
operations. These systems support data collection and storage, provide electronic health record
(EHR) functionality, produce Short-Doyle/Medi-Cal (SD/MC) and other third party claims, track
revenue, perform managed care activities, and provide information for analyses and reporting.
Page 41
Table 11— Primary EHR Systems/Applications
System/Application Function Vendor/Supplier Years Used Operated By
Cerner Community Behavioral Health (CCBH) Client Data
Practice Management Cerner 5 Kings View
CCBH - ATP Assessment and Treatment Plan
Cerner 5 Kings View
CCBH - Scheduling Appointment Scheduler Cerner 5 Kings View
CCBH Doctor's Homepage Clinical and ePrescribing Cerner 5 Kings View
CCBH-Clinician’s Homepage Clinical information and functionality
Cerner 5 Kings View
PLANS FOR INFORMATION SYSTEMS CHANGE
The MHP has no plans for information systems change.
ELECTRONIC HEALTH RECORD STATUS
Table 12 summarizes the ratings given to the MHP for Electronic Health Record (EHR) functionality.
Table 12—Current EHR Functionality
Function System/Application
Rating
Present Partially Present
Not Present
Not Rated
Alerts Cerner x
Assessments Cerner x
Document imaging/storage Cerner EHR x
Electronic signature—consumer Cerner EHR x
Laboratory results (eLab) x
Level of Care/Level of Service Cerner EHR x
Outcomes MORS, CBCL x
Prescriptions (eRx) Cerner EHR x
Progress notes Cerner EHR x
Treatment plans Cerner EHR x
Summary Totals for EHR Functionality 7 3
Page 42
Progress and issues associated with implementing an electronic health record over the past year
are discussed below:
The MHP has no current plans to implement eLab functionality.
The MHP has no current plans to implement electronic Level of Care/Level of Service
Assessments.
Consumer’s Chart of Record for county-operated programs (self-reported by MHP):
☐ Paper ☐ Electronic ☒ Combination
MAJOR CHANGES SINCE LAST YEAR
The MHP have met Meaningful Use standards for Phase 1 and will be able to use
Citrix/Cerner following system upgrade.
The MHP amended both the Medical Necessity form and Discharge Assessment form to
allow entry and scoring of the Children’s Behavioral Checklist (CBCL), which measures
outcomes.
PRIORITIES FOR THE COMING YEAR
The MHP plans to install a SQL upgrade, followed by installation of CCBH - Promotion
222. The installation of Promotion 222 will enable Ultra-Sensitive Exchange (USX) with
client portal which will allow secure client messaging.
The MHP has created two new assessments to accurately measure outcomes for anxiety
GAD-7 (General Anxiety Disorder – 7 Item) and major depressive disorders PHQ-9
(Patient Health Questionnaire – 9 Item). These assessments were sent to Kings View for
uploading in July 2016, but at the time of the review were not uploaded. The MHP is
projecting these will be implemented in mid-August 2016.
The MHP plans to install and begin implementation of a Telechart Patient Portal
following system upgrade in September 2016.
The MHP plan to develop and implement Client Portal documents, client training and
program security.
The MHP continues to pursue Meaningful Use, Stage I attestation for the EHR incentive
program with a projected target date of October 2016.
The MHP plans to begin test, training and implementation of the non-axial diagnosis
review form.
Page 43
OTHER SIGNIFICANT ISSUES
The MHP states that some of the clinicians are unable to perform the electronic provider
signature function when using their laptops out in the field.
MEDI-CAL CLAIMS PROCESSING
Normal cycle for submitting current fiscal year Medi-Cal claim files:
☒ Monthly ☐ More than 1x month ☐ Weekly ☐ More than 1x weekly
MHP performs end-to-end (837/835) claim transaction reconciliations:
☒ Yes ☐ No
If yes, product or application:
Kings View Reports
Method used to submit Medicare Part B claims:
☐ Clearinghouse ☒ Electronic ☐ Paper
INFORMATION SYSTEMS REVIEW FINDINGS—IMPLICATIONS
Access to Care
o The MHP mandated use of CCBH Scheduler to produce weekly reports to
determine what percentage of consumers are walk-ins without appointments.
Number
Submitted
Gross Dollars
Billed
Dollars
Denied
Percent
Denied
Number
Denied
Gross Dollars
Adjudicated
Claim
Adjustments
Gross Dollars
Approved
14,529 $2,744,177 $31,417 1.14% 255 $2,712,760 $66,501 $2,646,259
Table 13 - Colusa MHP Summary of CY15 Processed SDMC Claims
Note: Includes services provided during CY15 with the most recent DHCS processing date of May 19,2016
Page 44
o The MHP did not delineate between adults and children when tracking length of
time from initial contact to first psychiatric appointment for FY16-17. The MHP
did delineate between adults and children for FY15-16. The MHP reports this
was due to losing their child psychiatrist in June 2015.
Timeliness of Services
o The MHP does not track and trend no shows. However, the MHP noted that they
only recently were able to extract data from their EHR. The MHP would like
direction in tracking and trending from other counties as well as dialogue on
what other counties’ interventions have been successful to reduce their no show
rate.
Quality of Care
o The MHP plans to install SQL upgrade, followed by CCBH Promotion 222, and
then Ultra-Sensitive Exchange with Client Portal.
o The MHP has plans to develop and implement Client Portal documents, client
training and program security following.
o The MHP has recruited a Quality Improvement Coordinator who is a licensed
clinician to begin in August 2016. Her primary job responsibilities may
encompass more intensive data analytical tasks.
Consumer Outcomes
o The MHP amended both the Medical Necessity form and Discharge Assessment
form to allow entry and scoring of the Children’s Behavioral Checklist (CBCL),
which measures outcomes.
o The MHP did a Clinical PIP on utilizing the MORs to assess how interventions
assist consumers in recovery.
Page 45
SITE REVIEW PROCESS BARRIERS
The following conditions significantly affected CalEQRO’s ability to prepare for and/or conduct a
comprehensive review:
Review planning was challenging as the MHP’s Compliance Officer/Consultant works
one day per week, with communication being limited to that day. Further, on the date of
the review and due to unforeseen circumstances, the Compliance Officer/Consultant
was not able to attend. Alternatively, a conference call was held the following week for
further discussion.
Page 46
CONCLUSIONS
During the FY16-17 annual review, CalEQRO found strengths in the MHP’s programs, practices, or
information systems that have a significant impact on the overall delivery system and its
supporting structure. In those same areas, CalEQRO also noted opportunities for quality
improvement. The findings presented below relate to the operation of an effective managed care
organization, reflecting the MHP’s processes for ensuring access to and timeliness of services and
improving the quality of care.
STRENGTHS AND OPPORTUNITIES
Access to Care
Strengths:
o With its Cultura Vida program, the MHP has a robust outreach program in the
Hispanic community, with over 12 presentations on services and mental health
issues.
o The MHP has forged strong collaborative relationships with both Glenn and
Sutter-Yuba MHPs to serve Colusa County consumers post hospital closure.
Opportunities:
o The MHP has been experiencing difficulties with its crisis protocol since April
2016 with the closure of the only hospital in the county.
o The MHP does not have a formalized cultural competence committee, nor has it
updated its Cultural Competency Plan since FY11/12.
Timeliness of Services
Strengths:
o The MHP is responsive in providing intake assessments and responding to
needs for urgent appointments, consistently meeting their timeliness goals.
Opportunities:
o The MHP tracked their adult and children separately on the Timely Self-
Assessment for FY15-16 for the length of time from assignment after
assessment to first psychiatric appointment but did not track separately for
FY16-17. Further, they set a goal of 45 days from assignment after assessment to
first psychiatry appointment.
o The MHP changed its protocol for medication-only clients and eliminated
medication only clients for children’s services, requiring that children be in
other services to be eligible for medication services. The MHP has the
Page 47
opportunity to track timeliness for psychiatry for both children and adults,
separately as before, to determine the impact of the policy on both timeliness of
service, access and quality of care.
o The MHP has a need for full tracking and trending of crisis data for effective and
efficient staff utilization in response to hospital closure issues and meeting crisis
caseloads. The MHP has a need to evaluate the increase in hospitalization rates.
o The MHP has not yet developed a format for tracking and trending no shows.
Consulting with nearby counties on their criteria and methodology for tracking
and trending no shows as well as creating and enforcing a policy and procedure
could be beneficial.
Quality of Care
Strengths:
o The MHP hired a new QI Coordinator who will be responsible for data analysis
and trending.
o In spring of 2016, a county administrative officer was hired, with MHP
leadership as a direct report.
Opportunities:
o The MHP has recruited a QI Coordinator who is a licensed clinician to begin in
August 2016. Her primary job responsibilities may encompass more intensive
analytical tasks.
o The MHP has the opportunity to improve bi-directional communication and
implementation of staff feedback for effective resolution and involvement of
staff at higher levels of program planning and service delivery.
Consumer Outcomes
Strengths:
o The MHP utilizes both Milestones of Recovery Scores (MORS) for adults and has
utilized aggregate data for their Clinical PIP.
Opportunities:
o CBCL for children was installed December 2015 and training was done in
January 2016. No aggregate reporting and analysis has been done as yet with
CBCL.
Page 48
RECOMMENDATIONS
Investigate the feasibility of additional funding or resources to support crisis staff,
which currently is an informal mobile crisis unit with no vehicle. This is relevant and
timely with the recent closure of the local hospital and their ER services.
Evaluate contributing factors for the increase in hospitalization rates from FY 14-15 to
FY15-16. Use data to monitor and evaluate consumers’ access to crisis services as it relates
to the hospital closure. Use data on the contributing factors to identify gaps in service and
potential solutions.
Re-establish and formalize the Cultural Competence Committee with regularly
scheduled meetings to address other aspects of cultural responsiveness not limited
to Hispanic outreach, considering underserved populations, and gaps in staff
training. Actively recruit representatives from the community for stakeholder
participation and their input.
Evaluate current timeliness tracking procedures, and expand reporting (separately) on both
adult and children’s measures for all measures contained in the timeliness self-assessment,
including no shows. Track timeliness for psychiatry for both children and adults, to
determine the impact of the .5 elimination of child psychiatry and the implementation of the
policy requiring conjoint services for medication child clients.
Expand EHR functionality through consultation with Kings View and nearby
counties. Quantifiably show improvement and expansion through the addition of
applications, such as eLabs, Level of Care/Level of Service, Alerts, etc.
Page 49
ATTACHMENTS
Attachment A: Review Agenda
Attachment B: Review Participants
Attachment C: Approved Claims Source Data
Attachment D: CalEQRO PIP Validation Tools
Page 50
ATTACHMENT A—REVIEW AGENDA
Colusa EQRO Review FY16-17 Final Agenda 1
Behavioral Health Concepts, Inc. - California EQRO
5901 Christie Ave., Suite 502, Emeryville, CA94608 (855) 385-3776 www.caleqro.com
Colusa County MHP CalEQRO Agenda
Day 1 Thursday, August 4, 2016 All sessions located at Colusa County Behavioral Health 162 E. Carson St. Colusa, CA 95932 unless otherwise noted.
Time Activity
8:30 a.m. – 10 :00 am
Opening Session
Introduction to BHC
MHP Team Introductions Review of Past Year
Significant Changes and Key Initiatives
Responses to Last Year’s Recommendations
Use of Data in the Past Year
State Survey (#s, use of) Participants: MHP Leadership, Quality Management Staff, Key Stakeholders
Location: ROOM 102
BHC Staff: All
1 0 : 0 0 a m – 1 0 : 1 5 a m Break
1 0 : 1 5 a m – 1 2 : 0 0 p m MHP Clinical Staff Group Interview 6-8 Clinical line staff, representing various geographical regions of the county, including crisis staff, with no supervisory level staff included.
Location: ROOM 102 BHC Staff: CE, JL
Billing/It Key Staff Group Interview • Recommendation FY15-16 • ISCA Review • Continuous Training Initiative • CHAT Program Implementation • Analysis & Use of Data • Kings View/Dashboards • Process to Review Denied Claims and Claims Reconciliation • EHR Deployment (new functions in past year, pending enhancements) • Consumer Personal Health Record
Location: “SUNSHINE” ROOM
BHC Staff: JT
1 2 : 0 0 p m – 1 : 0 0 p m BHC Working Lunch Meeting
1 : 0 0 p m – 2 : 3 0 p m
Consumer/Family Member Focus Group 8-10 culturally diverse adult beneficiaries representing both high and low utilizers of service.
Access, Timeliness, Outcomes, and Quality • Timeliness Self-Assessment Document • MHP Timeliness Metrics and Procedures • Access • Medi-Cal Penetration Rates
Colusa EQRO Review FY16-17 Final Agenda 2
Time Activity
Location: ROOM 102
BHC Staff: JL
• MHP Cultural Competence Metrics and Procedures Location: “SUNSHINE” ROOM BHC Staff: CE, JT
2 : 3 0 p m – 3 : 3 0 p m Katie A. Implementation •Overview of current Katie A services, coordination, and future strategies •Technical Assistance Location: “Sunshine” Room
BHC Staff: JT
Performance Improvement Projects •Review Clinical PIP •Review Non-Clinical PIP •Technical Assistance
Location: Room 102 BHC Staff: CE, JL
3 : 3 0 p m – 3 : 4 0 p m BHC Staff Travel to Safe Haven Site
3 : 4 0 p m – 4 : 0 5 p m Safe Haven Wellness Center Visit Location: Safe Haven 411 MAIN STREET, COLUSA BHC Staff: All
4: 15 pm - 4 : 45 pm Exit Interview if needed for questions and next steps. • Summary of Findings • Collection of Requested Documentation • Next Steps
Location: ROOM 102
BHC Staff: All
CalEQRO Review Team: Cyndi Eppler – Lead Quality Reviewer [email protected]
Judith Toomasson – Information Systems Reviewer [email protected]
Janyce Leathers – Consumer/Family Member Consultant www.CalEQRO.com
Page 51
ATTACHMENT B—REVIEW PARTICIPANTS
Page 52
CALEQRO REVIEWERS CyndiEppler–LeadQualityReviewerJudithToomasson–InformationSystemsReviewerJanyceLeathers–Consumer/FamilyMemberConsultant
AdditionalCalEQROstaffmemberswereinvolvedinthereviewprocess,assessments,andrecommendations.Theyprovidedsignificantcontributionstotheoverallreviewbyparticipatinginboththepre‐siteandthepost‐sitemeetingsand,ultimately,intherecommendationswithinthisreport.
SITES OF MHP REVIEW
MHP SITES
ColusaCountyBehavioralHealth162E.CarsonSt.Colusa,CA95932SafeHavenWellnessCenter,411MainStreet,Colusa,CA95932
CONTRACT PROVIDER SITES
Nocontractprovidersiteswerevisited.
PARTICIPANTS REPRESENTING THE MHP
Name Position Agency
Angela Shields Family Specialist CCBH
Chantelle Estess Office Supervisor CCBH
Deana Fleming Deputy Director Adult Services CCBH
Elaine S. McCord EHR Coordinator CCBH
Emily Clark Therapist CCBH
Jack Joiner Compliance Officer, Consultant CCBH
James Balderama Case Manager CCBH
Jan Morgan Deputy Director Children’s Services CCBH
Jeannie Scroggins Therapist CCBH
Kim Perry Therapist CCBH
Mark McGregor Clinical Program Manager Children’s Services CCBH
Michael Laffin Deputy Director of Administration & Finance CCBH
Senaida Rangel MHSA Coordinator CCBH
Shannon Piper Clinical Program manager Adult Services CCBH
Page 53
Terrence Rooney Director of Behavioral Health CCBH
Tomika Arce Billing Department CCBH
Tracy Woo Quality Improvement Coordinator, Consultant CCBH
William McCloud EHR Manager CCBH
Page 54
ATTACHMENT C—APPROVED CLAIMS SOURCE DATA
Page 55
ThesedataareprovidedtotheMHPinaHIPAA‐compliantmanner.
TwoadditionaltablesareprovidedbelowonMedi‐CalACAExpansionbeneficiariesandMedi‐Calbeneficiariesservedbycostbands.
TableC1(AttachmentC)showsthepenetrationrateandapprovedclaimsperbeneficiaryfortheCY15Medi‐CalACAExpansionPenetrationRateandApprovedClaimsperBeneficiary.
TableC2(AttachmentC)showsthedistributionoftheMHPCY15DistributionofBeneficiariesbyApprovedClaimsperBeneficiary(ACB)Rangeforthevariouscategories;under$20,000;$20,000to$30,000,andthoseabove$30,000.
Entity
Average Monthly
ACA Enrollees
Number of
Beneficiaries
Served
Penetration
Rate
Total Approved
Claims
Approved Claims
per Beneficiary
Statwide 2,001,900 131,350 6.56% $533,318,886 $4,060
Small‐Rural 17,753 1,992 11.22% $5,569,311 $2,796
Colusa 803 122 15.19% $387,591 $3,177
Table C1 ‐ CY15 Medi‐Cal Expansion (ACA) Penetration Rate and Approved Claims per Beneficiary
Range of ACB
MHP Count of
Beneficiaries
Served
MHP
Percentage
of
Beneficiaries
Statewide
Percentage
of
Beneficiaries
MHP Total
Approved
Claims
MHP
Approved
Claims per
Beneficiary
Statewide
Approved
Claims per
Beneficiary
MHP
Percentage
of Total
Approved
Claims
Statewide
Percentage
of Total
Approved
Claims
$0K ‐ $20K 500 95.60% 94.46% $1,634,554 $3,269 $3,553 61.51% 61.20%
>$20K ‐ $30K 9 1.72% 2.67% $212,700 $23,633 $24,306 8.00% 11.85%
>$30K 14 2.68% 2.86% $810,021 $57,859 $51,635 30.48% 26.96%
Table C2 ‐ Colusa MHP CY15 Distribution of Beneficiaries by ACB Range
Page 54
ATTACHMENT D—PIP VALIDATION TOOL
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 1 of 12
PERFORMANCE IMPROVEMENT PROJECT (PIP) VALIDATION WORKSHEET FY16-17
GENERAL INFORMATION
MHP: Colusa ☒ Clinical PIP ☐ Non-Clinical PIP
PIP Title: Recovery
Start Date (MM/DD/YY): January 2015
Completion Date (MM/DD/YY): ongoing
Projected Study Period (#of Months): 3 year
Completed: Yes ☐ No ☒
Date(s) of On-Site Review (MM/DD/YY): 08/04/2016
Name of Reviewer: Cyndi Eppler
Status of PIP (Only Active and ongoing, and completed PIPs are rated):
Rated
☒ Active and ongoing (baseline established and interventions started)
☐ Completed since the prior External Quality Review (EQR)
Not rated. Comments provided in the PIP Validation Tool for technical assistance purposes only.
☐ Concept only, not yet active (interventions not started)
☐ Inactive, developed in a prior year
☐ Submission determined not to be a PIP
Brief Description of PIP (including goal and what PIP is attempting to accomplish):
The MHP, as a result of its participation in the Advancing Recovery Collaborative, collected MORS scores for all adult consumers beginning in January 2015. Higher than expected MORS scores for 13% of the study population led to the MHP focusing on transitioning consumers to lower levels of care. The MHP aimed to improve outcomes for adults, including “graduating” from services of the MHP. The PIP examined which interventions assisted consumers in recovery as measured using the Milestones of Recovery outcome tool (MORs), leading to shortened and consistent time to transition to lower level levels of care.
The MHP collected MORS scores monthly and found that individuals at MORS 7 were able to transition out of services, as well as those at MORS 6. The MHP then focused on individuals with a MORS score of 5. In contrast to the MORS 6 and 7 individuals, these individuals were not able to move quickly up the MORS scale. These individuals stayed at MORS 5 throughout the measurement cycle. Previously successful Interventions were doubled (Strength Assessment and Group Supervision). Even with these interventions, little movement was seen for the MORS 5 individuals. The MHP is now testing the outcome tools that
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 2 of 12
may be more sensitive to smaller changes (PHQ 9 and the GAD 7) for individuals with depression or anxiety diagnoses as they can be administered more frequently.
The study question design did not lend itself to be answerable through the PIP process. Recommend that for next PIP, set up question so that it is quantifiable and contains indicators, i.e. “Will ____ intervention improve ________(symptoms/problem) as measured by ________outcome tool?” As it stands, the PIP’s design does not allow the MHP to track the new outcome tools and does not identify which interventions improve which symptoms. Clarification on whether MORs 6s and 7s entered at the 5 level when they came in for services initially. This would allow for a meaningful comparison regarding how long it took to get to the next level. For next steps, the MHP plans to amend its question, evaluate the impact of strengths assessment on “leveling up” using the MORs, in conjunction with the anxiety and depression inventory tools. Recommended that this PIP be considered completed rather than going into year 3 of this PIP. Instead, develop a new PIP evaluating clinical outcomes post intervention using the anxiety and depression assessments. As it stands, this process didn’t make an improvement for clients as they were still discharged for the same reason which was not a result of better implementation of the MORS. Rather, the PIP provided more information regarding level of care and helped inform the administrative process of discharge/level of care.
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 3 of 12
ACTIVITY 1: ASSESS THE STUDY METHODOLOGY
STEP 1: Review the Selected Study Topic(s)
Component/Standard Score Comments
1.1 Was the PIP topic selected using stakeholder input? Did the MHP develop a multi-functional team compiled of stakeholders invested in this issue?
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
The persons involved in this PIP are noted by role above. Specifically they are: Deana Fleming, LCSW, Deputy Director, Clinical, Adult; Janet Morgan, LCSW, Deputy Director, Clinical, Children; Veronica Vasquez, Fiscal; Jack Joiner, LMFT, Consultant.
1.2 Was the topic selected through data collection and analysis of comprehensive aspects of enrollee needs, care, and services?
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Select the category for each PIP:
Clinical:
☐ Prevention of an acute or chronic condition ☐ High volume services
☒ Care for an acute or chronic condition ☐ High risk conditions
Non-Clinical:
☐ Process of accessing or delivering care
1.3 Did the Plan’s PIP, over time, address a broad spectrum of key aspects of enrollee care and services?
Project must be clearly focused on identifying and correcting deficiencies in care or services, rather than on utilization or cost alone.
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
The study topic of recovery is clearly the most important element in consumers’ “getting better”. We are committed to doing our best as a Department in helping consumers experience improvement in their ability to be an active participant in society
1.4 Did the Plan’s PIPs, over time, include all enrolled populations (i.e., did not exclude certain enrollees such as those with special health care needs)?
Demographics:
☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☐ Other
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Participants in this study come from throughout Colusa County, and are receiving treatment in the only clinic in the County, in Colusa. All participants are adults. All participants are engaged in outpatient treatment and are not “meds only” consumers.
Totals 4 Met Partially Met Not Met UTD
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 4 of 12
STEP 2: Review the Study Question(s)
2.1 Was the study question(s) stated clearly in writing?
Does the question have a measurable impact for the defined study population?
Include study question as stated in narrative:
Can the Milestones of Recovery (MORS) rating scale be used to effectively guide treatment interventions?
☐ Met
☐ Partially Met
☒ Not Met
☐ Unable to Determine
Question is not measurable, does not contain indicators. Examples of measurable question with indicator as follows:
Will the use of ________intervention(s) move consumers to the next level of care within _______ time period/within certain number of sessions?
or
What impact will the use of ________intervention(s) have on consumer outcomes as measured by MORs (or other outcome measurement tool)?
Totals Met Partially Met 1 Not Met UTD
STEP 3: Review the Identified Study Population
3.1 Did the Plan clearly define all Medi-Cal enrollees to whom the study question and indicators are relevant?
Demographics:
☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☐ Other
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Participants in this study come from throughout Colusa County, and are receiving treatment in the only clinic in the County, in Colusa. All participants are adults. All participants are engaged in outpatient treatment and are not “meds only” consumers.
3.2 If the study included the entire population, did its data collection approach capture all enrollees to whom the study question applied?
Methods of identifying participants:
☒ Utilization data ☐ Referral ☐ Self-identification
☐ Other: Text if checked
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Totals 2 Met Partially Met Not Met UTD
STEP 4: Review Selected Study Indicators
4.1 Did the study use objective, clearly defined, measurable indicators?
List indicators:
MORS score at least 7 for three months
MORS score of 5 for three months
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
The indicators were objective and measureable.
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 5 of 12
4.2 Did the indicators measure changes in: health status, functional status, or enrollee satisfaction, or processes of care with strong associations with improved outcomes? All outcomes should be consumer focused.
☐ Health Status ☒ Functional Status
☐ Member Satisfaction ☐ Provider Satisfaction
Are long-term outcomes clearly stated? ☐ Yes ☒ No
Are long-term outcomes implied? ☒ Yes ☐ No
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
MORs scores for the identified individuals were tracked.
Totals 2 Met Partially Met Not Met UTD
STEP 5: Review Sampling Methods
5.1 Did the sampling technique consider and specify the:
a) True (or estimated) frequency of occurrence of the event?
b) Confidence interval to be used?
c) Margin of error that will be acceptable?
☐ Met
☐ Partially Met
☒ Not Met
☐ Not Applicable
☐ Unable to Determine
As this PIP has progressed the MHP have found that individuals at MORS score 7 were indeed ready for discharge planning and in fact 10 of the 14 identified individuals did leave treatment. The MHP also found that for individuals with a greater level of impairment (as indicated by a MORS score of 5) required a more focused approach. The MHP was unable to focus this greater level of attention on all MORS 5 individuals so instead the MHP required each clinician to identify one client at this level to be the focus the interventions in this PIP. Each identified consumer participated in a Strengths Assessment and a Personal Recovery Plan and each clinician then presented this information in a Group Supervision model. As the PIP focus on MORS level 5 individuals continued it became clear that the MORS was good at identifying recovery level in a broad way but did not reflect the partial changes needed within this level to move forward in recovery. Unlike individuals at MORS 7 who were quickly able to move to discharge, individuals at MORS 5 have much more work to do before they are able to move up to MORS 6.
This sampling technique does not appear to be grounded in a way that prevents bias, rather the individuals were selected by level.
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 6 of 12
5.2 Were valid sampling techniques that protected against bias employed?
Specify the type of sampling or census used:
Text
☒ Met
☐ Partially Met
☐ Not Met
☐ Not Applicable
☐ Unable to Determine
All MORs 7, 6, 5 Clients were included.
5.3 Did the sample contain a sufficient number of enrollees?
______N of enrollees in sampling frame
______N of sample
______N of participants (i.e. – return rate)
☐ Met
☒ Partially Met
☐ Not Met
☐ Not Applicable
☐ Unable to Determine
Sample size was based on what each clinician could accomplish. The result was a sample size of 8. The sample size was not grounded in any statistical analysis. The MHP did not justify this number and show that those benefitting from the PIP are not disproportionately one “type of client.”
Totals 1 Met 1 Partially Met 1 Not Met UTD
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 7 of 12
STEP 6: Review Data Collection Procedures
6.1 Did the study design clearly specify the data to be collected?
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
MORS scores for the identified individuals.
6.2 Did the study design clearly specify the sources of data?
Sources of data:
☐ Member ☐ Claims ☐ Provider
☐ Other: Text if checked
☐ Met
☒ Partially Met
☐ Not Met
☐ Unable to Determine
MORS scores monthly on “MORS Monday”
6.3 Did the study design specify a systematic method of collecting valid and reliable data that represents the entire population to which the study’s indicators apply?
☐ Met
☒ Partially Met
☐ Not Met
☐ Unable to Determine
Methodology description did not provide any specifics or detail.
6.4 Did the instruments used for data collection provide for consistent, accurate data collection over the time periods studied?
Instruments used:
☐ Survey ☐ Medical record abstraction tool
☒ Outcomes tool ☐ Level of Care tools
☐ Other: Text if checked
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
6.5 Did the study design prospectively specify a data analysis plan?
Did the plan include contingencies for untoward results?
☐ Met
☐ Partially Met
☒ Not Met
☐ Unable to Determine
The data is extracted by a fiscal person who is familiar with the EHR. Analysis plan not specified.
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 8 of 12
6.6 Were qualified staff and personnel used to collect the data?
Project leader:
Name: Deana Flemming, LCSW
Title: Deputy Director Clinical, Adult
Role: Project leader
Other team members:
Names: PIP committee
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Clinicians obtained the MORS score with their respective clients.
Totals 3 Met 2 Partially Met 1 Not Met UTD
STEP 7: Assess Improvement Strategies
7.1 Were reasonable interventions undertaken to address causes/barriers identified through data analysis and QI processes undertaken?
Describe Interventions:
MORS Scoring
Strength Assessment
Personal Recovery Planning
Group Supervision
☐ Met
☐ Partially Met
☐ Not Met
☒ Unable to Determine
The interventions did not inform the study question in a way that measurable and quantifiable.
Totals Met Partially Met Not Met NA 1 UTD
STEP 8: Review Data Analysis and Interpretation of Study Results
8.1 Was an analysis of the findings performed according to the data analysis plan?
This element is “Not Met” if there is no indication of a data analysis plan (see Step 6.5)
☐ Met
☐ Partially Met
☒ Not Met
☐ Not Applicable
☐ Unable to Determine
Data analysis plan was not provided.
8.2 Were the PIP results and findings presented accurately and clearly?
Are tables and figures labeled? ☐ Yes ☐ No
Are they labeled clearly and accurately? ☐ Yes ☐ No
☒ Met
☐ Partially Met
☐ Not Met
☐ Not Applicable
☐ Unable to Determine
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 9 of 12
8.3 Did the analysis identify: initial and repeat measurements, statistical significance, factors that influence comparability of initial and repeat measurements, and factors that threaten internal and external validity?
Indicate the time periods of measurements: monthly___________
Indicate the statistical analysis used:__n/a_________________
Indicate the statistical significance level or confidence level if available/known:_______% ______Unable to determine
☐ Met
☐ Partially Met
☒ Not Met
☐ Not Applicable
☐ Unable to Determine
8.4 Did the analysis of the study data include an interpretation of the extent to which this PIP was successful and recommend any follow-up activities?
Limitations described:
Text
Conclusions regarding the success of the interpretation:
Based on the learning that has and is occurring from this PIP we consider it successful. We are learning the short comings of the MORS and are testing other measurements to see if more minor but important changes can assist in guiding treatment.
Recommendations for follow-up:
Text
☐ Met
☐ Partially Met
☒ Not Met
☐ Not Applicable
☐ Unable to Determine
This discovery of little apparent progress for the MORS 5 individuals also lead to a question about discharges in general. The MHP queried the EHR to learn what the most frequent reason for discharge was. They were surprised to find that almost all discharges were recorded as “administrative”. We knew this was not clinically correct so we launched a project to compare clinical reasons for discharge versus what the final recorded reason was. The MHP found that there was indeed a difference between what the clinician had recorded as the reason for discharge in one part of the EHR versus what was recorded as the final discharge reason. The two main reasons for the difference were a lack of communication between clinical and clerical staff regarding reason for discharge, and a lack of discharge codes to reflect a consumer who was seen in crisis only and did not require additional services (this could only be recorded as administrative). Additional codes were subsequently added to the system for these brief, self-contained episodes of care. Communications of discharge reasons between clerical and clinical staff were also emphasized.
This process didn’t make an improvement for clients as they were still discharged for the same reason which was not a result of better implementation of the MORS. Rather, the PIP provided more information regarding level of care and helped inform the administrative process of discharge/level of care.
Totals 1 Met Partially Met 3 Not Met NA UTD
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 10 of 12
STEP 9: Assess Whether Improvement is “Real” Improvement
9.1 Was the same methodology as the baseline measurement used when measurement was repeated?
Ask: At what interval(s) was the data measurement repeated?
Were the same sources of data used?
Did they use the same method of data collection?
Were the same participants examined?
Did they utilize the same measurement tools?
☐ Met
☐ Partially Met
☐ Not Met
☐ Not Applicable
☒ Unable to Determine
For meaningful data comparison, the timeframe for a client with MORs level 6 to become a level 7 cannot be compared to timeframes that a level 5 takes to become a 6 unless, the 6s and 7s came in at a level 5.
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 11 of 12
9.2 Was there any documented, quantitative improvement in processes or outcomes of care?
Was there: ☒ Improvement ☐ Deterioration
Statistical significance: ☐ Yes ☒ No
Clinical significance: ☒ Yes ☐ No
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
The MHP was not able to do the analysis of 9.3 and 9.4, therefore there isn’t documented improvement which is known to be the result of the PIP.
9.3 Does the reported improvement in performance have internal validity; i.e., does the improvement in performance appear to be the result of the planned quality improvement intervention?
Degree to which the intervention was the reason for change:
☐ No relevance ☐ Small ☐ Fair ☐ High
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
9.4 Is there any statistical evidence that any observed performance improvement is true improvement?
☐ Weak ☐ Moderate ☐ Strong
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
9.5 Was sustained improvement demonstrated through repeated measurements over comparable time periods?
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
Totals 2 Met Partially Met Not Met 3 NA UTD
Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 12 of 12
ACTIVITY 2: VERIFYING STUDY FINDINGS (OPTIONAL)
Component/Standard Score Comments
Were the initial study findings verified (recalculated by CalEQRO) upon repeat measurement?
☐ Yes
☒ No
n/a
ACTIVITY 3: OVERALL VALIDITY AND RELIABILITY OF STUDY RESULTS: SUMMARY OF AGGREGATE VALIDATION FINDINGS
Conclusions:
Met 13
Partially Met 3
Not Met 6
UTD 2
# Not applicable 4
# Applicable 24
Score 60.42%
Recommendations:
The study question design did not lend itself to be answerable through the PIP process. Recommend that for next PIP, set up question so that it is quantifiable and contains indicators, i.e. “Will ____ intervention improve ________(symptoms/problem) as measured by ________outcome tool?” As it stands, the PIP’s design does not allow the MHP to track the new outcome tools and does not identify which interventions improve which symptoms. Clarification on whether MORs 6s and 7s entered at the 5 level when they came in for services initially. This would allow for a meaningful comparison regarding how long it took to get to the next level. For next steps, the MHP plans to amend its question, evaluate the impact of strengths assessment on “leveling up” using the MORs, in conjunction with the anxiety and depression inventory tools.
This process didn’t make an improvement for clients as they were still discharged for the same reason which was not a result of better implementation of the MORS. Rather, the PIP provided more information regarding level of care and helped inform the administrative process of discharge/level of care.
Check one: ☐ High confidence in reported Plan PIP results ☐ Low confidence in reported Plan PIP results
☐ Confidence in reported Plan PIP results ☐ Reported Plan PIP results not credible
☒ Confidence in PIP results cannot be determined at this time
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 1 of 14
PERFORMANCE IMPROVEMENT PROJECT (PIP) VALIDATION WORKSHEET FY16-17
GENERAL INFORMATION
MHP: Colusa ☐ Clinical PIP ☒ Non-Clinical PIP
PIP Title: Parent Intervention
Start Date (MM/DD/YY): August 2015
Completion Date (MM/DD/YY): ongoing
Projected Study Period (#of Months): 2 years
Completed: Yes ☐ No ☒
Date(s) of On-Site Review (MM/DD/YY): 08/04/16
Name of Reviewer: Cyndi Eppler
Status of PIP (Only Active and ongoing, and completed PIPs are rated):
Rated
☒ Active and ongoing (baseline established and interventions started)
☐ Completed since the prior External Quality Review (EQR)
Not rated. Comments provided in the PIP Validation Tool for technical assistance purposes only.
☐ Concept only, not yet active (interventions not started)
☐ Inactive, developed in a prior year
☐ Submission determined not to be a PIP
Brief Description of PIP (including goal and what PIP is attempting to accomplish):
This PIP is at the beginning of its second year. Year 1 focused on shortening duration of services with the addition of conjoint family services as a standard
protocol. The second year of the PIP is aimed at evaluating the impact of conjoint (automatically assigned) family interventions, specifically Parent-Child
Interaction Therapy (PCIT) on client functioning. The overarching goal of this PIP will be to improve child functioning in many areas relevant to the child’s progress toward resilience. The MHP analyzed data that compared the length of treatment for children receiving family therapy of any kind or duration beyond
intake as part of their treatment with children who had not. The MHP looked at this data for fiscal year 2014/2015 and found that 145 children had received family therapy, where 62 children had not. The When comparing time in treatment for the two groups, the MHP found that children receiving any form of
family involvement (145 children) averaged 3.763889 months in care and those that received no form of family involvement (63 children) averaged 3.967742
months. Remeasurement has proven to be a challenge. The MHP had difficulty rolling out the CBCL due to scoring challenges and difficulty in getting the score entered into the EHR. Additionally, the MHP discovered that they were not getting discharge CBCL scores because families were choosing to withdraw from
care when the clinician began discussing the end of treatment. Further, no one has been in care for a full year since the implementation of the CBCL. The MHP experienced similar issues with the parent satisfaction survey. A way to ameliorate this type of issue would be to collect data on a quarterly basis.
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 2 of 14
Feedback included acknowledgment that the PIP focus seemed to naturally shift to evaluating the PCIT intervention. The study question needed clarification
so that it was measurable an objective. The interventions listed are not truly the intervention, but rather assigning all mild to moderate clients to PCIT and all moderate to severe clients to individual as a matter of standard procedure. It was recommended that the MHP then compare the CBCL scores of from a
baseline at start of treatment after a set period of time (i.e. 3 months). This would allow the county to see the true effectiveness of the new administrative procedure.
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 3 of 14
ACTIVITY 1: ASSESS THE STUDY METHODOLOGY
STEP 1: Review the Selected Study Topic(s)
Component/Standard Score Comments
1.1 Was the PIP topic selected using stakeholder input? Did the MHP develop a multi-functional team compiled of stakeholders invested in this issue?
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Director (Terry Rooney), two Deputy Directors (Jan Morgan and Deana Fleming), a data person (Veronica Vasquez) and a consultant (Jack Joiner). Also planned is the inclusion of parents who have received this service, initially via the locally developed satisfaction survey.
1.2 Was the topic selected through data collection and analysis of comprehensive aspects of enrollee needs, care, and services?
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Select the category for each PIP:
Clinical:
☐ Prevention of an acute or chronic condition ☐ High volume services
☐ Care for an acute or chronic condition ☐ High risk conditions
Non-Clinical:
☒ Process of accessing or delivering care All children entering system of care will receive conjoint family services.
1.3 Did the Plan’s PIP, over time, address a broad spectrum of key aspects of enrollee care and services?
Project must be clearly focused on identifying and correcting deficiencies in care or services, rather than on utilization or cost alone.
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
1.4 Did the Plan’s PIPs, over time, include all enrolled populations (i.e., did not exclude certain enrollees such as those with special health care needs)?
Demographics:
☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☐ Other
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Children entering care after start of PIP, which to date, was 208 children. Approximately 63 children were moderate to severe, while 145 children were mild to moderate.
Totals 4 Met Partially Met Not Met UTD
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 4 of 14
STEP 2: Review the Study Question(s)
2.1 Was the study question(s) stated clearly in writing?
Does the question have a measurable impact for the defined study population?
Include study question as stated in narrative:
“Will the provision parental intervention result in shorter duration of care, higher levels of parent satisfaction with care, and improved functioning as measured by the Child Behavior Checklist?”
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Totals 1 Met Partially Met Not Met UTD
STEP 3: Review the Identified Study Population
3.1 Did the Plan clearly define all Medi-Cal enrollees to whom the study question and indicators are relevant?
Demographics:
☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☐ Other
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Children entering care after start of PIP. Approximately 63 children were moderate to severe, while 145 children were mild to moderate.
3.2 If the study included the entire population, did its data collection approach capture all enrollees to whom the study question applied?
Methods of identifying participants:
☒ Utilization data ☐ Referral ☐ Self-identification
☐ Other:
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Children entering care after start of PIP. Approximately 63 children were moderate to severe, while 145 children were mild to moderate.
Totals 2 Met Partially Met Not Met UTD
STEP 4: Review Selected Study Indicators
4.1 Did the study use objective, clearly defined, measurable indicators?
List indicators:
Child Behavior Checklist Score
Length of time in care
Scores on parental satisfaction survey after parental intervention
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 5 of 14
4.2 Did the indicators measure changes in: health status, functional status, or enrollee satisfaction, or processes of care with strong associations with improved outcomes? All outcomes should be consumer focused.
☐ Health Status ☒ Functional Status
☐ Member Satisfaction ☒ Provider Satisfaction
Are long-term outcomes clearly stated? ☐ Yes ☒ No
Are long-term outcomes implied? ☒ Yes ☐ No
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Totals Met Partially Met Not Met UTD
STEP 5: Review Sampling Methods
5.1 Did the sampling technique consider and specify the:
a) True (or estimated) frequency of occurrence of the event?
b) Confidence interval to be used?
c) Margin of error that will be acceptable?
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
5.2 Were valid sampling techniques that protected against bias employed?
Specify the type of sampling or census used:
From UR data.
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
All children receiving services were/will be included once all numbers of sessions are completed.
5.3 Did the sample contain a sufficient number of enrollees?
______N of enrollees in sampling frame
______N of sample
______N of participants (i.e. – return rate)
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
Totals Met Partially Met Not Met UTD
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 6 of 14
STEP 6: Review Data Collection Procedures
6.1 Did the study design clearly specify the data to be collected?
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
CBCL score, Parent Satisfaction score, Length of treatment.
6.2 Did the study design clearly specify the sources of data?
Sources of data:
☒ Member ☒ Claims ☒ Provider
☒ Other: EHR, Survey, Outcome tool
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
6.3 Did the study design specify a systematic method of collecting valid and reliable data that represents the entire population to which the study’s indicators apply?
☐ Met
☒ Partially Met
☐ Not Met
☐ Unable to Determine
The data will be collected by clinical staff for CBCL and Parent Satisfaction. Length of treatment will be calculated by fiscal support staff. Parents not staying to complete the CBCL is interfering with data collection.
6.4 Did the instruments used for data collection provide for consistent, accurate data collection over the time periods studied?
Instruments used:
☒ Survey ☐ Medical record abstraction tool
☒ Outcomes tool ☐ Level of Care tools
☐ Other: Text if checked
☐ Met
☒ Partially Met
☐ Not Met
☐ Unable to Determine
The CBCL score will be entered into the EHR. Parent Satisfaction scores will be tabulated by a PIP Committee member. Length of treatment will be calculated as noted above. Parents not staying to complete the CBCL is interfering with data collection.
6.5 Did the study design prospectively specify a data analysis plan?
Did the plan include contingencies for untoward results?
☐ Met
☐ Partially Met
☒ Not Met
☐ Unable to Determine
The data will be analyzed by the Committee. No detail or methodology was described.
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 7 of 14
6.6 Were qualified staff and personnel used to collect the data?
Project leader:
Name: Janet Morgan
Title: Deputy Director, Clinical, Children
Role: Project Leader
Other team members:
Names: PIP Committee
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
The data will be collected by clinical staff for CBCL score and entered into the EHR. Parent Satisfaction scores will be tabulated by a PIP Committee member. Length of treatment will be calculated as noted above.
Totals 3 Met 2 Partially Met 1 Not Met UTD
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 8 of 14
STEP 7: Assess Improvement Strategies
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 9 of 14
7.1 Were reasonable interventions undertaken to address causes/barriers identified through data analysis and QI processes undertaken?
Describe Interventions:
Week 1 intervention: The focus will be on engagement, education and goal setting. The clinician will review the diagnosis and history and offer acknowledgement of parent’s strengths and areas of concern.
Week 2: After signing the treatment plan, the focus will be on educating parent regarding parenting styles (authoritative, permissive, supportive versus disciplinary) using the Berkeley study.
Week 3: The focus will be on addressing parent’s ability to provide support/warmth to their child through at least one of the following evidence based models:
(a) Teaching PRIDE skills (modified for children above age 8);
(b) Introducing skills through the Love and Logic model;
(c) Using the Nurturing Parent model.
Week 4: The focus will be on discipline/structure and help clarify between troubled and normal behaviors using Ericson’s Developmental Stages model and/or Transactional Analysis model. These models introduce parents to nurturing versus critical parenting and help parents review their ability to nurture and structure their inner child. Parents will also receive support for structured and supportive parenting.
Week 5: The focus will be on discipline/structure and help clarify between troubled and normal behaviors using Ericson’s Developmental Stages model and/or Transactional Analysis model. These models introduce parents to nurturing versus critical parenting and help parents review their ability to nurture and structure their inner child. Parents will also receive support for structured and supportive parenting.
Week 5: The focus will be on parent’s barriers to implementing skills that have been learned in the previous weeks. Possible evidence based models may include the communication
☒ Met
☐ Partially Met
☐ Not Met
☐ Unable to Determine
Note: The intervention is the changed process of automatically assigning child clients to conjoint individual and family services. The weekly interventions reflect implementation of the process. The process of automatically assigning mild to moderate clients to parent intervention has begun.
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 10 of 14
Totals 1 Met Partially Met Not Met NA UTD
STEP 8: Review Data Analysis and Interpretation of Study Results
8.1 Was an analysis of the findings performed according to the data analysis plan?
This element is “Not Met” if there is no indication of a data analysis plan (see Step 6.5)
☐ Met
☐ Partially Met
☒ Not Met
☐ Not Applicable
☐ Unable to Determine
Because the children on average have been in therapy for less than 4 months, they should have some data analysis comparing to the baseline data collected in Sept. 2015. Data collection and analysis should occur at least quarterly. Quarterly data collection would allow the MHP to identify issues sooner and allow changes to be implemented in the process to benefit others.
8.2 Were the PIP results and findings presented accurately and clearly?
Are tables and figures labeled? ☐ Yes ☐ No
Are they labeled clearly and accurately? ☐ Yes ☐ No
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 11 of 14
8.3 Did the analysis identify: initial and repeat measurements, statistical significance, factors that influence comparability of initial and repeat measurements, and factors that threaten internal and external validity?
Indicate the time periods of measurements:___________________
Indicate the statistical analysis used:_________________________
Indicate the statistical significance level or confidence level if available/known:_______% ______Unable to determine
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
8.4 Did the analysis of the study data include an interpretation of the extent to which this PIP was successful and recommend any follow-up activities?
Limitations described:
Text
Conclusions regarding the success of the interpretation:
Text
Recommendations for follow-up:
Text
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
Totals Met Partially Met 1 Not Met 3 NA UTD
STEP 9: Assess Whether Improvement is “Real” Improvement
9.1 Was the same methodology as the baseline measurement used when measurement was repeated?
Ask: At what interval(s) was the data measurement repeated?
Were the same sources of data used?
Did they use the same method of data collection?
Were the same participants examined?
Did they utilize the same measurement tools?
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 12 of 14
9.2 Was there any documented, quantitative improvement in processes or outcomes of care?
Was there: ☐ Improvement ☐ Deterioration
Statistical significance: ☐ Yes ☐ No
Clinical significance: ☐ Yes ☐ No
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
9.3 Does the reported improvement in performance have internal validity; i.e., does the improvement in performance appear to be the result of the planned quality improvement intervention?
Degree to which the intervention was the reason for change:
☐ No relevance ☐ Small ☐ Fair ☐ High
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
9.4 Is there any statistical evidence that any observed performance improvement is true improvement?
☐ Weak ☐ Moderate ☐ Strong
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
9.5 Was sustained improvement demonstrated through repeated measurements over comparable time periods?
☐ Met
☐ Partially Met
☐ Not Met
☒ Not Applicable
☐ Unable to Determine
Totals Met Partially Met Not Met 5 NA UTD
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 13 of 14
ACTIVITY 2: VERIFYING STUDY FINDINGS (OPTIONAL)
Component/Standard Score Comments
Were the initial study findings verified (recalculated by CalEQRO) upon repeat measurement?
☐ Yes
☒ No
NA
ACTIVITY 3: OVERALL VALIDITY AND RELIABILITY OF STUDY RESULTS: SUMMARY OF AGGREGATE VALIDATION FINDINGS
Conclusions:
Met 13
Partially Met 2
Not Met 2
UTD 0
# Not applicable 11
# Applicable 17
Score 82.35%
Recommendations:
MHP is experiencing clients leaving PCIT groups early and not completing all weeks of treatment. If parents are not finishing and given that the PCIT has its own timeline, the MHP can’t meaningful compare “length of time in treatment” (esp. if required to complete all assigned sessions). Feedback included acknowledgment that the PIP focus seemed to naturally shift to evaluating the PCIT intervention. The study question needed clarification so that it was measurable an objective. The interventions listed are not truly the intervention, but rather assigning all mild to moderate clients to PCIT and all moderate to severe clients to individual as a matter of standard administrative procedure. It was recommended that the MHP then compare the CBCL scores from a baseline at start of treatment for groups of children (mild/moderate, moderate/severe) after a set period of time of either individual or PCIT intervention (i.e. 3 months, or 6 months). This would allow the MHP to see the true effectiveness of the new administrative procedure. Because the children on average have been in therapy for less than 4 months, they should have some data analysis comparing to the baseline data collected in Sept. 2015. Data collection and analysis should occur at least quarterly. Quarterly data collection would allow the MHP to identify issues sooner and allow changes to be implemented in the process to benefit others.
Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 14 of 14
Check one: ☐ High confidence in reported Plan PIP results ☐ Low confidence in reported Plan PIP results
☐ Confidence in reported Plan PIP results ☐ Reported Plan PIP results not credible
☒ Confidence in PIP results cannot be determined at this time