1

The Accreditation ProcessPerformance Measurement and the ORYX Initiative (PM)

2

Speaker Information

Frank S. ZibratAssociate Director, Accreditation SystemsIntegration and ORYX, Division of Accreditation and Certification OperationsThe Joint Commission

3

Performance Measurement and the ORYX Initiative (PM)

4

Overview

• The use of performance measures and performance measure data are essential to the credibility of any modern evaluation activity.

• The Joint Commission’s ORYX® initiative integrates the use of performance measurement data into the standards-based survey and accreditation process.

• The use of ORYX performance measure data in the survey process supplements and helps guide that process by providing a more targeted basis for the regular accreditation survey.

• The Joint Commission’s ORYX performance measurement requirements also are intended to support Joint Commission–accredited hospitals in their quality assessment and improvement efforts through the continuous monitoring of actual performance, and by helping guide and stimulate continuous improvement.

5

The Continued Role of ORYX

• Hospitals and The Joint Commission use ORYX performance measure data to continuously assess key performance areas – ORYX data reported by hospitals provide surveyors with information on

hospital performance in important care, treatment, and services areas and the hospital’s ability to effect change in clinical processes.

– ORYX core measure data are:• One of the key elements included in the Priority Focus Process (PFP) (see “The

Accreditation Process” [ACC] chapter, page ACC-31)• Incorporated into the hospital’s quarterly Strategic Surveillance System (S3) report • Displayed on The Joint Commission’s Quality Report (see “The Joint Commission

Quality Report” [QR] chapter)

– ORYX core measure data also are applied in the accreditation process (see Use of Performance Measure Data on page PM-8.)

6

The Continued Role of ORYX (Continued)• Core measure information displayed on The Joint

Commission’s Quality Check® – Provides a wide array of audiences with valuable information on

hospital performance on the core measures– Provides Joint Commission–accredited hospitals with the

opportunity to distinguish themselves among other hospitals based upon their performance on the core measures and measure sets.

• A hospital can also use ORYX data in intra-cycle, continuous performance improvement activities to proactively identify potential opportunities for improvement.

7

The Continued Role of ORYX (Continued)

• To help hospitals prepare for performance measurement in the new health care environment, The Joint Commission will continue to expand the scope of ORYX core measures:

– Focus on accountability measures• Quality measures that meet four criteria designed to identify measures that produce the

greatest positive impact on patient outcomes when hospitals demonstrate improvement. • Increased emphasis on organization performance on accountability measures and the

integration of performance expectations on accountability measures into accreditation standards;

• Hospitals will not be required to directly address ORYX accountability measure performance in intracycle activities

• Hospital review of its ORYX accountability measure performance can help identify potential opportunities for improvement related to standards.

– To learn more about accountability measures, go to the Joint Commission’s website at: – http://www.jointcommission.org/accountability_measures

8

Current Requirements for Hospitals*

• Accredited hospitals with an average daily census of greater than 10 inpatients:– Must select the currently required minimum number of core

measure sets and/or noncore measures from Joint Commission listed vendors

– Notify The Joint Commission of their chosen core measure set(s) and/or noncore measures and the associated vendor

• For initial surveys, data collection must begin on the first day of the first calendar quarter following survey.

* For current ORYX requirements for critical access hospitals, please refer to the “Performance Measurement and the ORYX Initiative” chapter in the Comprehensive Accreditation Manual for Critical Access Hospitals.

9

Current Requirements for Hospitals*• Hospitals must select one of the following:

– If the hospital serves patient populations with conditions that correspond with four or more core measure sets, the hospital must select a minimum of four of the available measure sets and submit data for all the applicable measures in the measure set via its selected vendor.

– If the hospital serves patient populations with conditions that correspond with only three core measure sets, the hospital must collect data on all of the applicable measures in the three core measure sets along with data on three noncore measures. Core and noncore measure data must be submitted via the selected vendor.

10

Current Requirements for Hospitals* (Continued)

• Hospitals must select one of the following:– If the hospital serves patient populations with conditions that

correspond with only two core measure sets, the hospital must collect data on all of the applicable measures in the two core measure sets along with six noncore measures. Core and noncore measure data must be submitted via the selected vendor.

– If the hospital can only identify one core measure set related to its patient population, it must collect data on all the applicable measures in that core measure set along with nine noncore measures. Core and noncore data must still be submitted via the selected vendor.

11

Current Requirements for Hospitals* (Continued)

• Hospitals must select one of the following:– A hospital that cannot identify any applicable core

measure sets must collect and transmit data on nine noncore measures via its selected vendor.

• To find a complete list of vendors, go to the Joint Commission’s website at:

• http://www.jointcommission.org/Core_Systems_List

12

Requirements for Small Hospitals

• A number of modifications apply to small hospitals: – A hospital with an average daily census of 10 or fewer

inpatients is exempt from the requirement to transmit data via a listed vendor to The Joint Commission.

– Must select the currently required minimum number of core measure sets and/or non-core measures

– Notify The Joint Commission of their chosen core measure set(s) and/or non-core measures

• For initial surveys, data collection must begin on the first day of the first calendar quarter following survey.

13

Requirements for Small Hospitals

• Qualifying small hospitals are required to select one of the following:– If four of the core measure sets are relevant to the small hospital’s patient

population mix, the hospital is required to collect data internally on all of the measures in the four core measure sets, generate either run charts or control charts on each measure at least quarterly for use in internal performance improvement activities, and share data and conclusions with surveyors at the time of survey.

– If only three core measure sets are relevant to the small hospital’s patient population mix, the hospital must collect data internally on all of the applicable measures in the three core measure sets along with three noncore measures, generate either run charts or control charts on each measure at least quarterly for use in internal performance improvement activities, and share data and conclusions with surveyors at the time of survey.

14

Requirements for Small Hospitals (Continued)

• Qualifying small hospitals are required to select one of the following:– If only two core measure sets are relevant to the small hospital’s patient population

mix, the hospital must collect data internally on all of the applicable measures in the two core measure sets along with six noncore measures, generate either run charts or control charts on each measure at least quarterly for use in internal performance improvement activities, and share data and conclusions with surveyors at the time of survey.

– If only one core measure set is appropriate to a small hospital’s patient population mix, the hospital must collect data internally on all applicable measures in that set, along with data on nine other noncore measures, generate either run charts or control charts on each measure at least quarterly for use in internal performance improvement activities, and share data and conclusions with surveyors at the time of survey

15

Requirements for Small Hospitals (Continued)

• Qualifying small hospitals are required to select one of the following:– If none of the core measure sets is appropriate,

the hospital must collect data internally on nine non-core measures and generate either run charts or control charts on each measure at least quarterly for use in internal performance improvement activities, and share data and conclusions with surveyors at the time of survey.

16

Requirements for Small Hospitals (Continued)

• Qualifying small hospitals continue to have the option of participating with a Joint Commission listed vendor and submitting data to The Joint Commission.

• For additional information on performance measurement and ORYX, go to the Joint Commission’s website at:– http://www.jointcommission.org/accreditation/

performance_measurementoryx.aspx

17

Requirements for Psychiatric Hospitals• The following requirements apply to Joint Commission–

accredited psychiatric hospitals that are:– Surveyed under the Comprehensive Accreditation Manual

for Hospitals, and – that are not otherwise accredited as a site under the

accreditation of a general medical/surgical hospital, i.e.:• Participate with a Joint Commission–listed vendor• Submit aggregate monthly data on a quarterly basis to The Joint

Commission on all applicable measures and related age strata that comprise The Joint Commission’s Hospital Based Inpatient Psychiatric Services (HBIPS) core measure set

18

Requirements for Psychiatric Hospitals (Continued)

• Psychiatric hospitals with an average daily census of 10 or fewer inpatients:– Are exempt from the requirement to transmit data to The

Joint Commission via a Joint Commission-listed vendor– Must meet the following requirements:

• Collect data internally on all applicable measures and related age strata that comprise The Joint Commission’s Hospital Based Inpatient Psychiatric Services (HBIPS) measure set

• Generate either run charts or control charts on each measure, at least quarterly, for use in internal quality improvement activities (no data are required to be submitted to The Joint Commission)

19

Requirements for Psychiatric Hospitals (Continued)• Psychiatric hospitals with an average daily census of 10 or

fewer inpatients:– Make data reports available for review by surveyors during on-site

surveys and produce them upon request of The Joint Commission– At the time of survey, discuss how the data were used in

identifying priorities for performance improvement activities

• Qualifying small psychiatric hospitals continue to have the option of participating with a Joint Commission-listed vendor and submitting data to The Joint Commission

20

Performance Expectations for Accountability Measures• Effective January 1, 2012:

– Hospitals must meet a specific level of performance on ORYX accountability measures transmitted quarterly to The Joint Commission.

– “Performance Improvement” (PI) Standard PI.02.01.03 establishes a composite measure rate to assess hospital performance on ORYX accountability measures.

– Compliance with the new requirement, identified as a direct impact requirement, is based on performance on a single composite measure rate derived from all reported accountability measures for which the hospital submits data to The Joint Commission.

– Accountability measures are performance measures linked directly to a positive impact on patient outcomes when hospitals perform well on them.

– The hospital’s most recent composite rate is available on its quarterly ORYX Performance Measure Report available on its secure Joint Commission Connect™ extranet site. The report also identifies the accountability measures used to calculate the composite measure rate.

21

Performance Expectations for Accountability Measures (Continued)

• A hospital’s performance is:– Assessed using a single composite measure – Calculation of the composite score is based on

• the sum of all the numerator counts from a hospital’s reported accountability measures across all measure sets

• divided by the sum of all the denominator counts from across the same accountability measures.

– The composite rate is calculated following each submission of quarterly data to The Joint Commission using the most recent four consecutive quarters of data available for all of the accountability measures that a hospital reports to The Joint Commission.

– Data on newly designated accountability measures or accountability measures newly added by the hospital will be collected for 12 months before they are included in the composite measure.

22

ORYX Performance Measure Report• ORYX Performance Measure Reports:

– Are available to each accredited hospital and critical access hospital through their secure Joint Commission Connect extranet site.

– ORYX Performance Measure Reports are updated after each submission of quarterly measure data to The Joint Commission

• Currently data are due at The Joint Commission January 31, April 30, July 31, and October 31 of each year.

• Updated reports are available approximately four weeks after each submission of quarterly data

– Hospitals receive an electronic message notifying them that their reports are available.

23

ORYX Performance Measure Report (Continued)

• ORYX Performance Measure Reports:– Accredited hospitals that have the option of meeting their

ORYX performance measurement requirements without submitting data to The Joint Commission will continue to receive a report four times per year.

• The reports are provided to:– Remind hospitals of their ORYX requirements for the internal collection and

analysis of performance measure data for the appropriate required number of measures

– Better guide the survey process by informing surveyors to explore how the hospital is using the information derived from the internal collection and analysis of its ORYX performance measure data in their ongoing performance improvement activities

24

ORYX Performance Measure Report (Continued)

• ORYX Performance Measure Reports:– Are designed to better support and help guide accredited

hospitals in their performance assessment and improvement activities.

– Help surveyors better assess:• Hospital use of performance measure data in performance

improvement activities and, where applicable, • compliance with the performance expectations for accountability

measures as established under Performance Improvement Standard PI.02.01.03. (See Figures 1–5 for examples of selected sections of the ORYX Performance Measure Report.)

25

ORYX Performance Measure Report (Continued)

• The ORYX Performance Measure Report:– Provides a user-friendly format with

• summary dashboards• comprehensive measure details• automated links to specific sections, and • selective printing capabilities

– Highlights compliance with Standard PI.02.01.03– Highlights desirable and undesirable data trends

26

ORYX Performance Measure Report (Continued)

• The ORYX Performance Measure Report:– Key features include:

• A cover page that displays the hospital’s accountability measure composite rate based upon the most recent four quarters of accountability measure data displayed in the report. (See Figure 1)

• A “dashboard” of color-coded symbols at both the measure set and individual measure level

– Provides a quick and easy graphical summary of a hospital’s performance on its measures

– Each measure set and individual measure on the dashboard provides a hyperlink that the user can click on to access more detailed information.

27

ORYX Performance Measure Report (Continued)

• The ORYX Performance Measure Report:– Key features include:

• At the measure set level, the dashboard displays the – Total number of measures within the set used by the hospital– Number of accountability measures within the measure set– Highlights the set(s) where there may be desirable or

undesirable data trends, and/or statistical process control issues. (See Figure 2)

• An easy-to-use legend that defines each of the color-coded symbols (See Figure 3)

28

ORYX Performance Measure Report (Continued)

• The ORYX Performance Measure Report:– Key features include:

• At the individual measure level, the dashboard identifies: – Measures included in the calculation of the accountability composite rate– Desirable and undesirable data trends, and/or statistical process control

issues (See Figure 4)

• Control charts, target charts (core measures), and/or comparison charts (noncore measures) (See Figure 5)

• Individual measures that require a hospital’s attention are highlighted when potential standards compliance issues and/or undesirable trends are identified

29

Analyzing ORYX Data

• The Joint Commission uses a combination of control charts, target charts, and comparison charts to evaluate ORYX data. – Control chart analysis is based on a hospital’s own historical

(longitudinal) data and is used to assess internal process stability. – Target chart and comparison chart analysis is used to evaluate a

hospital’s relative performance level. – The use of target chart and comparison chart analysis in addition to

control chart analysis is a key feature of the Joint Commission’s analytic methods in the ORYX initiative.

• These types of analyses evaluate hospital performance from two distinct perspectives and thus can provide a more comprehensive framework to assess a hospital’s overall performance level.

30

Analyzing ORYX Data (Continued)

• Control chart, target chart, and comparison chart analyses may portray different interpretations of performance, e.g.:

– A control chart may show a desirable pattern (one that is statistically in control), but the target and/or comparison chart may illustrate undesirable outliers (e.g., a high rate of infections relative to others in the comparison group or the target range).

– Perhaps the hospital’s performance has been consistently less desirable than that of other hospitals using the same measure or below the target range. In such a case, the hospital needs to think about changing its process for the measure concerned in order to improve its performance.

– On the other hand, a hospital without outliers in the target and/or comparison chart analysis may have special cause variation (i.e., a statistically out-of-control pattern) detected in the control chart. In such a case, the hospital needs to investigate the special cause variation in its process before making any conclusions about performance level.

• In general, a hospital should perform control chart analysis before target and/or comparison chart analysis to ensure that a given process is stable before attempting to evaluate relative performance level.

31

Analyzing ORYX Data (Continued)

• For more information on ORYX data analysis, including use of control charts, target charts, and comparison charts, log on to the secure Joint Commission Connect extranet site and, under “Performance Measurement (ORYX),” select “Documentation and Related Links.”

32

Performance Measurement

33

Use of Performance Measure Data

• As part of the Joint Commission’s accreditation process, during the on-site survey, Joint Commission surveyors assess the following:– The hospital’s integration and use of ORYX data into

internal performance improvement activities– The hospital’s data collection processes (such as data

accuracy, reliability, and security)– The hospital’s data analysis methodologies and related

training– The dissemination of findings

34

Integrating Accountability Measure Data into the On-site Survey Process

• If at the time of survey, based upon the most recent four consecutive quarters of accountability measure data available, the hospital’s performance on the composite measure falls below the 85% threshold required under Standard PI.02.01.03, the hospital will automatically receive a Requirement for Improvement (RFI) on its Summary of Survey Findings Report. – The hospital can only clear an RFI through the submission of

quarterly data to The Joint Commission via the hospital’s selected ORYX vendor.

– The hospital will not have an opportunity to submit more recent data or clarify the finding.

35

Integrating Accountability Measure Data into the On-site Survey Process (Continued)

• An RFI for PI.02.01.03 will be resolved using a modified Evidence of Standards Compliance (ESC) process:

– A hospital that does not meet the 85% threshold established by the standard will be required to:

• Submit an ESC within 45 days in the form of a Plan of Correction that outlines the actions to be taken to meet the 85% threshold.

– To clear the RFI, the hospital must demonstrate an accountability measure composite rate that meets or exceeds the 85% threshold for any two consecutive quarters of regularly submitted data.

– Failure to resolve the RFI after 18 months following the on-site survey may result in a recommendation for Contingent Accreditation.

• For general information on ORYX requirements, contact the ORYX Information Line at 630-792-5085, or e-mail hcooryx@jointcommission.org.

36

• Figure 1. Accountability Measure Composite Rate– Page PM-9

• Figure 2. Measure Set Summary– Page PM-10

• Figure 3. Measure Set Legend– Page PM-10

• Figure 4. Measure Level Summary– Page PM-11

• Figure 5. Measure Level Detail– Page PM-12