Through collaborative use of improvement science methods, reduce preterm births &
improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Orchestrated Testing Aggregate Data
Michele Walsh, MD, MSE
Percent of Infants with Greater than 10% Weight Loss from Birth Weight in First Seven Days De-Identified Formula Groups
Number of Finnegan Scores Greater than 12 in 24 Hours Prior to Starting Pharmacologic Treatment De-Identified Formula Groups
Highest Finnegan Score in 24 Hours Prior to Starting Treatment De-Identified Formula Groups
Average Length of Stay for All Opioid Exposed Infants De-Identified Formula Groups
Average Length of Opiate Treatment De-Identified Formula Groups
Average Length of Stay for Pharmacologically Treated Babies De-Identified Formula Groups
Through collaborative use of improvement science methods, reduce preterm births &
improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
NAS-Orchestrated TestingGroup 1
Theresa Ruby, MSN, RNC, IBCLCSouthern Ohio Medical Center
Group 1: 22 kcal /low lactose formulaAkron Children’s St Elizabeth/ Mahoning Valley/St Joseph
Cincinnati Children’s Hospital Medical Center
Kettering Soin Medical CenterNCH Dublin Methodist Southern Ohio Medical CenterNCH Mt Carmel St Ann’s Southview Medical CenterNCH Riverside St Rita’s Medical CenterProMedica Bay Park Trumbull Memorial Rainbow Babies & Children’s UC Cincinnati
Akron Children’s St. Elizabeth
Mahoning ValleySt Joseph’s
Kettering
Rainbow Babies & Children’s
NCH Dublin Methodist
NCH Mt. Carmel St. Ann’s
NCHRiverside
ProMedica Bay Park
Cincinnati Children’s Soin Medical Center
Southern OhioMedical Center
Southview Medical Center
St. Rita’s Medical Center
Trumbull Memorial
UC Cincinnati
Through collaborative use of improvement science methods, reduce preterm births &
improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
NAS-Orchestrated TestingGroup 2
Gail Bagwell, DNP, APRN Nationwide Children’s Hospital
Group 2: 22 kcal /not low lactose formulaBethesda North Licking Memorial HospitalDayton Children’s Mercy Children’s HospitalGood Samaritan Tri-Health Nationwide Children’s Hospital
NationwideChildren’s
Bethesda North
Good Samaritan Tri-Health Hospital
Mercy Children’s Hospital
Dayton Children’s
Licking Memorial
Through collaborative use of improvement science methods, reduce preterm births &
improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
NAS-Orchestrated TestingGroup 3
Mischel Balazs, NNP, CNP-BC Toledo Children’s Hospital
Group 3: 19 kcal /low lactose formulaAkron Children’s Hospital Medical Center Mercy AndersonAkron Children’s Summa Mercy Medical CantonAkron Children’s General Miami Valley HospitalAtrium Hospital Mt Carmel EastAultman Hospital Mt Carmel WestCleveland Clinic NCH Doctors HospitalFairview Hospital NCH Grant MedicalFort Hamilton Hospital OSU Wexner Well Baby/OSU Wexner NICUGenesis Healthcare ProMedica Toledo Normal NewbornHillcrest Medical Center ProMedica Toledo Children’s Hospital
Toledo Children’sToledo Normal Newborn
OSU Well BabyOSU NICU
NCH Grant
NCH Doctors
Mt Carmel WestMt Carmel East
Miami Valley
Mercy Medical CantonHillcrest
Genesis
Fairview Cleveland Clinic
Aultman
Atrium
Fort Hamilton
Mercy Anderson
Akron Children’sACH SummaACH General
Through collaborative use of improvement science methods, reduce preterm births &
improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
NAS-Orchestrated TestingGroup 4
Tonia Deacon
Group 4: 19 kcal /not low lactose formulaAdena Regional Medical Center Mercy Health FairfieldElyria UHCMC Mercy Health WestGood Samaritan Dayton Mercy Regional LorainLima Memorial Hospital Springfield Medical CenterMarion General Hospital The Christ HospitalMetroHealth Upper Valley Medical Center
MetroHealthMercy Lorain
Good SamaritanDayton
Elyria
Upper Valley Medical Center
Lima Memorial
Marion
AdenaMercy Fairfield
Mercy West
Springfield
ChristHospital
Through collaborative use of improvement science methods, reduce preterm births &
improve perinatal and preterm newborn outcomes in Ohio as quickly as possible.
Data Analysis of Orchestrated Testing
Heather Kaplan, MD
OPQC Phase IIOrchestrated Testing
OT Analysis Team
Heather Kaplan, MD, MSCE
Maurizio Macaluso, MD, DPH Pierce Kuhnell, MS
55
Quality Improvement using the Model for Improvement
Hunches Theories
Ideas
Changes That Result in
Improvement
A PS D
A PS D
Very Small Scale Test
Follow-up Tests
Wide-Scale Tests of Change
Implementation of Change
What are we trying toaccomplish?
How will we know that achange is an improvement?
What change can we make thatwill result in improvement?
Model for Improvement
Sequential building of knowledge under a wide
range of conditions
Orchestrated Testing• OT involves planned testing across multiple sites
(within or across institutions)• Can use factorial design to…
– Be more systematic about simultaneous testing of different change ideas
– Look at the independent and combined effects of different changes
• Standardization of practices and reliable implementation is necessary
• Can result in faster and more efficient learning
OPQC OT Phase II
• Wide scale test of change examining the role of formula in non-pharmacologic care across 54 NICU/SCN sites
• Two change ideas (factors):– Type of formula– Calorie content of formula
• Two “levels” of each factor– Standard Lactose vs. Low-Lactose – Standard Calorie vs. Higher Calorie
Factorial Design
OPQC Factorial Design (22)
Group Calorie Content Formula Type
A 19 kcal/oz Low-Lactose
B 22 kcal/oz Standard
C 19 kcal/oz Standard
D 22 kcal/oz Low-Lactose
The Model for Improvement
Measures: Outcomes
• Primary Outcome– LOS (pharmacologically treated infants)
• Secondary Outcomes– LOS (all infants)– LOTx– Percent pharmacologically treated– Percent infants requiring dose escalation– Weight Loss >10%
These are similar outcomes to the outcomes examined in the RCT of standard vs. 24 kcal formula posted on clinicaltrials.gov (weight loss, LOS, LOTx, Finnegan scores)
Measures: Background Variables• Pharmacologic treatment approach*
– OCHA Morphine– Cincinnati Methadone– Other Morphine– Other Methadone
• Rates of breastmilk feeding• Percent outborn infants+
• Formula compliance• Time of formula initiation (early vs. time of treatment)• NAS Severity (Max Finnegan Score, Scores>12)• Change in typical formula treatment from Phase I to II
*Centers with extreme deviation from these protocols will not be included in analysis+Infants transferring between sites in different groups will be excluded
Analysis
• Main analyses are graphical– Run and control charts
• Explore differences on background variables
• Examine impact according to groups
– Response plots
Run and Control ChartsExplore Background Variables
1 2 3 4 5 6
Background Variable
Groups: y, c, z, x
Group X Group Y Group Z Group C
Run and Control ChartsExamine Impact
Factor 1 High High Low Low
Factor 2 Low High Low High
Low HighFactor 1
Low HighFactor 1
Low HighFactor 1
F2 High
F2 Low
F2 High
F2 High
F2 Low
Low HighFactor 1
F2 High
F2 Low
F2 Low
Response Plots: InteractionsNo Interaction Moderate Interaction
Moderate Interaction Strong Interaction
Replication
• Ultimate confirmation of findings from OT comes from replication across sites over time
• If positive results are seen for a change, process modification at additional sites with subsequent improvement further validates findings
• This is a critical step because conditions in the future (when the findings will be used) are likely different than the conditions during OT
How this works in “real life”• SLUG Bug QI Project with OT
– 17 participating centers– Examine 4 CLABSI prevention strategies
• Tubing change technique (sterile vs. clean)• Hub care monitoring (yes vs. no)• CVC access limitation (limitation vs. no limitation)• CVC removal monitoring (tracking policy vs. no policy)
– Used factorial design (24-1 8 groups)– Focused on reliability…16/17 centers achieved >75%
reliability on these processes
Piazza AJ, et al. Pediatrics. 2016
4 Factors
8 GroupsVariation in
Baseline Rates
Piazza AJ, et al. Pediatrics. 2016
Run and Control Charts
3/4 Sterile Tubing Change **Greatest decrease with addition of HC Monitoring+
** **** **+ + + +
Piazza AJ, et al. Pediatrics. 2016
Response Plots: Impact
Sterile Tubing Change Decreased CLABSI Rates by an average of 0.51
Piazza AJ, et al. Pediatrics. 2016
Response Plots: Interaction
The addition of hub care compliance monitoring with sterile tubing change created the greatest
impact…showing evidence of an interaction
Piazza AJ, et al. Pediatrics. 2016
Replication
Courtesy of Lloyd Provost, data unpublished
How this might look for OPQC
Group Calorie Content
Formula Type
#Hospitals
Baseline LOS
Phase II LOS
A 22 kcal LLF 11 17.36 ?
B 22 kcal Standard 5 20.51 ?
C 19 kcal LLF 17 20.00 ?
D 19 kcal Standard 12 21.71 ?
Hypothetical Control Chart
Type LLF Std LLF Std
Calorie 22 kcal 22 kcal 19 kcal 19 kcal
Hypothetical Response Plots
Std LLF19 kcal 22 kcal
Std LLF
19 kcal
22 kcal
LLF Std
19 kcal 18 21 19.5
22 kcal 16 16 16
17 18.5
Formula TypeC
alor
ies
LOS:
Keep in mind• We are examining the effect of 2 factors in a complex
system…there will always be the possibility that other variables (known background variables or unknown variables) account for some of what we see
• BUT, the complexity is also a strength—wide range of clinical conditions improves generalizability
• The strength of this OT approach is that is allows us to examine factors in real settings in a range of conditions (54 sites!)
• This is an new method of learning for OPQC and we have to figure out some things as we go (e.g., best approach to adjust for differences in baseline LOS)
• After June, we need to commit to implement the findings to achieve the full benefit of the OT approach
References for other geeks
Pallotto EK, et al. “Orchestrated Testing: An Innovative Approach to a Multicenter Improvement Collaborative.” American Journal of Medical Quality. 2015 Oct 19, [Epubahead of print]
Piazza AJ, et al. “SLUG Bug: Quality Improvement with Orchestrated Testing Leads to NICU CLABSI Reduction.” Pediatrics. 2016 Jan;13(1):1-12.
Moen RD, Nolan TW, Provost LP. Quality Improvement Through Planned Experimentation. 2nd ed. New York: McGraw-Hill, 1999
Polaris C Polaris F Hallway 1RB&C MetroHealth Dayton Children’sAtrium Mercy Anderson GenesisMt Carmel East Springfield Mercy FairfieldElyria The Christ Hospital LickingKettering NCH Riverside NCH Mt Carmel St Ann’sUC Health St Rita’s Medical Center Marion GeneralTrumbullHallway 2 Hallway 3Nationwide Children’s Bethesda NorthAultman Hospital Fairview HospitalToledo Children’s OSU NICU/OSU Well BabyMercy Lorain Mercy Children’sSoin Medical Center SouthviewMercy West Akron Children’s/Summa
Gemini B Gemini CMt Carmel West Good Samaritan Tri-HealthHillcrest Mercy CantonGood Samaritan Dayton NCH GrantCCHMC Akron St E/Mahoning ValleyDublin Methodist Southern Ohio Medical CenterMiami Valley Hospital Fort Hamilton
**LUNCH is 12:30 – 1:15PM after Storyboard Rounds**
Steal Shamelessly;
Share Seamlessly
Storyboard Rounds