Audit with Feedback as a CME Audit with Feedback as a CME Tool for Radiation Oncologists:Tool for Radiation Oncologists:Evaluation of efficacy, perception and Evaluation of efficacy, perception and
cost-effectivenesscost-effectiveness
Dept. Radiation Oncology,The Cancer Institute (TCI) Singapore
Tom Shakespeare MBBS, MPH, GradDipMed(ClinEpi),
FRANZCR, FAMS
BackgroundBackground
CME efficacy differs between medical specialties
little research of CME for radiation oncologists
literature review: no study evaluating RO-specific CME interventions*
*Shakespeare 2003. Ann Acad Med Singapore
BackgroundBackground
Primary CME: lectures, conferences
Secondary CME: eg audit with feedback (AWF), reminders– Secondary CME more effective than primary
CME: meta-analysis (Davis, JAMA, 1992)
– AWF effective: RCTs/meta-analysis (O’Brien et al, 2001, Cochrane Database)
TCI AWF CME programTCI AWF CME program
Commenced targeted AWF meeting for QI / CME June 2001
no other RO-specific CME program
TCI AWF CME programTCI AWF CME program
1. Fortnightly random chart audit– 2 records per consultant– independently scored using a 10-item checklist – checklist targets RO behavior-based on
RANZCR recertification instrument
Date:
Consultant:
Patient ID:
Treatment site:
Departmental file: Registry sheet completed
Letter to referring doctor
Rx intent described
Tumour stage described
Treatment chart: Treatment site described
Laterality doublet noted
RT dose point specified
Isodose plan: Isodose plan signed
Simulation film: Patient name
Film countersigned
Items audited in CME program checklist
TCI AWF CME programTCI AWF CME program
2. Departmental CME meetingaudit results presentedcase management discussedfeedback provided by senior peers (case and
audit)ranked audit results displayed at end
Study objectivesStudy objectives
Evaluation of educational endpoints– program perception– professional practice (behaviour and
performance)– cost-effectiveness
MethodsMethods
Chart reviewall new patients seen by ROs at 2 time points
– 2 months before program implementation (T0)– months 13-14 of the program (T1)
19-item checklistmeasured RO behaviour/performance
Targeted behaviour criteriaDepartmental notes
Registry sheet completeLetter to referring doctorTreatment intent describedTumour stage given
Treatment prescriptionTreatment site describedLaterality doublet notedRT dose point specified
Simulation/planningIsodose plan signedPatient name on filmFilm countersigned
Targeted because they are part of the CME items being audited
Behaviour criteria score
• “1” if documented, or if not applicable
•“0” if not documented, or unable to evaluate
Non-targeted behaviour criteriaDepartmental notes:
Primary tumour site documentedHistology documentedDecision for/not for radiotherapy documented
Performance criteriaIndication for treatmentTreatment intentRadiation modalityDoseFractionation scheduleField arrangement
Behaviour criteria score
• “1” if documented, or if not applicable
•“0” if not documented, or unable to evaluate
Performance criteria score
•“1” if Appropriate/Adequate or not applicable
•“0” if inappropriate/inadequate or unable to evaluate
Methods (cont)Methods (cont)19-item instrument
– 10 “targeted” behaviour items (similar to CME AWF checklist)
– 3 “non-targeted” behaviour items– 6 “performance” items
Score “1” if adequate/appropriate”reproducible*
*Shakespeare et al, 2003 Australasian Radiology
Methods (cont)Methods (cont)
Program perception/satisfaction evaluatedquestionnaire
Cost-effectivenesscost of salaries and consumablescost-per-criterion point gained
ResultsResults
113 and 118 patient cases evaluated at T0 and T1
targeted behaviour improved – 8.7 to 9.2 out of 10 (p=0.0001)
no significant change in non-targeted behaviour or performance items
ResultsResultsImprovement in
– documentation of decision for RT (non-targeted): adequacy increased 84% to 92% (p=0.08)
– letter to referring doctor: 53 to 66% (p=0.04)– description of treatment intent: 54 to 77%
(p=0.0002)– laterality noted twice: 91 to 98% (p=0.04)– isodose plan signed: 94 to 100% (p=0.006)
Change in Behavior & Performance of Radiation Oncologists before and 1 year after commencing CME program
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Behaviour / Performance Items Audited
Per
cen
tag
e P
rese
nt
at A
ud
it
2001 (pre-CME)
2002 (1 year of CME)
Behaviour item improvement: Behaviour item improvement: variation between ROsvariation between ROs
Radiation Oncologist
Criterion % Scoring “1” at T0
% Scoring “1” at T1
P value
RO 1
Treatment intent described 48.4% 88.5% 0.0001
Isodose plan signed 83.9% 100.0% 0.003
RO 2
Decision for treatment documented
82.0% 97.1% 0.03
Tumour stage described 91.8% 100.0% 0.09
RO 3
Treatment intent described 52.4% 84.4% 0.01
Laterality doublet on treatment chart
81.0% 96.9% 0.05
Results: Program perceptionResults: Program perception
initial perception ambivalent– mean score 3.2 (out of 5)
increased significantly after evaluation of CME was presented– mean score 3.7 (p=0.0001)
Results: Cost-EffectivenessResults: Cost-Effectiveness
Cost (over 1 y): $US7,897 (91% salaries)cost-per-point gained: $15.67
– deemed cost-effective by participants and TCI
ConclusionsConclusions
Targeted audit with educational / ranked feedback effectively increased targeted behaviour of ROs
–consistent with RCTs and meta-analyses
ConclusionsConclusionsHoweverno definite impact on non-targeted
behaviour itemsno impact on high-scoring non-targeted
performance criteria– highlights need to target CME interventions– consistent with published research
ConclusionsConclusions
All ROs benefited improvements varied between radiation
oncologists– consistent with literature– importance of “Educational Needs Assessment”
prior to program design
ConclusionsConclusions
AWF positively perceived after evaluation of the intervention was
discussedneed to evaluate and report on the efficacy
of CME interventions
ConclusionsConclusionsThe future
– further research required– improve program (eg modify criteria)– integrate CME with QA/QI programs – MS Access database with planned analysis July
2004
TCI QI and CME DatabaseTCI QI and CME DatabaseBeta-version available at: www.theshakespeares.com/CME
TCI Quality Improvement and CME Programme: Start Page
Audit With Feedback Meeting
CME Tutorial Meeting
Simulation Review Meeting
Created by Tom Shakespeare and Rahul Mukherjee. For more information email [email protected]