Formative Evaluation of the Implementation of the Medical Education Metrics Standard for
Continuing Education
MedBiquitous Annual Conference 2011
May 11, 2011 Baltimore, Maryland
Presenters and Disclosures
• Francis Kwakwa, MAAssistant Director, Data Management , Radiological Society of North AmericaDoes not have an interest in selling anything to CME professionals
• Andrew Rabin, BSChief Technology Officer, CECity Does have an interest in selling a program or service to CME professionals
• Sean Hayes, PsyD Vice President of AXDEV GroupDoes have an interest in selling a program or service to CME professionals
• Valerie Smothers, MA Deputy Director, MedBiquitousDoes not have an interest in selling anything to CME professionals
2
Objectives of Today’s Session
• Increase knowledge about use of Medical Education Metrics Standard (MEMS) in CME
• Share experience of using MEMS as part of Program and Activity Reporting System
• Present formative findings collected regarding the experience of those who have implemented MEMS
• Offer case study findings as to how to assess implementation of new systems
3
MedBiquitous Metrics Working Group
Mission
To develop XML standards & Web services
requirements and descriptions for the exchange
of aggregate evaluation data and other key
metrics for health professions education.
4
MEMS: Medical Education Metrics
• Technology standard for core evaluation data
• Users– Educators want best
practices, ability to compare– Funders want to measure
reach and efficacy– Accreditors want to
measure success of activity and provider
MEMS Data
• Activity Description– What’s being evaluated
• Participant Activity Evaluation– What did participants think
• Participation Metrics– How many people participated
• Learner Demographics
Program and Activity Reporting System (PARS)
7
• Manual data entry
• Comma delimited upload
• XML upload
Formative Evaluation
• Provides immediate feedback to program participants
• Fosters reflection and self-regulation
• Provides feedback about the program itself to designers and sponsors
• Oriented to improving the focus of the program
• Based upon a dynamic model
Relationship Between Formative and Summative Evaluation
Across Life of an Innovation
Rel
a tiv
e E
mp
has
is
Program Life
Summative Evaluation
Formative Evaluation
Worthen 1987
Focus Group
• MEMS implementers using XML in association with PARS
• 4 organizations represented
• Measured – Success and satisfaction– Resources for implementation– Characteristics of implementers– Decision factors
10
Results
• Implementers felt satisfied, but limited (not many customers implementing)
• Limited effort and resources required; change management important
• Implementers all fluent in XML • ACCME adoption key; implementers
seeking to better support customers, show competitive advantage, make links between different types of data
Case Study
• Implemented MEMS for PARS upload
• Experience implementing MedBiquitous standards
• Integrated new data into user interface and workflow
• Provided user training
Lessons Learned
16
• Standards implementation can bring about efficiencies when integrated into workflow and properly supported
• Validation tools needed
• Web services needed to realize full benefits, one click data transfer
Contact Us
17
• Valerie will forward questions to the working group
