Self-Assessment within a Learner-Prescribed CME/CE Curriculum

Destry Sulkes M.D., Vice President, Educational ServicesMedscape LLC

Acknowledgements

From We Move:• Judy Blazer

From Outcomes, Inc.:• Mazi Abdolrasulnia, PhD

• Linda Casebeer, PhD

• Hamid Doroodchi, MD

• Nancy Roepke, MBA

From Medscape LLC:• Martin Irvine, PhD

We wish to acknowledge the following for their contributions and hard work:

Disclosure

This activity was made possible by an unrestricted educational grant from Boehringer-Ingelheim Pharmaceuticals, Inc.

Sometimes things aren’t quite what they seem…

Starting Point

CME has been shown to improve knowledge, skills (cognitive) and attitude1

Self-assessments have been shown to be inaccurate when not externally informed2

Feedback from externally informed self-assessments, provided to both clinicians and Faculty, can be used to increase effectiveness of CME3

1. Evid Rep Technol Assess (Full Rep). 2007 Jan;(149):1-69. Effectiveness of continuing medical education. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum R, Bass EB.

2. JAMA. 2006 Sep 6;296(9):1094-102. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L.

3. Med Teach. 2008 Mar;30(2):124-45. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, McKinstry B. NHS Education for Scotland.

Current Model of Online Medical Education

Medical education programs for healthcare professionals are designed using broad needs assessments across target audiences

Individual activities within courses are not typically developed for specific segments of learners

The courses clinicians select to maintain their professional licensure may or may not be appropriate to address gaps in specific individuals’ clinical knowledge, skills and attitude (K/S/A)

Hypothesis for an Externally Informed Self Assessment

Learners who participate in an externally informed self-assessment and engage in the corresponding prescribed educational activities are more likely to:

• Engage in the educational activities

• Enhance their K/S/A relative to those who find the activities in an undirected fashion

Methodology

To address K/S/A gaps identified through a needs assessment of Medscape members, an online CME/CE curriculum of activities was developed in collaboration with We Move and Outcomes Inc

One part of the curriculum is a CME/CE-certified self-assessment activity that measures K/S/A related to recognition, diagnosis, and non-pharmacologic and pharmacologic management of RLS

• The online self-assessment provides automated activity recommendations based on the learners’ responses to K/S/A questions

Post-assessments follow each activity to assess the relative efficacy of this new educational approach vs the standard, undirected approach, to improving clinical management of RLS

Course Formats/Titles/Focus

Formats Titles Focus

Virtual LectureRestless Legs Syndrome: Understanding the Patient Experience to Improve Outcomes Attitude (A1)*

Expert Interview The Genetics and Pathogenesis of restless Legs Syndrome Knowledge (K1)*

Mini Review Article 1 Restless Legs Syndrome in Special Populations Skills (S1)*

Mini Review Article 2 Treatment of RLS in Patients with Psychiatric Comorbidities Knowledge

Expert Interview What’s New in the RLS Literature? Skills

Mini Clinical Case 1Key Considerations in the Diagnosis and Management of RLS in the Family Practice Setting

Skills

Mini Clinical Case 2Choosing the Appropriate Treatment Regimen for a Patient with RLS and Comorbid Conditions

Skills

Test and Teach Case 1 Is It Restless Legs Syndrome? A Case Study in Diagnosis Skills

Test and Teach Case 2The Importance of Accurately Diagnosing RLS in the Primary Care Setting Knowledge

Review ArticleMaximizing Outcomes in Patients with RLS: Focus on Relevance of the Disorder, Diagnosis, and Management

Attitude, Knowledge, & Skills

* prescribed courses

Curriculum Design

Technical Platform Supporting Individualized Feedback Feedback

What:

Each learner receives an email listing the recommended activities based on the question responses provided in the self-assessment.

How:

• Each night, a script that queries the database for any new self-assessment participants.

• The script evaluates each participant’s answer choices and maps the choices to the recommended CME activities.

• The data is fed to an email application where it is formatted and sent to each participant with individualized activity recommendations, plus a link to the curriculum page.

Sample immediate feedback / email message

Medscape CME Self-Assessment

Dear Dr. Dana Ricci,

Based on your answers to the Restless Legs Syndrome Self-Assessment CME activity, Medscape would like to recommend the following readings: CME/CE The Genetics and Pathogenesis of Restless Legs Syndrome: Implications for the ClinicianIn this expert interview, Dr. David Rye provides an overview of what is currently known about the genetics and pathogenesis of RLS and how to apply that knowledge to clinical practice.

CME/CE Restless Legs Syndrome in Special PopulationsDr John Winkelman provides a comprehensive overview of the similarities and differences in the diagnosis and management of primary RLS and RLS that is secondary to other conditions.

Process Schematic

Prescribed Learning:Learner sent

recommendation from Externally Informed Self Assessment

Undirected Learning:Learner finds

curriculum and/or individual activities by

standard search criteria

Attitude Course 1

Knowledge Course 1

Skill Course 1

Post Test A1: Includes mapped

Attitude questions from

Self Assessment

Automated response directing learner to

specific K, S, or A course

Post Test S1: Includes mapped Skills questions

from Self Assessment

Post Test K1: Includes mapped

Knowledge questions from

Self Assessment

Datasets and Minimums Required for Analysis

Non-Medscape Membership

Baseline Survey:N=400

(Identical to E-I self-

assessment)

Externally Informed Self-

Assessment on Medscape:

N=400(Identical to

baseline survey)

Prescribed Post Assessment A1

N=~100

Undirected Post Assessment A1

N=~100

Undirected Post Assessment: A

N= ~1400 over 7 activities

Prescribed Post Assessment K1

N=~100

Prescribed Post Assessment S1

N=~100

Undirected Post Assessment S1

N=~100

Undirected Post Assessment K1

N=~100

Undirected Post Assessment: K

N= ~1400 over 7 activities

Undirected Post Assessment: S

N= ~1400 over 7 activities

Outcomes 1 – Measures the courses prescribed vs. the courses actually completed in a three month time period, click-thru rate of A1/K1/S1 prescriptions that resulted in accessing the activities

Outcomes 2 – n=~100; Measures the Prescribed Post Assessment to the E. I. Self Assessment by Knowledge, Skills and Attitude

Outcomes 3 – n=~100; Measures the Prescribed Post Assessment to the Baseline Survey by Knowledge, Skills and AttitudeOutcomes 4 – n=~100; Measures the Prescribed Post Assessment to the Undirected Post Assessment by Knowledge, Skills and AttitudeOutcomes 5 – n=~100; Measures the Undirected Post Assessment (A1, K1, S1) to the Baseline survey by Knowledge, Skills and Attitude

Outcomes 6 – n=200; Measures the Undirected Post Assessment (4-10) to the Baseline Survey by Knowledge, Skills, Attitude

Learnings to date…

K/S/A model helpful and could benefit from further refinement incorporating established competencies

Email feedback is somewhat effective, but Medscape learners access activities primarily from their specialty homepage

• 65%: specialty homepage

• 25%: outbound email with editorial focus (note: different than the prescriptive emails in this initiative)

• 10%: external search

Recognition that current CME/CE activity “tagging” for site-location and rotation purposes can be enhanced to include competencies, thus allowing all activities (>3000 in 2007) to be targeted as needed in the future, perhaps on learners’ specialty homepages, and perhaps without explicitly communicating the targeting