Shifting Paradigms: Competency-based Medical Education

and the Quality of Care Problem

Reynolds Meeting2012

Disclosures

Employed by the American Board of Internal Medicine

I receive royalties from Mosby-Elsevier for a textbook on assessment

I am a member of the board of NBME and Medbiquitous

I serve on committees at the AAMC, ABMS, ACGME and NBME

The Quality of Care Problem

Teaching Vs. Non-Teaching Hospital Quality

Performance Ind. COTH Teaching

Non-COTH Teaching

Non-Teaching

30-day Mortality

• AMI 15.1% 15.9% 16.3%

• Pneumonia 10.8% 11.1% 11.7%

30-day Readmission

• AMI 20.3% 19.7% 19.6%

HCAHPS

• Nurse communicated well

70.5% 70.9% 74.9%

• MD communicated well

76.2% 77.0% 81%

• Help when wanted 55.0% 57.0% 64.1%

Shahian DM, Nordberg P, Meyer GS, et al. Contemporary performance of U.S. teaching and nonteaching hospitals. Acad Med. 2012; 87: online.

Care of the Vulnerable Elderly Study Performance on Geriatric Process of Care

ResidentClinics

Mean %

PracticingPhysicians

Mean %

UnivariateF

Structurecoefficients

Documentation of:

Gait evaluation 28.4% 74.2%

77.53** .90

Balance evaluation 21.6% 66.4%

65.51** .82

Medical surrogate 28.0% 54.4%

24.00** .65

End-of-life preferences 29.5% 49.3%

12.85** .55

Vision testing done 40.0% 64.7%

19.09** .55

Hearing assessment 23.3% 40.3%

8.06* .41

Screens for:

Falls risk 18.6% 60.8%

49.60** .67

Cognitive impairment 18.3% 52.0%

29.02** .60

Depression 33.7% 62.6%

24.09** .57

Lynn LA, et al. Acad Med. 2009.

Hospital Comparisons on Quality and Resource Use (Higher scores represent better performance)

Non-teaching (N= 997) Teaching (N=186)

Quality Composite Score

Res

ou

rce

use

Co

mp

osi

te S

core Exemplary

Teaching Hospitals

Source: L. Binder, CEO of Leapfrog Group, email communication, March 2010

“Every system is perfectly designed to achieve the results it gets.”

Paul Batalden

Medical Education: Restraining Forces on Change

The Current “Miracle” of Medical Education

Dwell Time

Medical Education Architecture1

1Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions. Med Educ. 2011; 45(1):69-80.

Thomas Kuhn and “Normal Science”

“Normal science, the activity in which most scientists inevitably spend almost all of their time, is predicated on the assumption that the scientific community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”

Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.

Could the Same be True of UME and GME?

“Normal education, the activity in which most educators inevitably spend almost all of their time, is predicated on the assumption that the educational community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost”

Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.

The “Big Assumption as Truth”

We operate on many assumptions that over time become “truth” without our testing or questioning the veracity of those assumptions– Test assumptions as assumptions

Immunity to change– Preservation of status quo through fear

– More comfortable to stay with familiar even when status quo isn’t effective

Kegan and Lahey. The Way We Talk Can Change the Way We Work; Immunity to Change.

Competency-based Medical Education:

A Way Forward?

Effective Systems: Where Education Must Occur

Nelson EC, et al. Quality by Design. 2007

Early Principles: CBME

World Health Organization (1978):– “The intended output of a competency-

based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.”

McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.

Traditional versus CBME: Start with System Needs

17

Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010

Competency-Based Medical Education

…is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies1

1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical education: theory to practice. Med Teach. 2010; 32: 638–645

Outcomes-based Education: General Principles

Patient outcomes ≈ Educational outcomes

Experience ≠ Expertise– Exposure and dwell time are not sufficient

proxies for competence• You can do something a 100 times wrong and

develop experience, but it’s still wrong!

Must engage in effective experiences• Critical role for work-based assessments

Need for System Approach:Assessment Perspective

Structured PortfolioITE (formative only)Monthly EvaluationsMiniCEXMedical record audit/QI projectClinical question logMultisource feedbackTrainee contributions (personal portfolio)

o Research project

TraineeReview portfolio Reflect on contentsContribute to portfolio

Program LeadersReview portfolio periodically and systematicallyDevelop early warning systemEncourage reflection and self-assessment

Clinical Competency CommitteePeriodic review – professional growth opportunities for allEarly warning systems

Program Summative Assessment Process

Licensing and Certification USMLEAmerican Boards of Medical Specialties

Assessment During Training: Components

Advisor

Structured Portfolio

Medical record audit

andQI project

MSF: Directed per protocolTwice/year

Practice-based learning and improvement

Systems-based prac

Mini-CEX:10/year

Interpersonal skills and Communication

ITE:1/year

Patient care

Faculty Evaluations

EBM/Question Log

Medical knowledge

Professionalism

Multi-faceted Evaluation

■ Trainee-directed ■ Direct observation

Time

AssessmentActivities

TrainingActivities

SupportingActivities

v v v v v v

= learning task

= learning artifact

= single assessment data-point

= single certification data point for mastery tasks

= learner reflection and planning= social interaction around reflection (supervision)

= learning task being an assessment task also

Model For Programmatic Assessment(With permission from CPM van der Vleuten)

Committee

Structured PortfolioITE (formative only)Monthly EvaluationsMiniCEXMedical record audit/QI projectClinical question logMultisource feedbackTrainee contributions (personal portfolio)

o Research project

TraineeReview portfolio Reflect on contentsContribute to portfolio

Program LeadersReview portfolio periodically and systematicallyDevelop early warning systemEncourage reflection and self-assessment

Clinical Competency CommitteePeriodic review – professional growth opportunities for allEarly warning systems

Program Summative Assessment Process

Licensing and Certification USMLEAmerican Boards of Medical Specialties

Assessment During Training: Components

Advisor

“Wisdom of the Crowd”

• Williams, Teach. Learn. Med. (2005)– No evidence that individuals in groups

dominate discussions.• No evidence of ganging up/piling on

• Thomas (2011) – Group assessment improved inter-rater reliability and reduced range restriction in multiple domains in an internal medicine residency

Narratives and Judgments

• Pangaro (1999) – matching students to a “synthetic” descriptive framework (RIME) reliable and valid across multiple clerkships

• Regehr (2007) – Matching students to a standardized set of holistic, realistic vignettes improved discrimination of student performance

• Regehr (2012) – Faculty created narrative “profiles” (16 in all) found to produce consistent rankings of excellent, competent and problematic performance.

Strategy toestablish trustworthiness Criteria

Potential Assessment Strategy (sample)

Credibility Prolonged engagement Training of examiners

Triangulation Tailored volume of expert judgment based on certainty of information

Peer examination Benchmarking examiners

Member checking Incorporate learner view

Structural coherence Scrutiny of committee inconsistencies

Transferability Time sampling Judgment based on broad sample of data points

Thick description Justify decisions

Dependability Stepwise replication Use multiple assessors who have credibility

Confirmability Audit Give learners the possibility to appeal to the assessment decision*With permission CPM van der Vleuten

The Road Forward: Kelly Caverzagie