Teaching Excellency in Competency-Based Education David L ...

David Lee Gordon, M.D., FAAN, FAHA

OUHSC Neurology

Teaching Excellence in Competency-Based Education

(for Medical Students)

1

OU Neurology

Teaching Excellency in Competency-Based Education

David L. Gordon, M.D., FAAN, FAHA

Professor & Chair, Department of Neurology

University of Oklahoma Health Sciences Center

Please turn your cell phones and pagers to silent or off.

Thank you!

OU Neurology

TEACHING EXCELLENCE IN

COMPETENCY-BASED

EDUCATION for Medical Students


Professor, Chair, & Clerkship Director

Department of Neurology

The University of Oklahoma Health Sciences Center


OUHSC Neurology



2

OU Neurology

DISCLOSURES

FINANCIAL DISCLOSURE

DLG has nothing to disclose

UNLABELED/UNAPPROVED USES

DISCLOSURE

DLG has nothing to disclose

OU Neurology

LEARNING OBJECTIVES

Describe a competency-based, objectives-

driven clinical curriculum and how it differs

from a traditional clinical curriculum

List and describe the 10 essential features

(ABCs) of competency-based education

Name three key factors in creating an

optimal clinical-learning environment

within a competency-based curriculum


OUHSC Neurology



3

OU Neurology

TEACHING IS AN ART,

EDUCATION IS A SYSTEM

Definitions per Merriam-Webster Dictionary

Teaching = the act, practice, or profession of a teacher (#1)

Education = the field of study that deals mainly with methods

of teaching and learning in schools (#2)

Essential Qualities per DLG

Teaching excellence

Ability to convey enthusiasm & make material accessible

Education excellence

Ability to develop and implement consistently effective

systems of teaching and learning

OU Neurology

WHAT IS THE CORRECT ANSWER?

Experts are the best instructors.

A.True

B.False

“Evidence from cognitive science, organizational

behavior, and educational psychology suggests

that experts are not always the best teachers.”

Huston T. Teaching What You Don’t Know (2009)


OUHSC Neurology



4

OU Neurology


Superior performance correlates best with: A. Deliberate practice

B. Domain-related knowledge

C. Extensive experience

D. General education

“Superior performance does not automatically develop from

extensive experience, general education, and domain-related

knowledge.”

“Observed performance does not necessarily correlate with

greater professional experience.”

“Expert performance can, however, be traced to active

engagement in deliberate practice.”

Ericsson KA. Acad Emerg Med 2008;15:988-994

OU Neurology


Professional expertise correlates best with: A. Deliberate practice

B. Perceived mastery of knowledge and skill

C. Reputation

“Traditionally, professional expertise has been judged by length of

experience, reputation, & perceived mastery of knowledge & skill.”

“Unfortunately, recent research demonstrates only a weak relationship between these indicators of expertise & actual, observed performance.”

“Expert performance can, however, be traced to active engagement in deliberate practice.”

Ericsson KA. Acad Emerg Med 2008;15:988-994


OUHSC Neurology



5

OU Neurology


You should teach to the test.

A. True

B. False

“When educational evaluation data are seen and used

as a tool, not a weapon, the outlook becomes

improvement and mastery rather than enforcement.”

McGaghie WC. Guidebook for Clerkship Directors (2005)

OU Neurology


External standardized tests such as the USMLE are valid measures of clinical performance.

A. True

B. False

“USMLE Step 1 & 2 scores do not correlate with reliable measures of clinical skill acquisition by medical students, residents, & fellows.”

“The validity argument about using USMLE Step 1 & 2 scores for postgraduate residency selection decisions is not structured, coherent, or evidence based.”

“Continued use of USMLE Step 1 & 2 scores for postgraduate medical residency selection is discouraged.”

McGaghie WC et al. Acad Med 2011;86:48-52


OUHSC Neurology



6

OU Neurology

TRADITIONAL CURRICULUM

IS TEACHER & PATIENT CENTERED

Ward teaching Difficult to standardize (variable patients & faculty)

Inefficient time (limited teaching, feedback, deliberate practice)

Not highest priority (secondary to patient care & safety)

Increasingly inadequate for training due to time constraints induced by changes in healthcare delivery & duty-hour limitations

Didactic teaching Lecture format (passive with limited long-term retention)

Based on faculty expertise (not student need)

Assessments Ward assessment—high validity, but low reliability, inconsistent

implementation, & often inconsistent with declared curriculum

Independent MCQ exam (e.g., “shelf”)—not linked to objectives or student experiences & does not assess skills, attitudes, behavior

Local expertise drives learning

& assessments do not reflect or drive performance

OU Neurology

COMPETENCY-BASED CURRICULUM

IS LEARNER CENTERED

Emphasizes learner needs, not teacher needs or knowledge

Objectives are not superfluous—they drive the process (“Begin with the end in mind” – Steven Covey)

Standardized learning and assessment for all learners, regardless of ward experiences

Assesses what is learned, not what is taught

Requires major changes in: Traditionally-held tenets of education (buy-in of key faculty)

Learner and institution schedules (buy-in of chair & deans)

Preferred by vast majority of learners

Most like structure & direction—everyone learns

Consensus guidelines & course objectives drive learning

& assessments reflect & drive performance


OUHSC Neurology



7

OU Neurology

COMPETENCY-BASED EDUCATION:

THE PROCESS GUARANTEES SUCCESS

1. Define “fundamental

knowledge”

2. Teach fundamentals

3. Test knowledge

4. Hope for best

3. Develop learning

experiences

2. Design measures and

standards of performance

1. Define successful graduate

Traditional / Flexnerian

“Planning Forwards”

Competency / Outcome-Based

“Planning Backwards”

Stephen R. Smith, M.D., MPH

A Practical Guide for Medical Teachers, 3rd ed. (2009)

Models for Curriculum Development

OU Neurology

THE LOGIC BEHIND

COMPETENCY-BASED EDUCATION

1. Competent physicians demonstrate certain knowledge, skills, attitudes, & behaviors (KSAB)

2. Graduating students & residents must demonstrate core KSAB

3. Assessment of core KSAB requires setting specific, measurable standards (= objectives or competencies), i.e., establishment of a competency-based curriculum

4. Learning core KSAB requires deliberate practice with feedback in a standardized curriculum

5. Demonstration of competence is an all-or-none phenomenon

6. Assessment of competence requires mastery testing with criterion-referenced grading


OUHSC Neurology



8

OU Neurology


10 ESSENTIAL FEATURES (THE ABCs)

Alignment of curriculum

Blended learning environment

Consistency—extramural & intramural

Core competencies

Deliberate practice for skills training

Experiential parallel to standardized learning

Formative & summative assessments

Grading criterion referenced (mastery testing)

Homogeneity (standardization)

Institutional support David Lee Gordon

OU Neurology

ALIGNMENT OF CURRICULUM

Step 1: Create practical, meaningful, clearly-stated, and measurable objectives (learning outcomes)

Step 2: Devise standardized learning experience consistent w/ objectives

Step 3: Make assessments (evaluations) consistent w/ objectives and learning

Assessment Learning Objectives

Objectives drive the process

A


OUHSC Neurology



9

OU Neurology

BLENDED LEARNING ENVIRONMENT Match learning & assessment strategies to objective type

Objective

Type

Directive Learning

Strategies

Assessment

Strategies

Cognitive Teach and test

core material

- Classroom

- Small group

- Readings

- Multimedia/web

Summative

- Written tests (MCQ,

matching, fill in blank)

- Oral exam

Skills Provide feedback,

deliberate practice

re: core material

- Patients

- SP / OSCE

- Multimedia/web

Formative & Summative

- Same as learning

strategies

Attitudes &

Behaviors

Integrate in

curriculum & role

model—openly

combat ethical

erosion of hidden

curriculum

- Patients

- SP / OSCE

- Reading / essay

- Small group

- Discussion

- Gaming

Formative

- Same as learning

strategies

B

OU Neurology

OU Neurology

BLENDED LEARNING ENVIRONMENT Match learning & assessment strategies to competence type

Does

Shows How

Knows How

Knows

Performance assessment in vivo

Undercover SPs, video, logs

Performance assessment in vitro

OSCE, SP-based test

Clinical-context-based tests

MCQ, essay, oral

Factual tests

MCQ, essay, oral

Miller, 1990

Wass et al., 2001

Miller’s Pyramid of Competence

B


OUHSC Neurology



10

OU Neurology

BLENDED LEARNING ENVIRONMENT The support-challenge model of learning (Bower, 1998)

SUPPORT

Core competencies /

Pre-identified objectives

Curricular alignment

Private formative

assessment (1-on-1

feedback)

CHALLENGE

Public formative

assessment (feedback

before or by peers)

Summative assessment

high

SUPPORT

CH

AL

LE

NG

E

RETREAT GROWTH

STASIS CONFIRMATION

high

B

“Rule with an iron fist & a heart of gold”

OU Neurology

CONSISTENCY

Extramural consistency Base course on consensus statement

Obtain external review

Intramural consistency Longitudinal

M1 through M4

KSAB (knowledge, skills, attitudes, behaviors)

Horizontal Intracurriculum / Intradepartment / Interdepartment

Hidden / informal curriculum

Across multiple clinical sites

Requires extensive communication and cooperation among course directors and faculty—and is most effective with institutional direction and oversight

C1


OUHSC Neurology



11

OU Neurology

CORE COMPETENCIES Required Characteristics

Pertinent (learner-population specific)

Integrated (defined first & as guide for rest of curriculum)

Comprehensive (KSAB, all 6 ACGME competencies)

Consistent (extramurally & intramurally)

Standardized (uniformly taught in curriculum)

Assessable (varied methods, matched to objective type)

(1) Patient care

(2) Medical knowledge

(3) Practice-based learning & improvement

(4) Interpersonal & communication skills

(5) Professionalism

(6) Systems-based practice

C2

OU Neurology

CORE COMPETENCIES Keys to Successful Implementation

Base all assessments on core competencies only

Publicize this fact to learners & teachers

Discourage teaching contrary to core competencies

Encourage teaching supplemental to core

competencies—but do not assess students on

supplemental information or skills

Assessing only core competencies leads to improved:

Retention of knowledge & skills

Learner confidence

Enthusiasm for the topic

Comprehension of supplemental learning

C2


OUHSC Neurology



12

OU Neurology

DELIBERATE PRACTICE FOR

SKILLS TRAINING…Evolution of a Concept

The value of experience is not in seeing much but in seeing wisely—Sir William Osler 1849-1919

Know that by practice alone can you become an expert—Sir William

Osler 1849-1919

Practice does not make perfect. Only perfect practice makes

perfect—Vince Lombardi 1913-1970

Expert performance can be traced to active engagement in

deliberate practice, where training (often designed and arranged by

teachers and coaches) is focused on improving particular tasks—K

Anders Ericsson 2008

D

“Experience” & “practice” are equally important to achieve

competence & expertise. This is true for all skill-based

professionals (e.g., athletes, musicians, dancers, pilots,

chess masters—and health professionals).

OU Neurology


SKILLS TRAINING…Definition & Components

Focused, repetitive practice consisting of domain-related activities necessary for improving performance & advancement to the level of expert

Essential components include: Motivated & attentive learner

Well-defined task and goals

Appropriate level of difficulty

Informative feedback from educational sources

Opportunities for repetition & refinements Modified from

Ericsson KA et al. Psychol Rev. 1993;100 (3):363-406

Ericsson KA Acad Emerg Med. 2008;15:988-94

McGaghie et al. Acad Med. 2011;86:706-11

D


OUHSC Neurology



13

OU Neurology


SKILLS TRAINING…Practical Implementation

Clinical environment alone is not conducive to consistent & comprehensive implementation of DP

Standardized curriculum that includes simulation is the only practical solution for DP implementation

Standardized patients (SPs)

Objective structured clinical exams (OSCEs)

Low-fidelity & high-fidelity simulators

Oral & written patient presentations

Case-based learning

Role-playing

Gaming

Multimedia – videos, avatars, etc.

Online delivery

D

OU Neurology

EXPERIENTIAL PARALLEL TO

STANDARDIZED LEARNING…Directors

Clinical experience:

Is ultimate & most valid venue for formative

assessment of skills, attitudes, & behaviors

Improves efficacy of deliberate practice—

provides context, with improved motivation

& attention

Must be associated with deliberate practice

to improve learner performance

Must be parallel to standardized

curriculum—concurrent & consistent

Must affirm

Can never contradict

Preferably supplements

Ward Rounds

Standardized

Patient

E


OUHSC Neurology



14

OU Neurology

EXPERIENTIAL PARALLEL TO

STANDARDIZED LEARNING…Attendings

Clinical preceptors can optimize student

skill acquisition and performance by

providing an environment:

1. Consistent with core competencies

established by the course director

2. Conducive to deliberate practice

3. With optimal balance of support & challenge

E

OU Neurology

FORMATIVE & SUMMATIVE

ASSESSMENTS…Principles & Definitions

General principles

Validity (appropriateness)

Reliability (consistency)

Feasibility (practicality)

Formative assessment (feedback—without grade)

Summative assessment (graded testing)

Make tests pertinent, assess core competencies

Set appropriate standards (minimal competence)

Overall course

Specific competencies

Use criterion-referenced (not norm-referenced) grading

Monitor results for course QI (curriculum evaluation)

Wass et al., 2001

Assessment drives learning—

accept & embrace it!

You should teach to the test.

F


OUHSC Neurology



15

OU Neurology

GRADING CRITERION REFERENCED

(MASTERY TEACHING)…vs. Norm Referenced

Criterion-Referenced (Absolute)

Individual vs. fixed standard (criterion)

Standard

Absolute—knowledge or skill learner

must have in order to pass

Based on instructor’s judgment

Known in advance (objectives)

Purpose is to assess competence

Mastery testing/assumption

Most students should perform well

(common mastery level 85%)

Bell-shaped curve skewed to right

Items assess core competencies,

linked to objectives

Diagnostic feedback relative to

performance is clear

Norm-Referenced (Relative)

Individual vs. others

Standard

Relative—based on class test

results (mean/standard deviation)

Influenced by examinees’ abilities

Not known in advance

Purpose is to discriminate

Normative assumption

Guaranteed some will fail

Bell-shaped curve, variable scores

Items assess broad content, not

linked to local objectives

Diagnostic feedback relative to

performance unclear

G

OU Neurology


(MASTERY TEACHING)…Sample Curves G

Mean SD SD CP Minimal 100%

CRITERION-REFERENCED

GRADING

NORM-REFERENCED

GRADING

Comparison to fixed standards Comparison to other students

HP

CP = Competent performer

HP = High performer


OUHSC Neurology



16

OU Neurology


(MASTERY TEACHING)…Assessment Goals

Curriculum vs. learner Outside test (e.g., NBME subject or “shelf” test)

Ensures question quality & security, saves time for educator

Conflicts w/ curricular alignment, assesses curriculum > learner

Internal test (created by curriculum developer)

Ensures curricular alignment, assesses learner > curriculum

Extramural consistency: base course on consensus statement

Quality: use question-writing guidelines & external review

Security & time: use same test repeatedly, but review tests personally in group format, never publish tests or questions

Competency vs. rank (lump vs. separate) Assessing competency requires criterion-referenced grading,

which is dichotomous (lumps)

Dichotomous grading does not preclude concurrent learner discrimination/ranking using scalar grading (separates)

G

OU Neurology


(MASTERY TEACHING)…with Scalar Grading

46.044.042.040.038.036.034.032.0

50

40

30

20

10

0

Std. Dev = 2.82

Mean = 43.0

N = 138.00

99

100

97

98

95

96

93

94

91

92

89

90

87

88

85

86

A = 97-100

B = 91-96

C = 85-90

F < 85

Curve is shifted to right (criterion-referenced), but

there are still high, average, and low performers (scalar)

G


OUHSC Neurology



17

OU Neurology

HOMOGENEITY

(STANDARDIZATION)

Curriculum based on mastering core competencies

and deliberate practice requires that ALL students

receive an equivalent—homogeneous or

standardized—learning experience

“Standardized” curriculum:

Is highly structured

Requires punctual & mandatory attendance

Takes precedence over ward responsibilities

Requires full attention of learners (pagers & phones off)

Accounts for majority of grade, but not student time

Requires strong, well-organized course director

H

OU Neurology

INSTITUTIONAL SUPPORT Leadership commitment

Curriculum champion

Faculty member delegated (not relegated) as director with mettle, motivation, authority, & protected time

Buy-in ability and bypass authority

Schedule 100% learner attendance

Cooperation of dept. (chair/faculty) & school (deans)

Simulation exercises, e.g., SPs, OSCEs, skills center

Resources Director > 50% FTE, coordinator > 50% FTE

Simulation / SP costs

Instructors (faculty / residents / nurses)

I


OUHSC Neurology



18

OU Neurology


10 ESSENTIAL FEATURES (THE ABCs)

Alignment of curriculum

Blended learning environment

Consistency—extramural & intramural

Core competencies

Deliberate practice for skills training

Experiential parallel to standardized learning

Formative & summative assessments

Grading criterion referenced (mastery testing)

Homogeneity (standardization)

Institutional support David Lee Gordon

OU Neurology

OPTIMAL CLINICAL-LEARNING ENVIRONMENT

IN COMPETENCY-BASED CURRICULUM

Three key factors:

1. Consistent with core competencies

established by the course director

2. Conducive to deliberate practice

3. With optimal balance of support & challenge


OUHSC Neurology



19

OU Neurology

THE END

The following 12 slides consist of

outcomes data (9 slides) and

references (3 slides)

OU Neurology

EFFECT OF COMPETENCY-BASED CURRICULUM

MIAMI NEUROLOGY CLERKSHIP 1998-2006 AAMC Graduation Questionnaires

93

.3

53

.6

57

.1

58

.6

60

.3

61

.5

62

.4

61

.6

61

.4 62

37

.5

34

.7

38

.5

31

.7

44

.3

83

.9 89

.1 94

.7

0

10

20

30

40

50

60

70

80

90

100

1998 1999 2000 2001 2002 2003 2004 2005 2006

Miami

All Schools

New curriculum

UMiami vs. All LCME Schools % Top Box % Students Who Rated Quality of Educational Experience Excellent or Good

Neurology

Clerkships

1998-2006

Excellent 64.2%

Excellent 24.9%


OUHSC Neurology



20

OU Neurology


OU-OKC NEUROLOGY CLERKSHIP 2005-2011 AAMC Graduation Questionnaires

37

.5

48

.7

31

.4

31

.7

94

.5

95

.3 97

61

.4 62

61

.2

61

.3

63

.3

73

.1

73

.4

0

10

20

30

40

50

60

70

80

90

100

2005 2006 2007 2008 2009 2010 2011

OU-OKC

All Schools

New curriculum

OU-OKC vs. All LCME Schools % Top Box % Students Who Rated Quality of Educational Experience Excellent or Good

Excellent 82.1%

Excellent 34.1%

Neurology

Clerkships

2005-2011

OU Neurology


OU-OKC CLERKSHIPS 2007-8 2009 AAMC Graduation Questionnaire (n = 73)

25

6.9

32.9

50.7

61.6 60.757.5

76.7

70.8

26.3

79.5

91.8

83.587.7 87.6

94.5

0

10

20

30

40

50

60

70

80

90

100

A B C D E F G NEURO

Excellent %

Top Box %


OUHSC Neurology



21

OU Neurology



44.3

8.2

23.4

30.2

56.1

45.8

57.5

75.579.2

57.2

77.6

70.8

89.7 88.8

96.2 95.3

0

10

20

30

40

50

60

70

80

90

100

A B C D E F G NEURO

Excellent %

Top Box %

OU Neurology

28.4

3.7

16.4

46.343.3

55.2

49.3

82.1

73.2

51.8

44.8

80.6

71.7

83.688.1

97

0

10

20

30

40

50

60

70

80

90

100

A B C D E F G NEURO

Excellent %

Top Box %




OUHSC Neurology



22

OU Neurology


OU-OKC TOP TWO RATED CLERKSHIPS 2009-2011 AAMC Graduation Questionnaires

3.4

3.5

3.4

3.8

3.7

3.7

3.2 3.4 3.6 3.8 4

2011

2010

2009

Neurology

Next HighestOU-OKC

Clerkship

2 wks

2 wks

4 wks

Length of

Neurology Clerkship

OU-OKC Clerkship Mean Scores 1 Poor, 2 Fair, 3 Good, 4 Excellent

OU Neurology

EFFECT OF COMPETENCY-BASED CURRICULUM UM STUDENTS CHOOSING NEUROLOGY CAREER

1

3 3 3

5

3 3

7 7

10

0

2

4

6

8

10

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

UMiami Students Choosing

Neurology or Child

Neurology as a Career

Graduates of new

curriculum


OUHSC Neurology



23

OU Neurology

EFFECT OF COMPETENCY-BASED CURRICULUM OU STUDENTS CHOOSING NEUROLOGY CAREER

1

2

0 0 0

1

0

8

7

10

6

0

2

4

6

8

10

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Graduates of new

curriculum

OU Students Choosing

Neurology or Child

Neurology as a Career

OU Neurology

EFFECT OF COMPETENCY-BASED CURRICULUM NEUROLOGY DEPARTMENT TEACHING AWARDS

TEACHING AWARD Before 2007 After 2007

Stanton L. Young Master

Teacher Award (f. 1984) Peggy W. Wisdom 1996

Herman E. Jones 2009

David Lee Gordon 2011

Edgar W. Young Lifetime

Achievement Award (f. 1987) Herman E. Jones 2011

M2 Aesculapian Award –

To Faculty (f. 1962) Herman E. Jones 2006


To Resident (f. 1975) AJ Vaughn 2011


To Faculty (f. 1962) Gunter Haase 1964 Herman E. Jones 2009


OUHSC Neurology



24

OU Neurology

SELECT TEXTBOOK REFERENCES

Dent JA, Harden RM. A Practical Guide for Medical Teachers. 3rd ed.

Edinburgh: Churchill Livingstone; 2009

Fincher RME, Cox S, DaRosa DA, Lynn DJ, Margo K, Morgenstern BZ,

Pangaro LN, Sierles FA. Guidebook for Clerkship Directors, 3rd edition.

Omaha: Alliance for Clinical Education; 2005

Huston T. Teaching What You Don’t Know. Cambridge, Mass.: Harvard

University Press; 2009

Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development

for Medical Education. A Six-Step Approach. Baltimore: The Johns

Hopkins University Press; 1998

Norman GR, van der Vleuten CPM, Newble DI. International Handbook

of Research in Medical Education. Dordrecht: Kluwer Academic

Publishers; 2002

OU Neurology

SELECT ARTICLE REFERENCES, 1 of 2

Bower DJ. Support-challenge-vision: a model for faculty

mentoring. Medical Teacher 1998;20:595-597

Ericsson KA, Krampe RT, Tesch-Römer C. The role of

deliberate practice in the acquisition of expert

performance. Psychological Review 1993;100:363-406

Ericsson KA. Deliberate practice and acquisition of expert

performance: a general overview. Acad Emerg Med

2008;15:988-994

McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH,

Wayne DB. Does simulation-based medical education

with deliberate practice yield better results than traditional

clinical education? A meta-analytic comparative review of

the evidence. Acad Med 2011;86:706-711


OUHSC Neurology



25

OU Neurology

SELECT ARTICLE REFERENCES, 2 of 2

McGaghie WC, Cohen ER, Wayne DB. Are USMLE step

1 & 2 scores valid measures for postgraduate medical

residency selection decisions? Acad Med 2011;86:48-52

Miller GE. The assessment of clinical skills / competence /

performance. Acad Med 1990;65:S63-S67

Smith SR, Dollase RH, Boss JA. Assessing students’

performance in a competency-based curriculum. Acad

Med 2003;78:97-107

Wass V, van der Vleuten C, Shatzer J, Jones R.

Assessment of clinical competence. Lancet

2001;357:945-949

OU Neurology

THE END

Date post:	12-Mar-2022
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Teaching Excellency in Competency-Based Education David L ...

Documents