Innovation in a traditional world-In-training assessment in PGMELord Cohen lecture, ASME 2010

Professor Charlotte Ringsted, MD, MHPE, PhD

Director of Centre for Clinical Education

University of Copenhagen and Capital Region DK, Rigshospitalet

Innovation in a traditional world

Innovation– Outcome-based education and in-training assessment in

Postgraduate Medical Education (PGME)

Traditional world– Undergraduate Medical Education (UGME)– End-of-training assessments and exams

In-training assessment programme– Anaesthesiology in DK– Internal medicine and Child and Adolescence psychiatry

Conclusion– Lessons learned and future directions

Danish Medical Education

Undergraduate• Bachelor - 3 years• Candidate - 3 years

Postgraduate– Internship 1½ year, Foundation 1 year

– Specialist education• Introduction 1 year• Specialist Residency 4-5 years

Continuous prof. development– Expert educations 1-2 years– Various

BasicScience

ClinicalScience

Intern

Intro-year

Specialistresidency

CPD

Expert ed.

Innovation: OBE and ITA

PGME 1991 NBH rules, guidelines Goals and objectives

– Specialist societies

Speciality courses Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation No exams

PGME reform 2001 NBH rules, guidelines Goals and objectives

– CanMEDS framework

Plus ’general courses’* Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation In-training assessment

Challenge - The seven roles

EFPO project, 1992– Undergraduate education,

Ontario, society’s needs, eight roles

Manager

CanMEDs project, 2000– Postgraduate education,

RCPSC, entire Canada, seven roles

Medical expert

Scholar

Communicator

Collaborator

Professional

Whole person

Health advocate

Canada and Denmark – Red and white; Neighbours; Hans Island

3 personsper km2

125 personsper km2

No 3

                            

Validity of CanMEDS roles

Survey among doctors in East DK– Responses from

3072 doctors

Roles important Increasing

confidence

Different specialitiesdifferent profiles

Ringsted et al. Med Educ 2006

Importance

Confindence*

CanMEDS roles Competence Teaching strategies Assessment

Med. exspert

Communicator

Health adv.

Collaborator

Manager.

Scholoar

Professional

Some challenge!

Intro-year Specialist Residency

What, how, when, who?

Internship

What is competence?

Competence is a habit of action

Competence = holistic overall capacity– ”The habitual and judicious use of communication,

knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served” Epstein and Hundert JAMA 2002

Competency = specific capability– ”Focuses on performance of the goal-state of instruction;

reflects expectations that are external to the programme; expressible in measurable behaviour; uses criterion standards for judging; informs learners and others about expectations” Albanese ME 2008

1. Challenge: Defining outcomes

Describing competence– Too detailed – Be able to manage - Lists of

procedural skills, diseases (+300 competencies)– Too general - The ‘other’ six roles - difficult to

define

Disintegration of the concept ‘competence’– Seven disciplines rather than an integrated,

context-based concept of competence

Expectations at various levels of training?– Different specialties, different traditions?

Various levels - AnaesthesiologyDreyfus, Epstein & Hundert

PERSONAdv. beginner

Novice

CONTEXTSmall teams

Close supervision

TASKSimpleSingle

1st yr

PERSONMasterExpert

CONTEXTComplex systems

Supervisor

TASKComplexAtypical

5th year

PERSONProficient

Competent

CONTEXTLarge teams

Distant supervision

TASKComplicated

Typical

2-4 year

Patients: ASA groups 1-5Surgery: Minor, medium, majorEM: Stay alive till assistance arrives

Internal medicine – levels?

About the difference between trainee levels:

”We pretty much do the same – patient encounters, ward rounds, ambulatory, etc. -The difference between levels is a matter of expectations - you are expected to take on a wider and deeper approach in managing the patients - and you are allowed to do more - make more decisions - simply because you are more experienced”...

Ringsted et al. Med Teach 2006

                                          

”We work in multi-professional teams – these are not stationary teams, but rather depending on the child and the situation.” – ”We learn from the team and gradually we

become more respected members of the team.– Our legitimacy is being a doctor with the right to

make certain decisions and prescribe medicine”

Child & Adolescence psychiatry

Davis et al. 2008

What is wrong

with this kid?

Conclusion - Different specialities

’Community of practice’Social task specialities

’Brick laying model’Technical task specialities

’Rings in the water’Cognitive task specialities

2. Challenge – assessment

What, when, how, who?

Beforeentry

Afterexit

Start ofrotation

End ofrotation

Duringrotation

MMI,Oral, MCQ

OSCE, Oral, MCQ

ITERABIM

Mini-CEXDOPS

Why in-training assessment?

Postgraduate education is work-based– 50% of the physician work-force are trainees– Quality of care relies on trainees’ competence

during training– ”End-of training examination is like reading

yesterday’s news”

In-training assessment, a tool for learning– Help clarify objectives according to broad aspects

of competence (CanMEDS roles)– Stimulate deep learning– Support effective and efficient education

Challenge in postgraduate education

Undergraduate education

Postgraduate education

Knows

Does

Can Can

Does

Knows

School-based

Work-based

?

• Learn how to manage cases• Learn from managing cases• Reflect in and on practice

Mastery or Development ?

Competency as capabilityrelated to specific tasks

1. 2. 3. 4. 5. 6.

Scoring

Competence as holisticcapacity related to any task•Aspects of the 7 roles

Time

987654321

No single method can measure it all – V.d .Vleuten 2010

assessment programmes are recommended

In-training-assessment programmes

Anaesthesiology

Internal Medicine

In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003

•Cusum scoring•Logbook on experience•Learning portfolio

•Communication skills (1)•Management/collaboration (2)•Academic competence (3)

1 st

year

trainingClinical skills assessments (12)

Observationin vivo / vitro

Observationin vivo / vitro

Assessment based on practice data and written reflective assignments/reports

Assessment based on practice data and written reflective assignments/reports

Longitudinalassessment

Longitudinalassessment

Assessment of written assignment

Reflection over a patient– Description of patient and operation– Theoretical and practical consideration regarding

choice of anaesthesiological approach related to patient condition, wishes, surgery, and context

– Describe potential problems and complications and discuss strategies to minimise these

– Describe actual patient course and events– Reflection related to pre-operative considerations– Use references from literature in the reflection

Why focus on theory and reflection? Klemola and Norros, ME 1997, 2001

Realistic orientation– Recognition of

uncertainty and unpredictability

– Communicative relationship: each patient is unique

Objectivistic orientation– No recognition of

uncertainty and unpredictability

– Authoritative relationship: ’a case’: coronary, asthmatic, etc

Anaesthesiology– Clinical physiology and pharmacology; Procedural skills; Monitoring

of respiratory and cardiovascular parameters; Context – patient, surgery, team

Two distinct patterns related to ‘experts’

Habit of action Klemola and Norros, ME 1997, 2001

Interpretative– Combine monitor

information with situational information and background knowledge

– Recognition of the versatility of information from several resources, oxygen SAT, End-tidal CO2, etc.

Reactive– Operate directly with

the numbers

– Contradictory use of monitors, emphasising importance regarding patient safety without understanding the mediated character and versatility of information

Knowledge and skills

Causal understanding of concepts, principles, and tool design affects retention and transfer of learning Woods et al. 2006, 2007, Schwartz 2004

Self-regulatory processes in development of expertise Zimmerman 2006

– Forethought: Task analysis, strategic planning– Performance: Contextual adaptation of strategies– Post-task: Evaluation and reflection

Bech et al. EJVS 2010

Knowledge and Anaesthsiology Klemola and Norros, ME 1997, 2001

Forethought: physiological potentials– ”He can go uphill without getting out of breath, so probably he

will tolerate anaesthesia well. Major problem might be oxygenation and ventilation.”

Adaptive strategy: physio-pharmacological experiment– ”You can’t tell how an elderly patient will react. You have to

check his responses and give drugs accordingly.”

Evaluation and reflection-in-action– ”The patient has capacity to compensate for side-effects of

anaesthesia through sympathetic activation, a kind of capacity that elderly patients do no necessarily possess. That is a safe thing to observe”

Flexner ?

Routine expert vs. Adaptive expert

”Most professionals reach a stable,average level of performanceand maintain this mediocrestatus for the rest of their careers.”

Routine expertsPerf

orm

an

ce

Experience

Ericsson 2004, Guest et al.2001, Choudhry et al. 2005, Schwartz 2004

Adaptive experts

A team and a coach

3. Challenge: Validity and reliability?

Assessment free area– Focus on evaluation of

quality of education• “To emphasize the educational

purpose of training, comprehensive formative evaluation is suggestedas alternative to specialist examinations.” Karle, Nystrup ME1995

– Strong humanist tradition in education, qualitative R&D

• Deprived of quantitative educational researchand psychometrics

– Ministry of research, innovation, and technology 2007, OECD review 2005

North America

Assessment rich area– Flooded by quantitative

data and psychometricians– National exams– Heavy focus on reliability

of tests and exams– Strong tradition of

cognitive psychology and behaviourism

• Hodges and Segouin, ME08

DK

Schuwirth & v.d. Vleuten ME 2006

A plea for new psychometric methods

In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003

•Cusum scoring•Logbook on experience•Learning portfolio

Trainees’ opinion of assessment (1-9)

15141414131315N =

10

8

6

4

2

0

610

2

143

14

115

Ringsted et al. AAS 2003

Internal medicine

Ringsted et al. Med Teach 2006; Davis et al 2005; Norgaard et al. Med Educ 2004

Intro-year trainees’ learning needs, case-mix, and quality assurance data

Structured approach to the tasks– Ward rounds– Complex patients – get an overview

Emergency care– Team-leader skills

Collaboration– Team-work and inter-personal skills

Ethics and professionalism– Difficult decisions

Usefulness of the assessment

Patient consultation (Lung) 5

Patient consultation (heart) 6

Review of primary patient record 6

Review of patient course, reflection 6.5

Audit of records (nutrition, pain, ability, temp.) 6.5

Presentation at conference 7

Ward round 7

EBM assignment 7

Presentation 7

Median (1-9)

About the written assignments

“Extremely good learning experience - to do this review of a patient’s course ”

“It was hard work” (Trainee) “This is really a valuable innovation in the

education - these assignments” (Trainee) “It was more easy than I thought - to

review these assignments” (Supervisor) “This is an advantage to the entire

department - we all learn from these..”

Kirsten Nørgaard, MHPE, 2004

Factors related to value of ITA Ringsted et al. ME 2004, Med Teach 2003, ASS 2003

The link to practice– Help in structuring teaching, training and learning

• Outcomes clear, monitoring progress, identify problems• Coupling of theory to practice

– Used as licence to practice rather than end-of-training assessment

The effect on learning– Should include a challenge to the learner– ‘We all learn more’

Assessors’ attitudes– Enthusiasm and rigour

Conclusion

Lessons learned

Future directions

Lessons learned

No 1 – Outcome-based education– ‘CanMEDs roles’ is a nice mental framework.

Need for both competency-goals (outcome/ efficiency) and competence-goals (process/ innovation)

No 2 – In-training assessment– A valuable tool for learning. Meaningful

programmes are tailored to clinical context and trainees’ level of professional development.

No 3 – The process– Useful to take a design-based research approach:

Cycle of critical review of literature, design, enactment, evaluation, and large working groups

Future direction

Some questions– Can deliberate practice, reflection in and on

practice, adaptive expertise be learned? Or is it an in-born trait in the minority of us?

– If it can be learned, how can we facilitate the development and measure the progress of it?

– How can we better align education, training, and learning with quality of practice and measure the effect?

– How can we turn education into a resource by using the capacity of our trainees to develop practice as a whole?

Time’s up

In the honour of Lord Cohen

Thank you for your attention

??