+ All Categories
Home > Documents > Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we...

Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we...

Date post: 14-Apr-2020
Category:
Upload: others
View: 2 times
Download: 0 times
Share this document with a friend
63
Between Scylla and Charybdis: Between Scylla and Charybdis: Navigating the discourses of Navigating the discourses of discourses of competence discourses of competence Brian Hodges MD, PhD, FRCPC Professor, Scientist and Director Wilson Centre for Research in Education Brian Hodges MD, PhD, FRCPC Professor, Scientist and Director Wilson Centre for Research in Education Wilson Centre for Research in Education University of Toronto Wilson Centre for Research in Education University of Toronto
Transcript
Page 1: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Between Scylla and Charybdis:

Between Scylla and Charybdis: y

Navigating the discourses of

yNavigating the discourses of discourses of competence discourses of competence

Brian Hodges MD, PhD, FRCPCProfessor, Scientist and Director

Wilson Centre for Research in Education

Brian Hodges MD, PhD, FRCPCProfessor, Scientist and Director

Wilson Centre for Research in EducationWilson Centre for Research in EducationUniversity of Toronto

Wilson Centre for Research in EducationUniversity of Toronto

Page 2: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Medical education discourseMedical education discourse

As every medical professor knows you just find a

As every medical professor knows you just find a knows, you just find a patient with some fractures and escort a gaggle of students to the bedside for

knows, you just find a patient with some fractures and escort a gaggle of students to the bedside for students to the bedside for a good long stare.

A l h ff i

students to the bedside for a good long stare.

A l h ff i As long as the sufferer is not prodded too hard, or denied a clear view of the

l h l l

As long as the sufferer is not prodded too hard, or denied a clear view of the

l h l ltelevision, no ethical rules are violated.television, no ethical rules are violated.

Economist, M h 24 2007March 24, 2007

Page 3: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Is this how you talk about Is this how you talk about Is this how you talk about medical education?

Is this how you talk about medical education?

We all have models (discourses) about medical education and competenceWe all have models (discourses) about medical education and competenceThe discourses we use frame the way we see and talk about what competence i

The discourses we use frame the way we see and talk about what competence iisThese discourses change in time and l

isThese discourses change in time and lplaceplace

Page 4: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Encountering different Encountering different Encountering different discourses by travelingEncountering different discourses by traveling

My interest in discourses took me on a 6 year voyage to real places (France, My interest in discourses took me on a 6 year voyage to real places (France, China, Ethiopia, Israel, Jordan, Pakistan…) and virtual locations (books, internet documents) where

China, Ethiopia, Israel, Jordan, Pakistan…) and virtual locations (books, internet documents) where internet, documents) where competence is discussed and debatedinternet, documents) where competence is discussed and debated

I am going to take you on an imaginary version of that voyageI am going to take you on an imaginary version of that voyageversion of that voyageversion of that voyage

Page 5: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

h hh hThis is our shipThis is our ship

The HippocratesHippocrates

Page 6: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’sOur map*Our map*

Harrison s Island

Miller’s Island

Cronbach’s Island

Taylor’s Island

Schön’s Island

*Conceptual destinations

Page 7: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Our guide booksOur guide books

Page 8: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.
Page 9: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Our dictionaries of language

Our dictionaries of language language language

Harrison’s IslandHarrison’s IslandHarrison s IslandKnowledge discourse

Miller’s Island

Harrison s IslandKnowledge discourse

Miller’s IslandMiller s IslandPerformance discourse

Cronbach’s Island

Miller s IslandPerformance discourse

Cronbach’s IslandCronbach s IslandPsychometric discourse

Taylor’s Island

Cronbach s IslandPsychometric discourse

Taylor’s IslandTaylor s IslandProduction discourse

Schön's Island

Taylor s IslandProduction discourse

Schön's IslandSchön s IslandReflection discourse

Schön s IslandReflection discourse

Page 10: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Our guide to culture and h

Our guide to culture and harchitecturearchitecture

Page 11: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’sHarrison s Island

Miller’s Island

Cronbach’s Island

Taylor’s Island

Schön’s Island

Page 12: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’s IslandHarrison’s Island

Copyright ©2000 BMJ Publishing Group Ltd.

Page 13: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Medical school architecture Medical school architecture Medical school architecture Medical school architecture Medical school architecture Medical school architecture on Harrison’s Islandon Harrison’s Island

Medical school architecture Medical school architecture on Harrison’s Islandon Harrison’s Island

Page 14: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’s IslandHarrison’s IslandKnowledge Knowledge discoursediscourse

Facts, foundational knowledge, basic science, first Facts, foundational knowledge, basic science, first principles, fund of knowledge, classic text books, principles, fund of knowledge, classic text books, classic articlesclassic articles

Roles and Roles and activitiesactivities

Teacher’s: Teacher’s: Lecturers, source of wisdomLecturers, source of wisdomStudent’s: Student’s: Memorize, reproduceMemorize, reproduce

Tests: MultipleTests: Multiple--choice examschoice exams

Believe in…Believe in… Knowledge accumulationKnowledge accumulation

ReligionReligion Cognitive psychologyCognitive psychology

Page 15: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Warnings about this Warnings about this islandisland

The preoccupation with doing well on standardized tests has conditioned the way young people …think. They have better-developed cognitive

The preoccupation with doing well on standardized tests has conditioned the way young people …think. They have better-developed cognitive y p gabilities to recognized random facts than to construct patterns or think systematically.

Jacques Barzun, 1988

y p gabilities to recognized random facts than to construct patterns or think systematically.

Jacques Barzun, 1988q ,

Student: Madam Madam –– do you have higher do you have higher conjugated or unconjugated conjugated or unconjugated

q ,

Student: Madam Madam –– do you have higher do you have higher conjugated or unconjugated conjugated or unconjugated conjugated or unconjugated conjugated or unconjugated bilirubin?bilirubin?

Harrison’s Island Medical Student

conjugated or unconjugated conjugated or unconjugated bilirubin?bilirubin?

Harrison’s Island Medical Student

Page 16: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’sHarrison s Island

Miller’s Miller’s IslandIsland

Cronbach’s Island

Taylor’s Island

Schön’s Island

Page 17: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Miller’s IslandMiller’s Island

Copyright ©2000 BMJ Publishing Group Ltd.

Page 18: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Medical School Architecture on Medical School Architecture on

Miller’s IslandMiller’s Island

Medical Student

Simulated Patient

Interaction

Physician Examiner

Medical classroom

Page 19: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Miller’s IslandMiller’s IslandPerformancePerformance Simulations simulated patient feedbackSimulations simulated patient feedbackPerformance Performance discoursediscourse

Simulations, simulated patient, feedback, Simulations, simulated patient, feedback, performance, skills, skills centre, OSCE, multiple performance, skills, skills centre, OSCE, multiple observationsobservations

Roles and Roles and activitiesactivities

Teacher’s:Teacher’s: Demonstrating and observing skills Demonstrating and observing skills Student’s: Student’s: Performing for observers Performing for observers Tests: PerformanceTests: Performance--based assessment based assessment

(simulation, OSCE) (simulation, OSCE)

Believe in…Believe in… Observation of performanceObservation of performance

ReligionReligion BehaviourismBehaviourism

Page 20: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The way they talk on The way they talk on The way they talk on The way they talk on The way they talk on The way they talk on Miller’s IslandMiller’s Island

The way they talk on The way they talk on Miller’s IslandMiller’s Island

In many places they would ask students to write an essay on the origin of the word shoelace or give them a multiple choice

In many places they would ask students to write an essay on the origin of the word shoelace or give them a multiple choice shoelace, or give them a multiple choice question on the design of shoelaces or even ask them to describe the steps in tying a h l

shoelace, or give them a multiple choice question on the design of shoelaces or even ask them to describe the steps in tying a h l shoelace.

Whereas really the only way of doing it is

shoelace.

Whereas really the only way of doing it is Whereas really the only way of doing it is showing you know how to tie a shoelace.

Ronald Harden 2005

Whereas really the only way of doing it is showing you know how to tie a shoelace.

Ronald Harden 2005

Page 21: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Warnings about Warnings about gMiller’s Island

gMiller’s Island

Exclusive reliance on a pedagogical approach of simulation training may be encouraging students to become “simulation doctors” who act out a good

Exclusive reliance on a pedagogical approach of simulation training may be encouraging students to become “simulation doctors” who act out a good grelationship with their patients but have no authentic connection with them

Hanna and Fins (2006)

grelationship with their patients but have no authentic connection with them

Hanna and Fins (2006)Hanna and Fins (2006)

Student: Oh that must be hard for you…wow that t b h d f h th t t b ll

Hanna and Fins (2006)

Student: Oh that must be hard for you…wow that t b h d f h th t t b ll must be hard for you… oh, yes that must be really

hard for youPatient: Can you stop saying that?!

must be hard for you… oh, yes that must be really hard for you

Patient: Can you stop saying that?!Medical Student, Miller’s IslandMedical Student, Miller’s Island

Page 22: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’sHarrison s Island

Miller’s Island

Cronbach’s Cronbach’s IslandIsland

Taylor’s Island

Schön’s Island

Page 23: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Cronbach's IslandCronbach's IslandCronbach's IslandCronbach's Island

Page 24: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Medical school architecture on Medical school architecture on

Cronbach’s IslandCronbach’s Island

Medical Student

Standardized Patient

Interaction

Ph i i

Interaction

Psychometricians

PhysicianExaminer OSCE

Station

Clinical skills testing centre

Page 25: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The way they talk on The way they talk on The way they talk on Cronbach’s island

The way they talk on Cronbach’s island

Knowledge of educational products and educational purposes must become quantitative, take the form of measurement

Knowledge of educational products and educational purposes must become quantitative, take the form of measurement of measurement.

Education is one form of human engineering and will

of measurement.

Education is one form of human engineering and will g gprofit by measurement of human nature and achievement as mechanical and electrical engineering have profited by using the foot-pound,

g gprofit by measurement of human nature and achievement as mechanical and electrical engineering have profited by using the foot-pound, e g ee g a e p o ted by us g t e oot pou d,calorie, volt and ampere.

Thorndike, 1922

e g ee g a e p o ted by us g t e oot pou d,calorie, volt and ampere.

Thorndike, 1922

Page 26: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Cronbach’s IslandCronbach’s IslandPsychometricPsychometric reliability validity generalizability datareliability validity generalizability dataPsychometric Psychometric discoursediscourse

reliability, validity, generalizability, data, reliability, validity, generalizability, data, psychometrician, candidate, checklist, itempsychometrician, candidate, checklist, item--banking, banking, cutcut--point, standardizationpoint, standardization

Roles and Roles and activitiesactivities

Teacher’s:Teacher’s: Preparing students for examsPreparing students for examsStudent’s:Student’s: Strategies to max scores on exams Strategies to max scores on exams Tests: Standardized, summative exams, Tests: Standardized, summative exams,

Believe in…Believe in… The conversion of human behaviour to numbers The conversion of human behaviour to numbers

ReligionReligion PsychometricsPsychometrics

Page 27: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Learning the Learning the language of Cronbach’s language of Cronbach’s Cronbach s Cronbach s

In the 1980s many people moved from the older Miller’s and Harrison’s Island to Cronbach’s and learned its language of psychometrics

In the 1980s many people moved from the older Miller’s and Harrison’s Island to Cronbach’s and learned its language of psychometricslearned its language of psychometrics

The standardized-patient technique …has many of the same

learned its language of psychometrics

The standardized-patient technique …has many of the same p q yadvantages of the multiple-choice question

It is a standardized item, can be given in multiples, and can be

p q yadvantages of the multiple-choice question

It is a standardized item, can be given in multiples, and can be It is a standardized item, can be given in multiples, and can be scored in reliable and valid ways.

Howard Barrows, formerly of Miller’s Island 1993

It is a standardized item, can be given in multiples, and can be scored in reliable and valid ways.

Howard Barrows, formerly of Miller’s Island 1993

Page 28: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Warning: There are severe Warning: There are severe paradigmatic storms around

Cronbach’s islandparadigmatic storms around

Cronbach’s islandCronbach s islandCronbach s island

H i Bi d BHarrison Bird Brown

Page 29: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Warnings about Warnings about gCronbach’s island

gCronbach’s island

In a standardized environment, it becomes a problem when [a patient] doesn’t look the same as everybody else -Diana Tabak 2006

In a standardized environment, it becomes a problem when [a patient] doesn’t look the same as everybody else -Diana Tabak 2006

I have heard enough anecdotes about the shotgun behaviour induced by checklists to shift the

I have heard enough anecdotes about the shotgun behaviour induced by checklists to shift the behaviour induced by checklists to shift the burden of proof onto the advocates of this strategy -Geoff Norman 2005

behaviour induced by checklists to shift the burden of proof onto the advocates of this strategy -Geoff Norman 2005

I think the problem is that as we’ve tried to get more valid and more reliable statistically, we’ve

I think the problem is that as we’ve tried to get more valid and more reliable statistically, we’ve actually come away from what we were really trying to do in the first place. -Jean Ker 2005actually come away from what we were really trying to do in the first place. -Jean Ker 2005

Page 30: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

From these choppy waters, …From these choppy waters, …From these choppy waters, …From these choppy waters, …

We now follow the route that some took from the older islands to two We now follow the route that some took from the older islands to two neighboring IslandsOff the coasts of Taylor’s and neighboring IslandsOff the coasts of Taylor’s and Off the coasts of Taylor s and Schön's Islands, there are dangers!Off the coasts of Taylor s and Schön's Islands, there are dangers!

Caught between the Caught between the S ll d Ch bdiS ll d Ch bdiScylla and CharybdisScylla and Charybdis

Page 31: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’sHarrison s Island

Miller’s Island

Cronbach’s Island

Taylor’s Island

Schön’s Island

Page 32: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Taylor’s IslandTaylor’s IslandTaylor s IslandTaylor s Island

Michael Wolf, Mother Jones July/August2005July/August2005

Page 33: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Medical School Architecture Medical School Architecture on Taylor’s Islandon Taylor’s Island

C did t St d di d

Physician Examiners

Candidate StandardizedPatient

Interaction

h i iOSCE Station

Management

Psychometricians

OSCE testingOSCE testinginstitution

Page 34: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Taylor’s IslandTaylor’s IslandProductionProduction Cost, production, finished product, accountability,Cost, production, finished product, accountability,Production Production discoursediscourse

Cost, production, finished product, accountability, Cost, production, finished product, accountability, outcomes, efficiencyoutcomes, efficiency

R l dR l d T h ’T h ’ M t d l bM t d l bRoles and Roles and activitiesactivities

Teacher’s:Teacher’s: Management and labourManagement and labour

Student role: Raw material or productStudent role: Raw material or product

Tests: Mass Testing quality assuranceTests: Mass Testing quality assuranceTests: Mass Testing, quality assuranceTests: Mass Testing, quality assurance

Believe Believe Productivity and efficiencyProductivity and efficiencyin…in…

ReligionReligion Production/CapitalismProduction/Capitalism

Page 35: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The way they talk The way they talk The way they talk on Taylor’s IslandThe way they talk on Taylor’s Island

The corporate order, with its assembly-line techniques job differentiation and

The corporate order, with its assembly-line techniques job differentiation and line techniques, job differentiation, and increased organizational size, demanded a different type of office

line techniques, job differentiation, and increased organizational size, demanded a different type of office demanded a different type of office space and a more regulated and regimented flow of time

demanded a different type of office space and a more regulated and regimented flow of time

Kwolek-Folland (1994) Kwolek-Folland (1994)

Page 36: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Learning the Learning the Learning the language of Taylor’s

Learning the language of Taylor’s

Today, we have reached new heights in the science of producing and testing medical students.

Today, we have reached new heights in the science of producing and testing medical students.students.

We run the operation using standardized and efficient p ocess

students.

We run the operation using standardized and efficient p ocessefficient process.

As a result we are in a much better position to

efficient process.

As a result we are in a much better position to passure the public that we are graduating a product with the skills for safe and effective practice.

passure the public that we are graduating a product with the skills for safe and effective practice.practice.practice.

Page 37: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Beware the Scylla lurking in lurking in the waters f T l ’ of Taylor’s

Island!Athena (2007) Women in Greek Mythsin Greek Myths

Page 38: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The siren call of d d h

The siren call of d d hproductivity and the

death of feedbackproductivity and the death of feedbackdeath of feedbackdeath of feedback

In a production model, the many In a production model, the many p , yassessments of “quality” are summative, for external bodies, and

p , yassessments of “quality” are summative, for external bodies, and there is no feedback to students.there is no feedback to students.

Health professionals live in an “examined society” in which constant surveillance and testing locates the responsibility for their

Health professionals live in an “examined society” in which constant surveillance and testing locates the responsibility for their testing locates the responsibility for their competence externally to the individual.testing locates the responsibility for their competence externally to the individual.

Page 39: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

External monitoring of External monitoring of External monitoring of quality may help to:

External monitoring of quality may help to:

Discover and remove a few very low Discover and remove a few very low performers (although the eventual fail rate on most health professional examinations is near zero)

performers (although the eventual fail rate on most health professional examinations is near zero)near zero)Reassure the public that the profession is monitoring knowledge and skills of the next

ti

near zero)Reassure the public that the profession is monitoring knowledge and skills of the next

tigenerationStandard setting across the country and internationally

generationStandard setting across the country and internationallyinternationallyinternationally

Page 40: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

But some say this But some say this culture of production

is dangerousculture of production

is dangerousis dangerousis dangerous

I di d l i I di d l i It distorts student learning Produces no feedback to advance learning L d t diti ti d t ti

It distorts student learning Produces no feedback to advance learning L d t diti ti d t tiLeads to commoditization and contracting-out of examinations, test preparation and even teaching, with significant increases in

Leads to commoditization and contracting-out of examinations, test preparation and even teaching, with significant increases in e e teac g, t s g ca t c easescosts for students and professionalse e teac g, t s g ca t c easescosts for students and professionals

If you create a sausage factory - you will get sausages.

If you create a sausage factory - you will get sausages.

Page 41: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harrison’sHarrison s Island

Miller’s Island

Cronbach’s Island

Taylor’s Island

Schön’s Schön’s IslandIsland

Page 42: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Schön's IslandSchön's Island

Page 43: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Medical school architecture Medical school architecture on Schön's Islandon Schön's Island

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Medical Student

Teacher/Examiner

Page 44: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The way they talk on The way they talk on The way they talk on Schön's Island

The way they talk on Schön's Island

The concept of learner as a mere The concept of learner as a mere The concept of learner as a mere processor of information has been replaced by the image of a self-

The concept of learner as a mere processor of information has been replaced by the image of a self-replaced by the image of a selfmotivated, self-directed problem solver.

replaced by the image of a selfmotivated, self-directed problem solver.

Ministry of Education 1980Ministry of Education 1980

Page 45: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Schön's IslandSchön's IslandSchön s IslandSchön s IslandReflection Reflection discoursediscourse

reflection, selfreflection, self--directed learning, learning contracts, directed learning, learning contracts, portfolios, adult learnerportfolios, adult learnerdiscoursediscourse portfolios, adult learnerportfolios, adult learner

Roles and Roles and ti ititi iti

Teacher: Teacher: Guide/Mentor/ConfessorGuide/Mentor/ConfessorSt d t lSt d t l R fl ti lfR fl ti lf l il iactivitiesactivities Student role: Student role: Reflecting, selfReflecting, self--analysisanalysisMeasures: PortfolioMeasures: Portfolio

Believe Believe in…in…

SelfSelf--assessment, selfassessment, self--direction, selfdirection, self--reflectionreflection

ReligionReligion SelfSelf--actualization, psychoanalysisactualization, psychoanalysis

Page 46: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The ‘trinity’ of reflective The ‘trinity’ of reflective The trinity of reflective technologies

The trinity of reflective technologies

Self-assessment, self-direction and self-regulationSelf-assessment, self-direction and self-regulationAdult learning theory and the idea that the locus for control of competence is i l

Adult learning theory and the idea that the locus for control of competence is i l internal Different technologies such as:

tf li fl ti di i l b k

internal Different technologies such as:

tf li fl ti di i l b k portfolios, reflective diaries, logbooks, self-directed web-based modules portfolios, reflective diaries, logbooks, self-directed web-based modules

Page 47: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

But beware!

But beware! beware! You can beware! You can vanish into the vanish into the into the

Charybdis!into the

Charybdis!Athena (2007) Women in Greek Myths, http://www.paleothea.com/index.html

Page 48: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Warning’s about Warning’s about Schön's IslandSchön's Island

In a review of many studies on self-assessment, “There is a subset of clinicians who appear, either by training or personality unable to judge

In a review of many studies on self-assessment, “There is a subset of clinicians who appear, either by training or personality unable to judge by training or personality, unable to judge themselves”

Dave Davis (2006)

by training or personality, unable to judge themselves”

Dave Davis (2006)

In adopting self-directed methods for maintenance of competence, “Regulators appear quite

In adopting self-directed methods for maintenance of competence, “Regulators appear quite p g pp qunconcerned about the lack of coherence between what is being monitored ‘at a distance’ and the actual professional knowledge (needed) to function skilfully and competently”

p g pp qunconcerned about the lack of coherence between what is being monitored ‘at a distance’ and the actual professional knowledge (needed) to function skilfully and competently”to function skilfully and competently

Nelson and Purkis (2004)

to function skilfully and competently

Nelson and Purkis (2004)

Page 49: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The archetype of the The archetype of the ypinternally, self-

regulating professional

ypinternally, self-

regulating professionalregulating professionalregulating professional

Reflects regularly on daily practiceSelf-assesses gaps in knowledge or skillReflects regularly on daily practiceSelf-assesses gaps in knowledge or skillSeeks opportunities to redress gapsInvests energy to learn (or relearn)I t k l d i t ti

Seeks opportunities to redress gapsInvests energy to learn (or relearn)I t k l d i t tiIncorporates new knowledge into practiceIncorporates new knowledge into practice

Handfield-Jones 2002

Page 50: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

But if we read the But if we read the But if we read the literature…

But if we read the literature…

We look for gaps? “S lf t ti ” f ti f lf fl ti

We look for gaps? “S lf t ti ” f ti f lf fl ti“Self-protective” function of self-reflection

We find gaps when we look? The ineffecti eness of self assessment

“Self-protective” function of self-reflection

We find gaps when we look? The ineffecti eness of self assessmentThe ineffectiveness of self-assessment

We try to address gaps through learning?We study what we are already good at

The ineffectiveness of self-assessment

We try to address gaps through learning?We study what we are already good atWe study what we are already good at

We incorporate new information into practice?

We study what we are already good at

We incorporate new information into practice?practice?

Ineffectiveness of CME for changing practicepractice?

Ineffectiveness of CME for changing practice

Page 51: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

A repeated patternA repeated patternA repeated patternA repeated pattern

8090

100

50607080

rcen

tile

Actual ScoresPerceived Scores

10203040Pe

r

Perceived (After)

010

BottomQuartile

2ndQuartile

3rdQuartile

TopQuartile

Kruger and Dunning 1999, Hodges et al 2000

Page 52: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

h fl h hh fl h hThe flaw in the theoryThe flaw in the theory

Theories of adult learning focus on the reasons why people DO learnTheories of adult learning focus on the reasons why people DO learn

Little or no research on why people DON’T learnLittle or no research on why people DON’T learn

It is impossible to make people understand h i i f i i k l d

It is impossible to make people understand h i i f i i k l d their ignorance, for it requires knowledge to

perceive it; and therefore, he that can perceive it, hath it not

their ignorance, for it requires knowledge to perceive it; and therefore, he that can perceive it, hath it not

Jeremy TaylorJeremy Taylor

Page 53: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

The voyage The voyage homehome

Page 54: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Traveler’s diseases you Traveler’s diseases you Traveler s diseases you don’t want to bring homeTraveler s diseases you

don’t want to bring home

U bl d li d k l dU bl d li d k l dUnusable decontextualized knowledgeDeficient or fake interpersonal skills Homogenization/suppression of diversity

Unusable decontextualized knowledgeDeficient or fake interpersonal skills Homogenization/suppression of diversityHomogenization/suppression of diversityCommoditization of education technology and rising costs to students and professionalsI i h t f t h t d t l ti hi

Homogenization/suppression of diversityCommoditization of education technology and rising costs to students and professionalsI i h t f t h t d t l ti hi Impoverishment of teacher-student relationship and the loss of feedbackReflection without substance

Impoverishment of teacher-student relationship and the loss of feedbackReflection without substance

Page 55: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Lessons from Harrison’s Lessons from Harrison’s Lessons from Harrison s and Miller’s Islands

Lessons from Harrison s and Miller’s Islands

Knowledge is important -but try to avoid teaching and testing “pure”

Knowledge is important -but try to avoid teaching and testing “pure” and testing “pure” knowledge - integrate with skills and real context

and testing “pure” knowledge - integrate with skills and real context

Skills are important by must be integrated with Skills are important by must be integrated with ust be teg ated tknowledgeWhen using simulations ensure transfer to

ust be teg ated tknowledgeWhen using simulations ensure transfer to ensure transfer to ‘authentic’ clinical settingsensure transfer to ‘authentic’ clinical settings

Page 56: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Lessons from Cronbach’s Lessons from Cronbach’s IslandIsland

Highly standardized psychometric methods can be helpful, but they have

Highly standardized psychometric methods can be helpful, but they have be helpful, but they have some serious side-effects

Commit to di e sit in

be helpful, but they have some serious side-effects

Commit to di e sit in Commit to diversity in teaching/testing, reduce rigid standardization -real lif i “ ”

Commit to diversity in teaching/testing, reduce rigid standardization -real lif i “ ”life is “messy”

Minimize checklists and

life is “messy”

Minimize checklists and Minimize checklists and “reductionistic” measuresMinimize checklists and “reductionistic” measures

Page 57: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Lessons from Taylor’s IslandLessons from Taylor’s IslandLessons from Taylor s IslandLessons from Taylor s IslandDon’t created a big teaching Don’t created a big teaching g gand testing factory that does only end-of-line quality assurance

g gand testing factory that does only end-of-line quality assurance

Use small-scale teaching and frequent local assessment Use small-scale teaching and frequent local assessment frequent, local assessment whenever possible versus “one-shot” mass assessment

frequent, local assessment whenever possible versus “one-shot” mass assessment

Avoid mechanization, labour-management, manufacturing Avoid mechanization, labour-management, manufacturing g , gmodels that dehumanize medical education

g , gmodels that dehumanize medical education

Page 58: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Lessons from Schön's IslandLessons from Schön's IslandLessons from Schön s IslandLessons from Schön s IslandUse reflection carefully: Enough

t l i t t t Use reflection carefully: Enough

t l i t t t external input to correct inaccuracies of self-judgment; not so much it deforms learning

external input to correct inaccuracies of self-judgment; not so much it deforms learning

Think of teacher as coach who can help a student (or a MD in Think of teacher as coach who can help a student (or a MD in can help a student (or a MD in practice) be sure they know “when to slow down” or “when to look it up”

can help a student (or a MD in practice) be sure they know “when to slow down” or “when to look it up”to look it up

Use “self-selected external t ” d f ll

to look it up

Use “self-selected external t ” d f ll assessments” and follow-up on

learning plansassessments” and follow-up on learning plans

Page 59: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

I hope you have enjoyed I hope you have enjoyed I hope you have enjoyed the trip!

I hope you have enjoyed the trip!

Page 60: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

BibliographyBibliographyAilia Athena (2007) Women in Greek Myths, http://www.paleothea.com/index.htmlBarzun J. (1988) Multiple Choice flunks out. New York Times, October 11. Cathell DW. (1890) Book on the physician himself. FA Davis, Philadelphia and London.Davis DA MA Thomson AD Oxman and RB Haynes (1995) Changing physician

Ailia Athena (2007) Women in Greek Myths, http://www.paleothea.com/index.htmlBarzun J. (1988) Multiple Choice flunks out. New York Times, October 11. Cathell DW. (1890) Book on the physician himself. FA Davis, Philadelphia and London.Davis DA MA Thomson AD Oxman and RB Haynes (1995) Changing physician Davis DA, MA Thomson, AD Oxman, and RB Haynes. (1995). Changing physician

performance: A systematic review of the effect of continuing medical education strategies. JAMA274(9) (6 Sept.): 700–5.

Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. (2006) Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA.. 296(9):1094-102.

Davis DA, MA Thomson, AD Oxman, and RB Haynes. (1995). Changing physician performance: A systematic review of the effect of continuing medical education strategies. JAMA274(9) (6 Sept.): 700–5.

Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. (2006) Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA.. 296(9):1094-102.y ( )

Eva KW. (2003) On the generality of specificity. Medical Education. 37, 587-588.Eva KW, Cunnington JP, Reiter HI, Keane DR and Norman GR. (2004) How can I know what

I don’t know? Advances in Health Sciences Education. 9, 211-24.Eva KW and G. Regehr. (2005). Self-assessment in the health professions: A reformulation

and research agenda Academic Medicine 80(10 Suppl): S46–54

y ( )Eva KW. (2003) On the generality of specificity. Medical Education. 37, 587-588.Eva KW, Cunnington JP, Reiter HI, Keane DR and Norman GR. (2004) How can I know what

I don’t know? Advances in Health Sciences Education. 9, 211-24.Eva KW and G. Regehr. (2005). Self-assessment in the health professions: A reformulation

and research agenda Academic Medicine 80(10 Suppl): S46–54and research agenda. Academic Medicine 80(10 Suppl): S46 54.Eva K and Regehr G. (2007). Knowing when to look it up: A new conception of self-

assessment ability. Academic Medicine, 82(10) October 2007 Supplement, In press.Foucault M. (1969) The Archaeology of Knowledge and the Discourse on Language.

Pantheon Books: New York.Foucault M (1965) Madness and Civilization; A History of Insanity in the Age of Reason

and research agenda. Academic Medicine 80(10 Suppl): S46 54.Eva K and Regehr G. (2007). Knowing when to look it up: A new conception of self-

assessment ability. Academic Medicine, 82(10) October 2007 Supplement, In press.Foucault M. (1969) The Archaeology of Knowledge and the Discourse on Language.

Pantheon Books: New York.Foucault M (1965) Madness and Civilization; A History of Insanity in the Age of Reason Foucault M. (1965) Madness and Civilization; A History of Insanity in the Age of Reason.

Random House: New York, USA.Foucault, M. (1995). Discipline and punish: The birth of the prison. (A. M. Sheridan,

Trans.). New York: Vintage Books. (Original work published in 1975).Hanna M and Fins JJ. (2006) Why simulation training ought to be complemented by

experiential and humanist learning Academic Medicine 81(3):265-270

Foucault M. (1965) Madness and Civilization; A History of Insanity in the Age of Reason. Random House: New York, USA.

Foucault, M. (1995). Discipline and punish: The birth of the prison. (A. M. Sheridan, Trans.). New York: Vintage Books. (Original work published in 1975).

Hanna M and Fins JJ. (2006) Why simulation training ought to be complemented by experiential and humanist learning Academic Medicine 81(3):265-270experiential and humanist learning. Academic Medicine. 81(3):265 270.experiential and humanist learning. Academic Medicine. 81(3):265 270.

Page 61: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Harden RM and Gleason FA. (1979) Assessment of clinical competence using an objective structured clinical examination. Medical Education. 13:41-47.

Harden RM and Gleason FA. (1979) Assessment of clinical competence using an objective structured clinical examination. Medical Education. 13:41-47.

Harden R. (2005) Interview. Recorded at Dundee Scotland. January 2005. Unpublished.

Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. (1999) OSCE checklists do not capture increasing levels of expertise. A d i M di i 74(10) 64 69

Harden R. (2005) Interview. Recorded at Dundee Scotland. January 2005. Unpublished.

Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. (1999) OSCE checklists do not capture increasing levels of expertise. A d i M di i 74(10) 64 69 Academic Medicine, 74(10):64-69

Hodges B, Regehr G, Martin D. (2001) Difficulties in recognizing one’s own incompetence: Doctors who are unskilled and unaware of it. Academic Medicine. 76(10 Suppl):S87-9.

Academic Medicine, 74(10):64-69 Hodges B, Regehr G, Martin D. (2001) Difficulties in recognizing

one’s own incompetence: Doctors who are unskilled and unaware of it. Academic Medicine. 76(10 Suppl):S87-9.

Hodges B. (2003a) OSCE! Variations on a theme by Harden. Medical Education. 37 (12):1134-1140.

Hodges B. (2003b) Validity and the OSCE. Medical Teacher. 25(3), 250-254.

Hodges B. (2003a) OSCE! Variations on a theme by Harden. Medical Education. 37 (12):1134-1140.

Hodges B. (2003b) Validity and the OSCE. Medical Teacher. 25(3), 250-254.

Hodges B. (2004) Medical student bodies and the pedagogy of self-reflection, self-assessment and self-regulation. Journal of Curriculum Theory. 20(2), 41-51.

Hodges B. (2005a) The birth and world-wide adoption of the OSCE: A d d i hi t i l t d A i ti f M di l

Hodges B. (2004) Medical student bodies and the pedagogy of self-reflection, self-assessment and self-regulation. Journal of Curriculum Theory. 20(2), 41-51.

Hodges B. (2005a) The birth and world-wide adoption of the OSCE: A d d i hi t i l t d A i ti f M di l An advanced socio-historical study. Association for Medical Education in Europe, Amsterdam.

Hodges B. (2005b) The many and conflicting histories of medical education in Canada and the United States: An introduction to the paradigm wars Medical Education 39:613 621

An advanced socio-historical study. Association for Medical Education in Europe, Amsterdam.

Hodges B. (2005b) The many and conflicting histories of medical education in Canada and the United States: An introduction to the paradigm wars Medical Education 39:613 621paradigm wars. Medical Education. 39:613-621.paradigm wars. Medical Education. 39:613-621.

Page 62: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Kasper DL (Ed). (2005) Harrison's principles of internal medicine.16th ed. New York: McGraw-Hill.

Miller G (1990) The Assessment of Clinical Skills/Competence/Performance

Kasper DL (Ed). (2005) Harrison's principles of internal medicine.16th ed. New York: McGraw-Hill.

Miller G (1990) The Assessment of Clinical Skills/Competence/Performance Miller G. (1990) The Assessment of Clinical Skills/Competence/Performance. Academic Medicine. 65(9):S63-S67.

Moulton CA, Regehr G, Mylopolous M &MacRae H. (2007) Slowing down when you should: A new model of expert judgment. Academic Medicine, 82(10), October Supplment, In press.

Miller G. (1990) The Assessment of Clinical Skills/Competence/Performance. Academic Medicine. 65(9):S63-S67.

Moulton CA, Regehr G, Mylopolous M &MacRae H. (2007) Slowing down when you should: A new model of expert judgment. Academic Medicine, 82(10), October Supplment, In press.

Nelson S and Purkis ME. (2004) Mandatory reflection: the Canadian reconstitution of the competence nurse. Nursing Inquiry. 11(4):247-257.

Norman GR, van der Vleuten CP, De Graaff E. (1991) Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Medical Education. 25 (2):119-26.

Nelson S and Purkis ME. (2004) Mandatory reflection: the Canadian reconstitution of the competence nurse. Nursing Inquiry. 11(4):247-257.

Norman GR, van der Vleuten CP, De Graaff E. (1991) Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability. Medical Education. 25 (2):119-26.Education. 25 (2):119 26.

Norman GR. (1999). The adult learner: A mythical species. Academic Medicine 74(8): 886–9.

Norman G. (2005) Editorial--checklists vs. ratings, the illusion of objectivity, the demise of skills and the debasement of evidence. Adv Health Sci Educ Theory Pract 10 (1):1 3

Education. 25 (2):119 26.Norman GR. (1999). The adult learner: A mythical species. Academic

Medicine 74(8): 886–9.Norman G. (2005) Editorial--checklists vs. ratings, the illusion of objectivity,

the demise of skills and the debasement of evidence. Adv Health Sci Educ Theory Pract 10 (1):1 3 Theory Pract. 10 (1):1-3

Ontario Ministry of Education. (1980) Issues and Directions. Government of Ontario, Toronto.

Royal College of Physicians and Surgeons of Canada (RCPSC). (2006) Maintenance of Competence Program Guide.

Theory Pract. 10 (1):1-3 Ontario Ministry of Education. (1980) Issues and Directions. Government of

Ontario, Toronto.Royal College of Physicians and Surgeons of Canada (RCPSC). (2006)

Maintenance of Competence Program Guide. a te a ce o Co pete ce og a Gu dehttp://rcpsc.medical.org/main_e.php. Accessed September 1, 2006.

Schon DA. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass.

a te a ce o Co pete ce og a Gu dehttp://rcpsc.medical.org/main_e.php. Accessed September 1, 2006.

Schon DA. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass.

Page 63: Between Scylla and Charybdis: Navigating the discourses of ......language of Taylor’s Today, we have reached new heights in the science of producing and testing medical students.

Segouin C and Hodges B. (2005) Educating Physicians in France and Canada: Are the differences based on evidence

Segouin C and Hodges B. (2005) Educating Physicians in France and Canada: Are the differences based on evidence or history? Medical Education 39:1205-1212.

Shorter, E. (1985). Doctors and their patients: A social history. New York: Simon and Schuster.

Schurwirth LWT and van der Vleuten CPM (2006) A plea for

or history? Medical Education 39:1205-1212.Shorter, E. (1985). Doctors and their patients: A social

history. New York: Simon and Schuster.Schurwirth LWT and van der Vleuten CPM (2006) A plea for Schurwirth LWT and van der Vleuten CPM. (2006) A plea for

new psychometric models in educational assessment. Medical Education. 40:296-300

van de Vleuten CPM. (2000) Validity of final examinations in d d t di l t i i B iti h M di l J l

Schurwirth LWT and van der Vleuten CPM. (2006) A plea for new psychometric models in educational assessment. Medical Education. 40:296-300

van de Vleuten CPM. (2000) Validity of final examinations in d d t di l t i i B iti h M di l J l undergraduate medical training. British Medical Journal.

321:1`217-1219.van der Vleuten CPM, Norman GR, De Graaff E. (1991) Pitfalls

in the pursuit of objectivity: issues of reliability. Medical

undergraduate medical training. British Medical Journal. 321:1`217-1219.

van der Vleuten CPM, Norman GR, De Graaff E. (1991) Pitfalls in the pursuit of objectivity: issues of reliability. Medical p j y yEducation. 25(2):110-8.

Wallace P. (1997) Following the treads of innovation: The history of standardized patients in medical education. Caduceus. Vol. 13(2), 5-28.

p j y yEducation. 25(2):110-8.

Wallace P. (1997) Following the treads of innovation: The history of standardized patients in medical education. Caduceus. Vol. 13(2), 5-28.Caduceus. Vol. 13(2), 5 28.Caduceus. Vol. 13(2), 5 28.


Recommended