The Challenges in Surgical Education:The Challenges in Surgical Education:Autonomy, Assessment, & AccountabilityAutonomy, Assessment, & Accountability
Edward D. Verrier, MDEdward D. Verrier, MD
Surgical Director of EducationSurgical Director of Education
Joint Council on Thoracic Surgery EducationJoint Council on Thoracic Surgery EducationJoint Council on Thoracic Surgery EducationJoint Council on Thoracic Surgery Education
K. Alvin K. Alvin MerendinoMerendino Professor of Cardiovascular SurgeryProfessor of Cardiovascular Surgery
University of WashingtonUniversity of Washington
•• To vision the future one must acknowledge the To vision the future one must acknowledge the
pastpast
SURGICAL EDUCATION
United States
•• Flexner report 1910Flexner report 1910
–– Carnegie Foundation CommissionCarnegie Foundation Commission
–– “Medical Education in US and Canada”“Medical Education in US and Canada”
–– Embraced Johns Hopkins ModelEmbraced Johns Hopkins Model
•• Recommendations:Recommendations:
Medical School Education Structure
•• Recommendations:Recommendations:
–– Prerequisite EducationPrerequisite Education
•• High school, University (>2 years)High school, University (>2 years)
–– Medical School Requisite Education Medical School Requisite Education
•• 4 years4 years
•• 2 basic science, 2 clinical clerkship2 basic science, 2 clinical clerkship
Post graduate resident training essentialPost graduate resident training essential
•• Apprenticeship model gone except in SurgeryApprenticeship model gone except in Surgery
Remained essentially unchanged for last century
•• William William Halsted,MDHalsted,MD (1903)(1903)–– Formalization of apprenticeship modelFormalization of apprenticeship model
–– Science foundationScience foundation
–– Graduated responsibilityGraduated responsibility
–– Defined structureDefined structure
–– Standardization of trainingStandardization of training
–– Pyramidal system of surgical educationPyramidal system of surgical education
•• William Osler, MD (1908)William Osler, MD (1908)
Early Surgical Education Structure
•• William Osler, MD (1908)William Osler, MD (1908)–– Residents at different levels working togetherResidents at different levels working together
–– Supervision / mentorship by competent facultySupervision / mentorship by competent faculty
–– Bedside teachingBedside teaching
•• Edward Churchill, MD (1933Edward Churchill, MD (1933))–– Eliminate pyramidal systemEliminate pyramidal system
–– Recommended rectangular system of surgical educationRecommended rectangular system of surgical education
–– Mandated ultimately by ACGMEMandated ultimately by ACGME
Robert Hinckley 1853-1945
MGH Ether Dome
First ether anesthesia
Remained essentially unchanged for last century
We believe that in the future, We believe that in the future,
expertise rather than experience, expertise rather than experience,
will underlie competency will underlie competency -- based based will underlie competency will underlie competency -- based based
practice and....certificationpractice and....certification
Aggarwal & Darzi, 2066)
Competency Based Medical Education Competency Based Medical Education
(CBME)(CBME)
•• ....competency based education is an approach to ....competency based education is an approach to
preparing physicians for practice that is preparing physicians for practice that is
fundamentally orientated to graduate outcome fundamentally orientated to graduate outcome
abilities and organized around competencies abilities and organized around competencies abilities and organized around competencies abilities and organized around competencies
derived from an analysis of societal and patient derived from an analysis of societal and patient
needs. It deemphasizes time based training and needs. It deemphasizes time based training and
promises greater accountability, flexibility, and promises greater accountability, flexibility, and
learnerlearner-- centeredness.centeredness.
International CBME Collaborators, 2011
•• Competencies: “Specific knowledge, skills, behaviors and Competencies: “Specific knowledge, skills, behaviors and
attitudes and the appropriate educational experiences attitudes and the appropriate educational experiences
required of residents to complete GME programs.”required of residents to complete GME programs.”
•• GME Competencies:GME Competencies:
–– Patient CarePatient Care
–– Medical KnowledgeMedical Knowledge
What then is competency ?What then is competency ?
ACGME: ~1990 “Outcomes Project”ACGME: ~1990 “Outcomes Project”
–– Medical KnowledgeMedical Knowledge
–– ProfessionalismProfessionalism
–– Interpersonal and Communication SkillsInterpersonal and Communication Skills
–– Practice Practice -- based Learning and Improvementbased Learning and Improvement
–– Systems Systems -- based Practicebased Practice
•• Evolving concepts about competencyEvolving concepts about competency
–– Technical skill ?Technical skill ?
–– Life Long (Maintenance of Certification)Life Long (Maintenance of Certification)
–– Competency Competency vsvs ExpertiseExpertise
–– Milestone Project 2012Milestone Project 2012
The Challenges in Surgical EducationThe Challenges in Surgical Education
AutonomyAutonomy
•• Trainee Trainee -- Trainer DisconnectTrainer Disconnect
–– Trainee:Trainee:
•• Exposure / Experience = CompetencyExposure / Experience = Competency
•• Facts / Data = KnowledgeFacts / Data = Knowledge
–– Google GenerationGoogle Generation
–– Information OverloadInformation Overload–– Information OverloadInformation Overload
•• Adult learnersAdult learners
•• Generational / Gender IssuesGenerational / Gender Issues
•• Work Hour PrioritiesWork Hour Priorities
–– Trainer:Trainer:
The Generational DivideThe Generational Divide
TraditionalistsTraditionalists Baby Baby
BoomersBoomers
Generation Generation
XX
MillennialsMillennials
BirthBirth
YearsYears
Before 1945Before 1945 19461946--19641964 19651965--19801980 19791979--20062006
Business Business QualityQuality Long hoursLong hours ProductivityProductivity ContributionContributionBusiness Business
FocusFocus
QualityQuality Long hoursLong hours ProductivityProductivity ContributionContribution
MotivatorMotivator SecuritySecurity MoneyMoney Time offTime off Time offTime off
Company Company
LoyaltyLoyalty
HighestHighest HighHigh LowLow LowLow
Money isMoney is LivelihoodLivelihood Status Status
symbolsymbol
Means to an Means to an
endend
Today’s Today’s
payoffpayoff
ValueValue Family/CommunityFamily/Community SuccessSuccess TimeTime IndividualityIndividuality
Gender Related Gender Related IIssuesssues
•• “Men are from MarsJ. “Men are from MarsJ.
Women are from Venus”Women are from Venus”11
–– Different reward perspectivesDifferent reward perspectives
–– Different responses to stressDifferent responses to stress–– Different responses to stressDifferent responses to stress
–– Biological / hormonal differencesBiological / hormonal differences
1John Grey, PhD 1992
ANDROLOGYANDROLOGY
The Study of Adult LearningThe Study of Adult Learning
Critical AssumptionsCritical Assumptions
•• The “adult” learnerJJThe “adult” learnerJJ
–– is self is self –– directingJrelatedness to othersdirectingJrelatedness to others
–– enters an educational situation with a great deal of enters an educational situation with a great deal of
experienceexperienceexperienceexperience
–– learn when they perceive a need to knowlearn when they perceive a need to know
–– mmotivated to learn after they experience a need in otivated to learn after they experience a need in
their life situationJproblem focus, task centeredtheir life situationJproblem focus, task centered
–– motivated to learn because of internal factors, not motivated to learn because of internal factors, not
external pressureexternal pressure
•• PROCESS > CONTENTPROCESS > CONTENT
“The Adult Learner;” Knowles et al
Learning StylesLearning Styles
•• CognitiveCognitive
•• AffectiveAffective
•• PhysiologicalPhysiological•• PhysiologicalPhysiological
•• Interpersonal Interpersonal
The Challenges in Surgical EducationThe Challenges in Surgical Education
AutonomyAutonomy
•• Trainee Trainee -- Trainer DisconnectTrainer Disconnect
–– Trainer:Trainer:
•• No Educational TrainingNo Educational Training
•• Resistant to “Change” by definitionResistant to “Change” by definition
•• Priority issuesPriority issues
•• May face Promotion IssuesMay face Promotion Issues•• May face Promotion IssuesMay face Promotion Issues
•• Many with Huge Many with Huge CClinical loadslinical loads
•• Large administrative LoadLarge administrative Load
–– Trainee:Trainee:
•• Dewey’s “Keys Concepts” of Adult TeachingDewey’s “Keys Concepts” of Adult Teaching
–– ExperienceExperience
•• Process not resultProcess not result
–– DemocracyDemocracy
•• No room for autocracy / harshnessNo room for autocracy / harshness
–– ContinuityContinuity
Critical Concepts in Adult TeachingJ..a Science
–– ContinuityContinuity
•• Logical growth Logical growth
–– InteractionInteraction
•• Bandura’s Extension of Dewey’s Concepts Bandura’s Extension of Dewey’s Concepts
–– Teaching through Inquiry / DiscoveryTeaching through Inquiry / Discovery
•• Self Self –– directed learningdirected learning
•• ProblemProblem--solving learningsolving learning
–– Teaching Through ModelingTeaching Through Modeling
•• Imitation / Identification / ModelingImitation / Identification / Modeling
•• Expertise: Expertise: Power of knowledge and preparationPower of knowledge and preparation
–– Knows something beneficial, knows it well, is prepared to conveyKnows something beneficial, knows it well, is prepared to convey
•• Empathy: Empathy: Power of Understanding and Power of Understanding and Consideration Consideration
–– Realistic understanding of learner needsRealistic understanding of learner needs
–– Adapts to learner’s level of experience and skill developmentAdapts to learner’s level of experience and skill development
Characteristics and Skills of Motivating
Instructors
–– Adapts to learner’s level of experience and skill developmentAdapts to learner’s level of experience and skill development
•• Enthusiasm: Enthusiasm: Power of Commitment and AnimationPower of Commitment and Animation
–– Cares about and values what is being taughtCares about and values what is being taught
–– Appropriate degrees of emotion, and energyAppropriate degrees of emotion, and energy
•• Clarity: Clarity: Power of Language and OrganizationPower of Language and Organization
–– Can be understood and followed by learnersCan be understood and followed by learners
–– Has ability to adapt to second presentationHas ability to adapt to second presentation
•• Timely repetitionTimely repetition
•• Formative feedbackFormative feedback
–– personalpersonal
•• ChoiceChoice
Adult Adult EEducation ducation
Learning BackboneLearning Backbone
–– contentcontent
–– timetime
–– levellevel
–– depthdepth
•• MentorshipMentorship
•• LLearning objectivesearning objectives
The Challenge of Surgical EducationThe Challenge of Surgical Education
AssessmentAssessment
•• Challenge in all medical specialtiesChallenge in all medical specialties
•• Relationship to “Competency”Relationship to “Competency”
•• Part of all Modern Part of all Modern EEducational ducational CCurricular designurricular design•• Part of all Modern Part of all Modern EEducational ducational CCurricular designurricular design
–– Learning Objectives, Curriculum design, Assessment tools, Learning Objectives, Curriculum design, Assessment tools,
Validation, Adoption to certifyValidation, Adoption to certify
•• Learning Learning MManagement Systems (LMS)anagement Systems (LMS)
Purpose of Assessment?Purpose of Assessment?
•• To aid learning through constructive feedback:To aid learning through constructive feedback:
–– Assessment Assessment forfor Learning (formative)Learning (formative)
–– Must be done frequently e.g. WBAMust be done frequently e.g. WBA–– Must be done frequently e.g. WBAMust be done frequently e.g. WBA
•• To check knowledge or skill has been learned:To check knowledge or skill has been learned:
–– Assessment Assessment ofof Learning (summative)Learning (summative)
–– Done infrequently e.g. ExamsDone infrequently e.g. Exams (MSF?)(MSF?)
Classic Surgical Training ModelClassic Surgical Training Model
•• Classic Apprenticeship & ExaminationClassic Apprenticeship & Examination
–– Time basedTime based
–– Minimal training of trainersMinimal training of trainers
–– Formative evaluations: Formative evaluations:
•• In training examsIn training exams
•• Faculty or Program Director evaluationsFaculty or Program Director evaluations•• Faculty or Program Director evaluationsFaculty or Program Director evaluations
•• Little structureLittle structure
–– Summative assessmentSummative assessment
•• Case LogsCase Logs
•• Program Director recommendation of competencyProgram Director recommendation of competency
•• Qualifying examination (cognitive)Qualifying examination (cognitive)
•• Certifying examinations(affective)Certifying examinations(affective)
•• No technical examNo technical exam
•• Do our current methods of assessment guarantee Do our current methods of assessment guarantee
competency?competency?
–– Case NumbersCase Numbers–– High variabilityHigh variability
–– Loose definitions / Little auditLoose definitions / Little audit
–– No established benchmarkNo established benchmark
–– ObservationObservation
Skill AssessmentSkill Assessment
–– ObservationObservation–– InterInter-- and intraand intra--rater variabilityrater variability
–– Case variabilityCase variability
–– No uniform instrumentNo uniform instrument
–– No established benchmarkNo established benchmark
–– Now splitting casesNow splitting cases
–– Summative exams (Qualifying / Certifying)Summative exams (Qualifying / Certifying)–– Little correlation with later success as surgeonLittle correlation with later success as surgeon
–– Remains our benchmarkRemains our benchmark
Accuracy of medical staff assessment of trainee’s
operative performance. Paisley AM et al, Med Teach
2005.
•• What has been tried?What has been tried?•• Simulation testingSimulation testing
–– OSATSOSATS
–– Virtual Simulators (MISTVirtual Simulators (MIST--VR)VR)
•• Structured observationStructured observation
Skill AssessmentSkill Assessment
•• Structured observationStructured observation
–– Live observationLive observation
–– Recorded observationRecorded observation
•• Error detectionError detection
•• Motion trackingMotion tracking
•• Procedure timeProcedure time
Comparing the psychometric properties of
Checklists and global rating scales for assessing
Performance on an OSCE-format examination.
Regehr G. Acad Med 1998
Workplace based
assessment in surgical
training : the UK experience
(so far)
Chris Munsch
Cardiac Surgeon, Leeds UK
Chair, Joint Committee on Surgical Training (2007-2011)
Assessing competencies in the new curriculum:
traditional assessment tools
CanMeds MCQs Clinicals Vivas
Medical expert
Communicator
Collaborator
Manager
Health Advocate
Scholar
Professional
Workplace based assessment
tools� Case Based Discussion (CBD)
� Mini clinical evaluation exercise (miniCEX)
� Multisource Feedback (MSF)
� Direct observation of procedural skills (DOPS)
� Procedure based assessment (PBA)
� Linked to Educational Supervisors Report
� Feeds into Annual Review of Competence Progression
CanMeds Curriculum
domains
CBD Mini
CEX
MSF DOPS PBA AES
report
Exams ARCP
Medical expert Knowledge
Clinical Skills
Technical Skills
Communicator Clinical Skills
(professional and
generic)
Collaborator Technical Skills
(professional and
generic)
Manager Knowledge
(professional and
generic)
Health advocate Knowledge
(professional and
generic)
Scholar Knowledge
Professional Clinical Skills
Technical Skills
(professional and
generic)
Faculty Development� Training and Assessment in Practice (TAIP) course
� Webcasts on ISCP website
� Regional delivery of training sessions
� Leader as educator programme
Effective Assessment in CBMEEffective Assessment in CBME
•• Continuous and FrequentContinuous and Frequent
–– Robust onRobust on--going formative feedbackgoing formative feedback
•• Criterion basedCriterion based
–– Milestones or benchmarksMilestones or benchmarks
•• Emphasis on what Trainee will ultimately doEmphasis on what Trainee will ultimately do
–– Robust work Robust work -- based assessmentsbased assessments
•• Quality assessment toolsQuality assessment tools
•• Qualitative measures and methodsQualitative measures and methods
–– Judging portfoliosJudging portfolios
•• Emphasize group wisdomEmphasize group wisdom
–– Activate trainee involvementActivate trainee involvement
The Challenge of Surgical EducationThe Challenge of Surgical Education
AccountabilityAccountability
•• Who will hold us accountable for outcomes, Who will hold us accountable for outcomes,
performance, quality and safety?performance, quality and safety?
–– GovernmentGovernment
–– PayorsPayors–– PayorsPayors
•• Pay for PerformancePay for Performance
–– Medical societiesMedical societies
–– ACGME / Specialty BoardsACGME / Specialty Boards
•• Milestone ProjectMilestone Project
–– Society in generalSociety in general
•• Airline standardsAirline standards
–– OurselvesOurselves
Are We Achieving “Competency” Are We Achieving “Competency”
in Surgical Training in 2011?in Surgical Training in 2011?
Competency
in Surgical Training in 2011?in Surgical Training in 2011?
60
65
70
75
80
85
90
95
100Qualifying examination
Certifying examination
ABS Examination ResultsABS Examination Results
2010
Qualifying exam pass rate:75%
Certifying exam pass rate: 75%
50
55
60
2006 2007 2008 2009 2010
2010
Qualifying exam pass rate: 73%
ABTS Examination ResultsABTS Examination Results
Qualifying exam pass rate: 73%
Certifying exam pass rate: 68%
ExpertiseExpertise
Should we be Striving for Should we be Striving for Should we be Striving for Should we be Striving for
Expertise in Surgical Training in Expertise in Surgical Training in
2012?2012?
•• Must be put into context of evolution of Must be put into context of evolution of
medical educationmedical education
–– Emphasis on competencyJ.expertiseEmphasis on competencyJ.expertise
•• Must meet acceptable outcomes and safety Must meet acceptable outcomes and safety
standardsstandards
Competency
standardsstandards
–– Internally or externally definedInternally or externally defined
•• Must meet societal standardsMust meet societal standards
–– Airline standards Airline standards
Traditional
• Halstedian - “See one, do one, teach one”
• Determined by patient flow
• “Conventional” fixed didactic lectures
• Subjective personal evaluation
• Specific time and place
Surgical Education - Paradigm Shift
• Specific time and place
Next Generation
• Simulation – “Do many, mentored always”
• Each student every variation at own pace
• Interactive, updated (web based) lectures
• Standardized, objective, criterion based evaluation• Formative and Summative
• Continuous at point of clinical care
•• Remains time basedRemains time based
•• Assessment / Competency:Assessment / Competency:–– Case logsCase logs
–– Summative examsSummative exams
–– Program Director recommendationsProgram Director recommendations
The Evolving Structure of Resident Education
–– Program Director recommendationsProgram Director recommendations
•• “Integrated” training algorithms increasing“Integrated” training algorithms increasing
–– Plastics / Vascular / CardiothoracicPlastics / Vascular / Cardiothoracic
•• Duty hours limitations real impact in surgeryDuty hours limitations real impact in surgery
•• Discussions of “Criterion” based trainingDiscussions of “Criterion” based training–– Formative evaluationsFormative evaluations
Can We Measure Competency? Can We Measure Competency?
Do We Measure Do We Measure CCompetency?ompetency?
How Should We Measure Competency?How Should We Measure Competency?
•• Time spent in trainingTime spent in training–– Apprenticeship holdover Apprenticeship holdover
•• Log book Log book –– Case numbers, ABTS determinedCase numbers, ABTS determined–– Case numbers, ABTS determinedCase numbers, ABTS determined
•• Program Director recommendationProgram Director recommendation–– Can they say no?Can they say no?
•• Summative examsSummative exams–– Qualifying / Certifying examsQualifying / Certifying exams
•• Formative feedbackFormative feedback–– VariableVariable
Goals and Purposes for Learning
Individual and Situational Differences
Core Adult Learning Principles
1: Learner’s Need to Know
Why, What, How
2: Self Concept of the Learner
Autonomous
Self - Directing
3: Prior Experience of the Learner
Socie
tal g
row
th
Situ
atio
nal D
iffere
nces
Institu
tional gro
wth
Subje
ct M
att
er D
iffe
rences
Adult Learning
3: Prior Experience of the Learner
Resource
Mental Models
4: Readiness to LearnLife related
Developmental task
5: Orientation to Learn
Problem centered
Contextual
6: Motivation to Learn
Intrinsic Value
Personal payoff
Individual Learner differences
Individual growth
Socie
tal g
row
th
Situ
atio
nal D
iffere
nces
Institu
tional gro
wth
Subje
ct M
att
er D
iffe
rences
Knowles MS et al: The Adult Learner; Elsevier, 2005
There are several purposes to formative assessment:There are several purposes to formative assessment:
to provide feedback for teachers to modify subsequent learning activities to provide feedback for teachers to modify subsequent learning activities
and experiencesand experiences
to identify and remediate group or individual deficiencies;to identify and remediate group or individual deficiencies;
to move focus away from achieving grades and onto learning processes, to move focus away from achieving grades and onto learning processes,
in order to increase self efficacy and reduce the negative impact of in order to increase self efficacy and reduce the negative impact of
extrinsic motivationextrinsic motivationextrinsic motivationextrinsic motivation
frequent, ongoing assessment allows both for finefrequent, ongoing assessment allows both for fine--tuning of instruction tuning of instruction
and student focus on progress.“and student focus on progress.“
Feedback is the central function of formative assessment. It typically Feedback is the central function of formative assessment. It typically
involves a focus on the detailed content of what is being learnt rather involves a focus on the detailed content of what is being learnt rather
than simply a test score or other measurement of how far a student is than simply a test score or other measurement of how far a student is
falling short of the expectedfalling short of the expected