7/28/2019 6 Mitchell 2012 APDVS Milestones

1/27

Erica L. Mitchell, M.D.

Associate Professor Surgery

Howto

Determine

Appropriate

Milestones:EducationalPerspective

7/28/2019 6 Mitchell 2012 APDVS Milestones

2/27

Overview

AimsoftheACGMEmilestoneproject

Competencieswe

need

to

assess

Toolscurrentlyusedtomeasurecompetence

Howwe

can

develop

milestones

for

Vascular

Surgery

7/28/2019 6 Mitchell 2012 APDVS Milestones

3/27

ACGMEMilestone

Project

Processof

educationNext

Accreditation

Cycle(NAS) Educationaloutcomes

7/28/2019 6 Mitchell 2012 APDVS Milestones

4/27

Whatdo

we

need

to

assess?

medicalknowledge intellectual

practical

patientcare

professionalism communicationandinterpersonalskills practice

based

learning

and

improvement

systemsbasedpractice

Knowledge,skills&attitudes

7/28/2019 6 Mitchell 2012 APDVS Milestones

5/27

Endproduct

of

Vascular

Surgical

Training

Workingknowledgeofvasculardiseases Coordinatesthecareofpatients

Performs

appropriate

procedures

safely Trustworthyandempathetic Leadershipanddecisionmakingability

Communication

skills Maintainsscholarship Informedonevidencebasedpractices

CompetentVascularSurgeon

7/28/2019 6 Mitchell 2012 APDVS Milestones

6/27

Howcanweassessthedomainsof

competence?

Theassessmentofclinicalskills/competence/performance

Professionalauthentici

ty

behavior

cognition

novice

expertPerformanceintegrated

intopractice

Demonstrationof

learning

Interpretation/

application

of

knowledge

Factgathering

Knowledge,skills&attitude MillerGE.AcadMed1990

7/28/2019 6 Mitchell 2012 APDVS Milestones

7/27

Howcanweassessthedomainsof

competence?

Writtenassessments

Oralexaminations

Simulation

Workplacebasedassessments

Multisource(360

degree)

feedback

(MSF)

Typesofassessmentcommonlyused

RMEpstein

Assessmentin

Medical

Education

NewEnglJMed2007356:387396

7/28/2019 6 Mitchell 2012 APDVS Milestones

8/27

Assessmentof

competence

VSITE&ABSqualifyingexam

Multichoicequestions Singlebestanswer

Knowledge&

reasoning

ability

Writtenassessments

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X

Limitations:

cannot

measure

behavioral

and

non

technical

skills

7/28/2019 6 Mitchell 2012 APDVS Milestones

9/27

Assessmentof

competence

Mockorals&ABScertifyingexam

Casespecific Unstructuredfacetofacesession

Knowledge,skills

and

judgment

Oralexaminations

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X X X X

Limitations:

cannot

reliably

predict

performance

across

range

of

cases

andcannotassesstechnicalskills

7/28/2019 6 Mitchell 2012 APDVS Milestones

10/27

Assessmentof

competence

Miniclinicalevaluationexercise(miniCEX)

Directobservationofproceduralskills(DOPS)

Procedurebasedassessment(PBA)

Nontechnical

skills

for

surgeons

(NOTSS

Multisource(360degree)feedback(MSF)

Workplacebasedassessments

Formativeassessment

s Assesswhatthetraineeactuallydoesintheworkplace

Feedback Limitations:supervisedsetting

NorciniJ

Workplace

based

assessment

as

an

educational

toolMedTeacher200729:855871

7/28/2019 6 Mitchell 2012 APDVS Milestones

11/27

Assessmentof

competence

Directobservationofatraineeinteractionwithapatientintheclinicalsetting

Casespecific

Miniclinicalevaluationexercise(miniCEX)

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X X X X

Limitations: maynotreliablypredictperformanceacrossrangeofcases

7/28/2019 6 Mitchell 2012 APDVS Milestones

12/27

Effective2012theABSisrequiring

2clinicalassessments/resident

Canbeusedinanumberof

differentclinicalsettings.

Potentiallyused

to

evaluate

residentpatientinteractions:

Dialysisaccess

Claudication

Criticallimb

ischemia

Carotidstenosis

Aneurysmaldisease

7/28/2019 6 Mitchell 2012 APDVS Milestones

13/27

Howcanweassessthedomainsof

operativecompetence?

Surrogatemeasures

Simulation

Workplace

based

assessments

Objectivestructuredassessmentoftechnicalskills(OSATS)

Procedurebasedassessment(PBA)

Nontechnicalskillsforsurgeons(NOTSS)

Typesofassessmentcommonlyused

7/28/2019 6 Mitchell 2012 APDVS Milestones

14/27

Assessmentof

technical

competence

Operativespeed

Correlatespoorlywithpatientoutcome

Casenumbers

lacksvalidityandreliabilityasamethodofassessingtraineessurgicalskill

Surrogatemeasures

Assessmentof

operative

competence

7/28/2019 6 Mitchell 2012 APDVS Milestones

15/27

Assessmentoftechnicalskillinaskillslab:

Procedurespecific

Proceduralskillorsequencing

Simulation

Limitations:needtosamplewidelyandconsiderbothcontextandconstruct,

cangame

the

simulator,

ceiling

effect

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X

Assessmentof

operative

competence

7/28/2019 6 Mitchell 2012 APDVS Milestones

16/27

Assessmentof

operative

competence

Checklists

Procedurespecific

Captureoccurrenceofparticularbehavior

Perceivedtobemoreobjective

Scoresslightly

more

reliable

Goodforassessingtrainees

Globalratingscores

Itemscommon

to

any

procedure

Capturethequalityofaperformance

Perceivedtobemoresubjective

Scoresslightly

more

valid

Goodforassessingexperts

Objectivestructuredassessmentoftechnicalskills

(OSATS)

7/28/2019 6 Mitchell 2012 APDVS Milestones

17/27

Traineesobservedperformingaprocedureonarealpatient

Procedurespecific

Directobservationofproceduralskills(DOPS)

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X X X X

Limitations: maynotreliablypredictperformanceacrossrangeofcases

Assessmentof

operative

competence

7/28/2019 6 Mitchell 2012 APDVS Milestones

18/27

Effective2012theABSisrequiring

2operativeassessments/resident

Potentiallybeusedtoevaluate

residentsperformingroutine

coresurgical orminorvascular

procedures:

I&Dabscess

Radialarterialline

Centralvenouscatheter

Temporarydialysis

catheter

7/28/2019 6 Mitchell 2012 APDVS Milestones

19/27

AssessmentoftechnicalskillintheOR:

Procedurespecific

OSATS

Knowledge,skills

&

attitudes

Procedurebasedassessment(PBA)

Limitations:needtosamplewidelyandconsiderbothcontextandconstruct

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X X X X

Assessmentof

operative

competence

7/28/2019 6 Mitchell 2012 APDVS Milestones

20/27

Assessmentof

operative

competence

AssessmentofnontechnicalskillintheOR:

Situationawareness

Decisionmaking

Communicationand

teamwork

Leadership

Nontechnicalskillsforsurgeons(NOTSS)

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X X X X

Limitations:needtosamplewidelyandconsiderbothcontextandconstruct

7/28/2019 6 Mitchell 2012 APDVS Milestones

21/27

Assessmentof

competence

Peerassessment(untrainedraters) Teammembers,physicianextenders

Referringservices,anesthesiastaff

Clinic

staff,

ward

clerks Self

Multisource(360)feedback(MSF)

Medicalknowledge Patientcare Professionalism Communication&

interpersonal

skills

Practicebased

learning&

improvement

Systemsbased

practiceIntellectual Practical

X X X X X X X

Limitations:needmultipleassessments

7/28/2019 6 Mitchell 2012 APDVS Milestones

22/27

Makingjudgmentsaboutthe

performanceofpeershasformedthe

basisof

the

referral

process

in

medicine

andtheotherprofessionsforcenturies.

NorciniJJ

Peerassessmentofcompetence

MedEducation200337:539543

7/28/2019 6 Mitchell 2012 APDVS Milestones

23/27

Developingmilestones

for

vascular

surgery

Definemarkersofachievementacrossthecontinuumofvascularsurgicaleducation

Blueprintthese

to

the

entire

curriculum

Knowledge,skills&attitudes

6corecompetencies

Coresurgical&specialtyspecificskills

Consensus:Delphiprocessusingspecialtyexperts

Systematicapproach

7/28/2019 6 Mitchell 2012 APDVS Milestones

24/27

Whatwe

need

to

do

as

aspecialty

Identifyorcreatetoolsthatcanbeembraced asmeaningfulmeasuresofperformance Acrossthecontinuumofvascularsurgicaltraining

Assessacross

multiple

contexts:

Clinical,simulatedandORsetting Clinical&operativecompetencies

Assessmentcontent:taskanalysisandDelphiprocessesusingspecialtyexperts

Systematicapproach

7/28/2019 6 Mitchell 2012 APDVS Milestones

25/27

Whatwe

need

to

do

as

aspecialty

Needtoensureassessmentsare: Reliable Valid

Acceptable Costeffective Educationalimpact

Vander

Vleutens

utility

index

Utility=Validityw xreliabilityw xacceptabilityw xeducationalimpactw xcostw W=weighted

Systematicapproach

7/28/2019 6 Mitchell 2012 APDVS Milestones

26/27

Assessment

Standardizedtesting VSITE,ABSQE

Operativeandcaselogdata

Assessmentof

patient

care

Clinicalsetting(miniCEX)

ClinicorOR(DOPS)

Operativesetting(PBA)

360 evaluations(MSF)

Compliancewithqualityperformanceindicators

Competencyassessed

medicalknowledge intellectual

practical

patientcare

professionalism

communicationandinterpersonalskills

practicebasedlearningandimprovement

systemsbasedpractice

Evidencedocumenting

trainees

learning

Learningportfolio

7/28/2019 6 Mitchell 2012 APDVS Milestones

27/27

Endproduct

of

Vascular

Surgical

Training

Workingknowledge

of

vascular

diseases

Coordinatesthecareofpatients Performsappropriateproceduressafely Trustworthyandempathetic Leadershipanddecisionmakingability Communicationskills Maintainsscholarship Informedonevidencebasedpractices

CompetentVascularSurgeon