EVALUATING SURGICAL SKILL AND OR PERFORMANCE

Neal E. Seymour, MDBaystate Medical Center

Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab

2010 APDS Annual Meeting, San Antonio, Texas

DisclosuresNo disclosures

Evaluation of Surgical Skill Skills that are uniquely surgical

Evidenced in ORSurgical patient management

Observed during training eventsClinical (e.g. OR)Laboratory

Occurs throughout residencyProgression of skills

Improved Task Performance

0

50

100

150

200

1 3 5 7 9 11 13 15 17

Tota

l Sco

re

Trial Number

VR Laparoscopic Suturing and Knot-Tying

Residents

Experts

Effect of VR Training on Laparoscopic Suturing Performance: Video Analysis

0

10

20

30

40

50

60

70

Pre-VR Post-VR

Video Score

Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9

Effect of VR Training on Laparoscopic Suturing Performance: Time

0

50

100

150

200

250

300

350

400

Pre-VR Post-VR

Time (sec)

Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9

Proficiency-Based Lab Training

p > 0.006*

Skills Transfer: “VR to OR” Study

0

1

2

3

4

5

6

7

8

9

10M

ean

num

ber o

f erro

rs

VR Trained Standard Training

*

Assessment of Performance in OR

Procedure specific instruments GOALS OSATS Analysis of video recorded tasks Differences between investigative

instruments and assessment in frequent, routine use.Limitations on obtrusive methodsResource intensive

Design of New Assessment Tool

User-friendly in routine use, valid, reliable.Opportunity for delivery of feedbackLeverage technology to manage large

volumes of data generated.Generate useful reports on resident

performance.

Resident OR Performance Assessment at Baystate Medical Center

Objective: Develop OR performance assessment tool for routine use

OpRate: Dept. of Surgery and IS project:Multi-item global assessment of skillAttending surgeon raterEase of use = compliance with use.Interfaced to other databases

Evaluated Items

3 preop (preparedness) questionsPatient, technical and disease-specific knowledge

6 open technical skills questionsTissue, instrument handling, knot tying, efficient

planning and ability to function independently 6 laparoscopic technical skills questions 4 point frequency-based responses for

specific technical performance behaviors

Assessment of Residents with OpRateJuly 2004 - June 2006

15 29 21 36 3863

146101

237

113

050

100150200250

Appen

decto

my

Ventra

l Herni

a

Colon R

esec

tion

Cholecy

stecto

my

Inguin

al Hern

ia

Case

Num

ber

2004-5 2005-6

799 OpRate evaluations logged

579 cases reviewed 29 residents

33 attending surgeon evaluators◦ 6 contributed 75% of

assessments

Preop Preparedness

0

20

40

60

80

100

120

140

160

PGY1 PGY2 PGY3 PGY4 PGY5

Training Year

Num

ber o

f Cas

es

3.5

3.6

3.7

3.8

3.9

4

Perfo

rman

ce S

core

Cases

Performance

Cronbach’s α (3 test items) = 0.80

Laparoscopic Appendectomy

Cronbach’s α (4 test items) = 0.80

0

5

10

15

20

25

30

35

PGY1 PGY2 PGY3 PGY4 PGY5Training Year

Num

ber o

f Cas

es

2

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

4

Perfo

rman

ce S

core

Cases

Performance

Laparoscopic Cholecystectomy

Cronbach’s α (4 test items) = 0.80

05

101520253035404550

PGY1 PGY2 PGY3 PGY4 PGY5Training Year

Num

ber o

f Cas

es

22.22.42.62.833.23.43.63.84

Perfo

rman

ce S

core

Cases

Performance

Colon Resection

Laparoscopic (n = 37): Cronbach’s α (4 test items) = 0.82 Open (n = 36): Cronbach’s α (6 test items) = 0.84

0

5

10

15

20

25

30

35

40

PGY1 PGY2 PGY3 PGY4 PGY5Training Year

Num

ber o

f cas

es

2.5

2.7

2.9

3.1

3.3

3.5

3.7

3.9

Perfo

rman

ce S

core

CasesPerformance

VR laparoscopic gall bladder excision (time) vs laparoscopic OR performancep = 0.0066, r2 = 0.4450 (Spearman correlation)

J Surg Res 2009;154:177-178

Conclusions Experience with OpRate confirms predictions of

experience-dependent resident performance increases, but does not validate attending rating behaviors.

Still Required:Systematic rater trainingInter-observer agreementConcurrent validity studies (lab and OR)Mandatory compliance

OpRate Build 2 CQL Server database Edit redundant items Add important items

Bimanual skillsAbility to assistDepth perception in laparoscopic cases

Status of Use Remediation decisions 2 SD below mean for peer group

High likelihood of performance problems Compliance to be augmented

Variable compensation linked to useCompliance monitored and posted monthlyExpectation that residents will advocate for

use

Patient Management and Teamwork Skills

Electrical Burns VTE Septic Shock Hemorrhagic Shock Spinal shock Post-Op MI Anaphylaxis

FPS Curriculum Thoracic trauma Hyperkalemia Intraabdominal

catastrophe Pediatric trauma Pheochromocytoma

Recognition of importance of teamwork orientation to effective clinical careStrong foundation principlesAlignment with growing body of work on “non-

technical skills” Effort to take advantage of simulated patient

care, simulated hospital environments (e.g. OR)Limited precedent in surgery

ACS-APDS Phase III Curriculum

Unique communications rating instrument

Checklisted trauma assessment SPIKES protocol (delivering bad news) NOTECHS Ottawa Global Scale Need to strengthen and consolidate

assessment

Assessment

Conclusions Assessment of surgical skills and OR

performance:Opportunities

Expanded uses:Identification of performance outliers.Definition of areas where focused training is required

for specific residents.Measurement of effects of educational interventions.

Measurement of curriculum effectiveness