Strategies to improve acquisition of technical skill in surgical residents:
from screening technical ability at the time of selection to incorporating
performance adjuncts during training.
by
Marisa Louridas
A thesis submitted in conformity with the requirements
for the degree of Doctor of Philosophy in Medical Science
Institute of Medical Science
University of Toronto
© Copyright by Marisa Louridas 2016
ii
Marisa Louridas
Doctor of Philosophy
Institute of Medical Science
University of Toronto
2016
Strategies to improve acquisition of technical skill in surgical residents:
from screening technical ability at the time of selection to incorporating
performance adjuncts during training.
Abstract
Introduction: Evidence suggests that not all trainees reach technical competence.
Therefore the purposes of the included studies were to improve resident selection
by investigating screening tools (visual spatial tests (VSTs) and technical tasks
(TTs)) that may predict technical ability of incoming trainees, and to determine
whether metal practice is beneficial as a performance enhancement strategy during
training.
Methods: Screening with VSTs as a predictor of laparoscopic ability was
evaluated using the PicSOr, cube comparison (CC) and card rotation (CR) tests and
correlated to technical performance on the camera navigation (LCN) and
iii
laparoscopic circle cut (LCC) tasks. To screen trainees using TTs, a Delphi of
Canadian general surgery (GS) program directors (PD), was performed to gain
consensus on the simulated TTs best suited for incoming trainees. K-mean
clustering learning curve (LC) analysis was used to determine acquisition of TTs.
Next, mental practice was evaluated in a randomized control trial to assess its
impact on advanced laparoscopic technical performance.
Results: Thirty-seven residents were screened using VSTs. Residents who scored
higher on the CC test had more accurate LCN path length (rs(PL) =-0.36, p=0.03)
and angle path (rs(AP) =-0.426, p=0.01) scores. Eleven of 14 GS PDs participated in
the Delphi, and consensus was reached that both basic laparoscopic and open skills
would be appropriate for the assessment of TTs. LC analysis of 65 students
revealed that 7-15% of trainees did not reach proficiency in laparoscopic skills.
These students demonstrated poor innate ability, and remained disadvantaged with
inconsistent performance throughout their LC. During training, mental practice
significantly improved technical performance (p =0�003).
Conclusion: LC analysis of simulated technical skills proved more dependable
than VSTs to screen for technical ability in novice trainees, while mental practice
is an affective adjunct to technical skills performance and would be a beneficial
addition to skills training for senior residents.
iv
Acknowledgments
I would like to express my heartfelt gratitude to my program advisory committee Drs. Tulin Cil
and Simon Graham, who have provided experienced advice and ongoing support throughout my
degree. I would also like to thank my research supervisor Dr. Teodor Grantcharov who has spent
countless hours discussing project designs, fine tuning methodology and providing the guidance
required to execute novel timely research within the field of surgical education. Through their
mentorship I have had the opportunity to publish broadly and present consistently throughout my
PhD at both national and international conferences.
This PhD would not have been possible without the Surgeon Scientist Training Program and the
Clinical Investigator Program. These programs provide financial support and superb academic
resources, to create an environment where clinical residents can dedicate time to pursue
advanced degrees in science, forming the foundation for a career as a clinical investigator. Drs.
Najma Ahmed, Andy Smith, Carol Swallow, James Rudka, Andrea McCart and Norm
Rosenblum, thank you for this opportunity.
Finally I would like to thank my friends, family and colleagues, who have offered words of
wisdom, unwavering support, love and balance during the course of this degree. Katy Louridas,
Georgia Louridas, Dr. Micahel Zywiel, Dr. Peter Szasz, Dr. Sandra de Montbrun, Dr. Esther
Bonrath, Dr. Nicolas Dedy and Dr. Andras Fecso - thank you!
v
Contributions
Marisa Louridas independently prepared this thesis and all aspects of the included original
research studies from: study design, data collection, analysis and writing. This thesis contains
five original manuscripts with Marisa Louridas as the primary author. All contributions by
coauthors are described in detail below:
Supervisor – Dr. Teodor Grantcharov – mentorship, guidance for study design, laboratory
resources, introductions to collaborators and manuscript/thesis editions.
Thesis committee members – Dr. Tulin Cil and Dr. Simon Graham, study design guidance and
thesis preparation
Lauren Quinn – Contributed to the data collection, analysis and preparation of the manuscript in
Chapter 3.
Dr. Peter Szasz – assisted in grading quality of studies included in section 2.4 and assisted with
study design and participant correspondence in Chapter 4 and trained participants in Chapter 5
while contributing to the preparation and editions of all three manuscripts.
Dr. Sandra de Montbrun – assisted in study design, analysis and manuscript preparation and
editions of section 2.4 and Chapter 4.
Dr. Michael Zywiel – assisted in the study design, analysis interpretation and manuscript
preparation and editions of Chapter 5.
Dr. Andras Fecso- assisted in preparing the technical skills curriculum trained participants in
Chapter 5 while contributing to the preparation and editions of this manuscript
Parisa Lak and Dr. Ayse Bener - experts in data science and machine learning, assisted in the k-
means analysis of Chapter 5 and edited the manuscript.
Drs. Esther Bonrath, Dana Sinclair and Nicolas Dedy contributed to the planning, design,
execution and manuscript preparation of Chapter 6
ACK
CON
TAB
LIST
LIST
LIST
CHAP
1.1
1.2
C2
2.1
2.1.1
2.1.2
2.1.3
2.1.4
2.1.5
perfo
2.2
2.3
2.3.1
2.3.2
2.3.3
2.4
2.4.1
2.4.2
KNOWLED
NTRIBUTIO
LE OF CON
T OF TABL
T OF FIGUR
T OF ABBR
PTER1:GE
THESISOVE
HYPOTHESE
CHAPTER2
THECURREN
CaRMSw
National
Informal
TheCana
Evidence
ormance......
SURGICALT
POTENTIAL
Self‐selec
Surrogat
Therole
SIMULATION
Opensim
Minimall
GMENTS..
ONS.............
NTENTS...
LES...............
RES............
REVIATION
ENERALINT
ERVIEW.........
ESANDAIMS
:LITERATU
NTRESIDENT
writtenappl
interviewp
discussion
adiannation
eforthecur
.......................
TRAINEES’VAR
PREDICTORS
ctionasapr
emarkersa
ofsimulate
NMODALITIE
mulationmo
ly‐invasives
Tab
.....................
.....................
.....................
.....................
.....................
NS................
TRODUCTIO
.....................
.....................
UREREVIE
TSELECTIONP
ication.........
process.........
........................
nalresidenc
rrentselecti
........................
RIABLESUCCE
SOFTECHNICA
redictoroft
aspredictor
dtechnical
ESFORTECHN
odels..............
simulationm
vi
ble of Con
......................
......................
......................
......................
......................
......................
ON...............
......................
......................
W.................
PROCESS........
........................
........................
........................
cymatch......
ionparamet
........................
ESSINACQUIR
ALABILITYFO
technicalab
rsoffuturet
taskstoinf
NICALSKILLTR
........................
models.........
ntents
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
........................
........................
........................
........................
tersandthe
........................
RINGTECHNIC
ORUSEDURIN
biliy................
technicalab
formtrainee
RAINING......
........................
........................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
.....................
........................
........................
........................
........................
eirabilityto
........................
CALSKILL.....
NGTHESELEC
........................
bility.............
eselection..
.....................
........................
........................
......................
......................
......................
......................
......................
......................
......................
......................
......................
......................
......................
.......................
.......................
.......................
.......................
opredictres
.......................
......................
CTIONPROCE
.......................
.......................
.......................
......................
.......................
.......................
...........IV
.............V
...........VI
...........XI
........XIII
..........XV
.............1
.............1
.............2
.............5
.............5
...............6
...............7
...............8
...............8
sident
...............9
..........13
ESS.....14
............14
............16
............41
..........43
............44
............46
2.5
2.5.1
2.5.2
2.5.3
2.5.4
2.6
2.6.1
2.6.2
C3
SPAT
RESI
3.1
3.2
3.3
3.3.1
3.3.2
3.3.3
3.3.4
3.3.5
3.3.6
3.4
3.4.1
3.4.2
3.4.3
3.5
C4
NATI
4.1
4.1.1
4.2
INSTRUMEN
Taskspe
Globalra
Taskmet
Usinglea
TECHNICAL
Structuri
Mentalp
CHAPTER3
TIALABILIT
IDENTSENT
ABSTRACT.
INTRODUCT
MATERIALS
Participa
Demogra
Visuospa
Laparosc
Assessme
Statistica
RESULTS....
Demogra
Participa
Correlati
DISCUSSION
CHAPTER4
IONALCON
ABSTRACT.
Backgrou
METHODS..
NTSFORTHEA
cificcheckli
atingscales.
trics..............
arningcurve
SKILLTRAIN
ingcurricula
racticeasa
:PREDICTI
TYTESTIN
TERINGPO
.....................
TION..............
SANDMETHO
antsandsett
aphicsandp
atialtesting
copicskillsf
entoflapar
alMethods.
.....................
aphics...........
antquestion
ionofvisual
N....................
:OPTIMIZI
NSENSUS....
.....................
und................
.....................
ASSESSMENT
ists.................
........................
........................
estoassess
INGCURRICU
afortechni
nadjunctfo
IVEVALUE
GONLAPA
OSTGRADUA
.....................
.....................
ODS...............
ting................
participantq
........................
familiaritys
roscopicbas
........................
.....................
........................
nnaire............
lspatialskil
.....................
INGTHESE
.....................
.....................
........................
.....................vii
OFTECHNICA
........................
........................
........................
technicalsk
LA.................
calskilltrai
ortechnical
OFBACKG
AROSCOPIC
ATESURGI
......................
......................
......................
........................
questionnai
........................
session.........
selineskill...
........................
......................
........................
........................
llsandlapa
......................
ELECTIONO
......................
......................
........................
......................
ALSKILL........
........................
........................
........................
killprogres
.....................
ining.............
skilltrainin
GROUNDEX
CBASELINE
ICALTRAIN
.....................
.....................
.....................
........................
ire..................
........................
........................
........................
........................
.....................
........................
........................
roscopicba
.....................
OFGENERA
.....................
.....................
........................
.....................
.....................
........................
........................
........................
ssion..............
.....................
........................
ng...................
XPERIENCE
EPERFORM
NING............
.....................
.....................
.....................
........................
........................
........................
........................
........................
........................
.....................
........................
........................
aselineperfo
.....................
ALSURGER
.....................
.....................
........................
.....................
......................
.......................
.......................
.......................
.......................
......................
.......................
.......................
ESANDVISU
MANCEAMO
......................
......................
......................
......................
.......................
.......................
.......................
.......................
.......................
.......................
......................
.......................
.......................
ormance.....
......................
RYRESIDEN
......................
......................
.......................
......................
..........49
............51
............51
............53
............54
..........55
............55
............56
UAL
ONG
..........69
..........69
..........70
..........72
............72
............72
............73
............75
............75
............75
..........76
............76
............79
............79
..........80
NTS‐A
..........84
..........84
............85
..........86
4.2.1
4.2.2
4.2.3
4.2.4
4.2.5
4.2.6
4.2.7
4.3
4.3.1
4.3.2
4.4
4.5
C5
SELE
5.1
5.2
5.3
5.3.1
5.3.2
5.3.3
5.3.4
5.3.5
5.4
5.4.1
5.4.2
5.4.3
5.4.4
5.4.5
lapar
5.5
5.6
Currents
Delphico
Expertpa
Anonymi
Roundso
Consensu
Recomm
RESULTS....
Currents
Delphico
DISCUSSION
CONCLUSION
CHAPTER5
ECTIONCUR
ABSTRACT.
BACKGROUN
METHODS..
Participa
SampleS
Demogra
Training
Dataana
RESULTS....
Participa
Learning
Associati
Individua
Concerni
roscopictas
DISCUSSION
CONCLUSION
selectionpr
onsensusm
anel..............
ity..................
ofquestions
us...................
endationsf
.....................
selectionpr
onsensusm
N....................
N..................
:PRACTICE
RRICULAIN
.....................
ND................
.....................
ants................
Size................
aphicquesti
curriculum
lysisandsta
.....................
ants................
gcurvespatt
ionsbetwee
alperforma
inglearning
sks..................
N....................
N..................
ractices.........
ethodology
........................
........................
s.......................
........................
forincorpor
.....................
rocess............
ethodology
.....................
.....................
EDOESNOT
NMODERN
.....................
.....................
.....................
........................
........................
ionnaire.......
m.......................
atisticalme
.....................
........................
ternsforlap
enbackgrou
ancedifferen
gcurvefeatu
........................
.....................
.....................viii
........................
y.......................
........................
........................
........................
........................
rationintos
......................
........................
y.......................
......................
......................
TALWAYS
SURGICAL
......................
......................
......................
........................
........................
........................
........................
ethods...........
......................
........................
paroscopic
undcharacte
ncesbetwee
uresforind
........................
......................
......................
........................
........................
........................
........................
........................
........................
selection......
.....................
........................
........................
.....................
.....................
MAKEPER
LTRAINING
.....................
.....................
.....................
........................
........................
........................
........................
........................
.....................
........................
andopensk
eristicsand
entasks.......
dividualsun
........................
.....................
.....................
........................
........................
........................
........................
........................
........................
........................
.....................
........................
........................
.....................
.....................
RFECT‐TH
G...................
.....................
.....................
.....................
........................
........................
........................
........................
........................
.....................
........................
kills...............
dperforman
........................
abletoreac
........................
.....................
.....................
.......................
.......................
.......................
.......................
.......................
.......................
.......................
......................
.......................
.......................
......................
......................
ENEEDFO
......................
......................
......................
......................
.......................
.......................
.......................
.......................
.......................
......................
.......................
.......................
nceclusters
.......................
chproficien
.......................
......................
......................
............86
............87
............87
............87
............88
............88
............88
..........89
............89
............91
..........96
..........99
OR
........101
........101
........102
........103
.........103
.........104
.........104
.........105
.........108
........109
.........109
.........111
.........116
.........118
cyin
.........120
........121
........125
C6
ENHA
6.1
6.2
6.3
6.3.1
6.3.2
6.3.3
6.4
6.4.1
6.4.2
6.5
C7
7.1
7.2
7.2.1
2D‐3
7.2.2
achie
7.2.3
techn
7.2.4
adva
7.2.5
tasks
7.2.6
simu
from
CHAPTER6
ANCINGAD
ABSTRACT.
INTRODUCT
METHODS..
Mentalp
Randomi
Statistica
RESULTS....
Mentalp
Randomi
DISCUSSION
CHAPTER7
THESISSUM
FINDINGSW
Toevalua
Dvisualspa
145
Tosolicit
evethemini
Toestabl
nicalskillst
Toquant
ncedlaparo
Todeterm
sofvarying
Toidenti
ulatedtasks
mtheirpeers
:RANDOM
DVANCEDL
.....................
TION..............
.....................
racticescrip
izedtrial.....
alanalysis...
.....................
racticescrip
izedtrial.....
N....................
:GENERAL
MMARY..........
WITHRESPECT
atewhether
atialtestsco
tprogramd
imumtechn
lishanation
thatwouldb
tifydifferen
oscopicand
minewheth
difficulty(b
ifyasubset
anddeterm
s......................
IZEDCLINI
LAPAROSCO
.....................
.....................
.....................
ptdevelopm
........................
........................
.....................
pts..................
........................
.....................
LDISCUSSIO
.....................
TTOTHEAIM
rpreviouss
orrelatewit
directors’op
nicalstanda
nalconsens
bemostind
tlearningp
openskills
hertrainees
basicandad
oftrainees
minethefeat
........................
ix
ICALTRIAL
OPICSURGI
......................
......................
......................
ment..............
........................
........................
......................
........................
........................
......................
ON................
......................
SOFTHETHE
surgicalexp
thlaparosco
pinionsonth
rdsexpecte
usonthede
icativeoffu
patternsam
.......................
staywithin
dvanced)an
whoconsis
turesofthe
........................
LTOEVALU
ICALPERFO
.....................
.....................
.....................
........................
........................
........................
.....................
........................
........................
.....................
.....................
.....................
ESIS..............
periences,no
opicskillsin
heproporti
edatthetim
esiredattrib
utureperfor
ongtrainee
........................
ntheirlearn
ndtype(min
stentlyfailto
irlearningc
........................
UATEMENT
ORMANCE
.....................
.....................
.....................
........................
........................
........................
.....................
........................
........................
.....................
.....................
.....................
.....................
on‐surgical
nthenovice
ionoftraine
meofgradua
butesofGS
rmance........
esforbothb
........................
ningpattern
nimallyinva
oreachpro
curvesthat
........................
TALPRACT
.....................
......................
......................
......................
.......................
.......................
.......................
......................
.......................
.......................
......................
......................
......................
......................
experience
esurgicaltr
eeswhodo
ation.............
candidates,
.......................
basicandmo
.......................
nsacrosssim
asiveandop
ficiencyon
separateth
.......................
TICEIN
........127
........127
........128
........129
.........129
.........130
.........133
........133
.........133
.........135
........141
........144
........144
........145
esand
rainee
not
.........147
,andthe
.........148
ore
.........149
mulated
pen)150
hem
.........150
7.2.7
7.2.8
perfo
7.2.9
impr
7.3
7.3.1
7.3.2
selec
7.3.3
C8
C9
9.1.1
9.1.2
9.1.3
realo
COP
10
Todevelo
151
Toassess
ormance......
Todeterm
rovementin
LIMITATION
Defining
Theimpl
ctionandsu
Adequate
CHAPTER8
CHAPTER9
Exploring
Establish
Quantifyi
operatingro
PYRIGHT A
REFERENC
opaMPscr
stheeffectiv
.......................
minewheth
nnon‐techni
NS.................
cutoffscor
lementation
rgicaltraini
esampling.
:CONCLUS
:FUTURED
gintraopera
hassessmen
ingmentalp
oom..............
ACKNOWL
CES...............
riptforthep
venessofM
........................
herMPisas
icalskillsin
.....................
res...................
nofglobalra
ing.................
........................
ION.............
DIRECTION
ativetechni
ntmeasures
practiceobj
........................
LEDGEMEN
.....................
x
performanc
MPonadvan
........................
sociatedwi
nasimulated
......................
........................
atingscales
........................
........................
......................
NS..................
icalskillsas
sofnon‐tech
jectivelyan
........................
NTS.............
......................
eofanadva
ncedlaparos
........................
ithdifferenc
dcrisisscen
.....................
........................
sasaroutin
........................
........................
.....................
.....................
ssessment...
hnicalcomp
dtransferri
........................
.....................
.....................
ancedlapar
scopictechn
........................
cesinstress
nario.............
.....................
........................
neassessme
........................
........................
.....................
.....................
........................
petencies....
ingthistech
........................
.....................
.....................
roscopicpro
nicalskill
.......................
slevelsand
.......................
......................
.......................
entmethodd
.......................
.......................
......................
......................
.......................
.......................
hniquesinto
.......................
......................
......................
ocedure
.........152
.........152
........153
.........153
during
.........158
.........159
........161
........164
.........164
.........165
othe
.........165
........167
........169
xi
List of Tables
Table 1: GRADE classification and assessment method of included studies organized by
subjective or objective assessment method ................................................................................... 25
Table 2: Summary of background characteristics, surgical and non-surgical experiences as
predictors of surgical performance collected by participant questionnaires ................................. 26
Table 3: A summary of visual spatial tests as predictors of surgical performance ....................... 28
Table 4: Summary of dexterity tests as predictors of surgical performance ................................. 32
Table 5: Task specific checklist applicable to both the placement of one interrupted suture and
intracorporeal knot tying ............................................................................................................... 50
Table 6: Objective Structure of Technical skills - Global Rating Scale (Martin et al., 1997) ..... 52
Table 7: Assessment tools of non-technical skills in the operating room ..................................... 68
Table 8: Demographic and background characteristics of study participants .............................. 77
Table 9: Previous surgical and non-surgical experiences ............................................................. 78
Table 10: Correlation of 2D-3D innate ability tests with laparoscopic surgical skill ................... 80
Table 11: Participating Canadian General Surgery Programs ...................................................... 90
Table 12: General Surgery program director's responses to clinical knowledge, decision-making
and technical skill during selection and at the time of graduation ................................................ 92
Table 13: Desired candidate attributes for selection into General Surgery .................................. 93
Table 14: Appropriate simulated surgical skills for selection into General Surgery .................... 94
Table 15: Proficiency scores for open tasks ............................................................................... 108
Table 16: Comparing clusters 1-4: start points, end points and repetitions to proficiency ........ 114
xii
Table 17: Demographics and non-surgical experiences and their association with performance
clusters ........................................................................................................................................ 117
Table 18: Consistency of performance clusters for laparoscopic and open technical skill ........ 119
Table 19: Demographics of study participants ........................................................................... 136
Table 20: Result of baseline assessments of technical skill and mental rotation ability ............ 136
Table 21: Technical skill results at baseline and following training .......................................... 140
Table 22: Mental imagery ability at baseline and following training ......................................... 141
xiii
List of Figures
Figure 1: Flow diagram of search strategy ................................................................................... 20
Figure 2: Simulated open technical skills models a. low fidelity synthetic, b. high fidelity
cadaveric porcine, c. high fidelity synthetic (picture provided courtesy of Dr. Sandra de
Montbrun) ..................................................................................................................................... 45
Figure 3: (a) the laparoscopic box trainer can be used for both basic and advanced laparoscopic
tasks, using either a low fidelity model such as (b) a plastic penrose drain, or a high fidelity
model such as (c) cadaveric porcine small bowel with attached mesentery. ................................ 47
Figure 4: Virtual reality simulator for laparoscopic technical skill training and assessment. ...... 49
Figure 5: Task metrics for the laparoscopic knot tie task, using the laparoscopic box trainer ..... 53
Figure 6: Examples of expected smoothed learning curves for the performance of (a) a basic task
and (b) a difficult task. Actual performance of an individual learning a basic task may display
additional fluctuations between attempts (c). ............................................................................... 55
Figure 7: Adapted from Lang’s model of input and output variables for emotional imagery. ..... 59
Figure 8: Mental Practice protocol for surgery, adopted from (S. Arora et al., 2010). ................ 64
Figure 9: Pictorial Surface orientation test (PicSOr) used to assess 2D-3D perception ability. a.
Setup to change between practice and experiment mode. b. rotating arrow oriented to lie 90
degrees to the underlying cube. .................................................................................................... 74
Figure 10: Paper tests used to assess 2D-3D visual spatial ability included the (a) card rotation
test (CR) and (b) cube comparison test (CC) ................................................................................ 74
Figure 11: Learning curves for clusters 1 (top performers) to 4 (low performers) on the
laparoscopic (a) peg transfer, (b) circle cut, and (c) intracorporeal knot tie tasks, ordered from
basic to advanced. Proficiency scores are demarcated with a dashed line. ................................ 111
xiv
Figure 12: Learning curves for clusters 1 (high performers) to 4 (low performers) on the open (a)
one handed tie and (b) laparotomy closure tasks, ordered from basic to advanced. ................... 112
Figure 13: Learning curves for a representative individual from cluster 3 (moderate performer)
and from cluster 4 (low performer) on the laparoscopic intracorporeal knot tie, demonstrating
variability in performance times and associated normalized scores. .......................................... 121
Figure 14: CONSORT diagram illustrating progress through the phases of the study .............. 135
Figure 15: Comparison of Objective Structured Assessment of Technical Skill (OSATS) change
scores between groups. Median (line within box), interquartile range (box), and range (error
bars) excluding outliers (circles) are shown. Dotted line indicates baseline performance. P
=0�003 (Mann–Whitney U test) ................................................................................................. 138
Figure 16: Comparison of bariatric Objective Structured Assessment of Technical Skill
(BOSATS) change scores between groups. Median (line within box), interquartile range (box),
and range (error bars) excluding outliers (circles) are shown. Dotted line indicates baseline
performance. P =0�003 (Mann–Whitney U test) ........................................................................ 139
Figure 17: A five-stage model of the mental activities involved in directed skill acquisition
(adapted from (Dreyfus & Dreyfus, 1980)). ............................................................................... 155
Figure 18: Standard setting using: a. contrasting groups and b. borderline group methodology
(images adapted from (de Montbrun, Statterthwaithe, & Grantcharov, 2015)) .......................... 156
Figure 19: Standard setting using borderline regression methodology (image adapted from (de
Montbrun et al., 2015)) ............................................................................................................... 157
xv
List of Abbreviations
TT – technical task
VST – visual spatial test
ACGME - Accreditation Council of Graduate Medical Education
FLS – Fundamentals of laparoscopic surgery
MP – mental practice
MI – mental imagery
MR – Mental rehearsal
CaRMS - Canadian Resident Matching Service
USMELE - United States Medical Licensing Examination
JJ – Jejunojejunostomy
RYGB – Roux-en-y gastric bypass
FLS – Fundamental of Laparoscopic Surgery
MIQ - Mental Imagery Questionnaire
MIQ –RS - Movement Imagery Questionnaire Revised Second version
OSATS – Objective Structure Assessment of Technical Skills
BOSATS – Bariatric Objective Structure Assessment of Technical Skills
STAI – State trait anxiety questionnaire
BP – Blood pressure
xvi
HR – heart rate
NOTSS – non-technical skill for surgeons
CC – Cube comparison test
CR – Card rotation test
PicSOr – Pictorial Surface Orientation test
LCC – laparoscopic circle cut
LCN – laparoscopic camera navigation
PGY – post-graduate year
CUSUM - curve cumulative summation test
MS - Motivation-Specific
MG-M - Motivational General-Mastery
MG-A - Motivation General-Arousal
CS - Cognitive Specific
CG - Cognitive General
Chap
1.1
Techn
surgic
This th
pter 1: G
Thesis o
nical skill ab
al training in
1. At the ti
have the ap
2. During
have starte
simulation
3. Followin
and to lear
hesis focuse
Chapter 1:
Chapter 2:
in training
predict futu
to predict s
technical s
techniques
practice: th
surgery.
General i
overview
bility, develo
ncluding:
ime of select
ptitude to rea
surgical trai
ed surgical
n center to en
ng completio
rn new surgic
s on the first
the specific
the backgro
. A detailed
ure resident
simulated tec
skills selectio
s and the ava
he history, hy
introduc
opment and
tion – to ens
ach technica
ining – to d
residency, w
nhance perfo
on of formal
cal techniqu
t two trainin
c hypothesis
ound for the
literature rev
clinical perf
chnical skill
on curriculum
ailable assess
ypothesized
tion
d assessment
ure that stud
al competenc
develop adju
with the go
ormance in th
l training – t
es.
g stages with
and aims of
first two trai
view of curr
formance an
. Furthermor
m, a review
sment metho
theories and
t can be im
dents selecte
ce during tra
uncts to tech
al of shorte
he operating
to improve a
h the structu
f each projec
ining levels
rent selection
nd a systemat
re, to provid
of the pros a
ods for each.
d existing lit
mplemented a
ed to enter su
aining;
hnical trainin
ening the le
g room;
already acqui
ure outlined
ct.
mentioned a
n practices, t
tic review of
de the backgr
and cons of d
. Finally, a r
terature in bo
at various s
urgical reside
ng for stude
earning curv
ired surgical
as follows:
above: select
their ability
f surrogate m
round to cre
different sim
review on me
oth sports an
1
stages of
ency
ents who
ve in the
l skills
tion and
to
markers
eate a
mulated
ental
nd
1.2
At the
Hypot
It is hy
and lap
studen
and th
Manus
screen
Manus
learne
tasks,
Chapter 3,
selection. C
are able to
survey on g
suitable for
learning cu
of mental p
residents p
Chapter 7,
directions.
Hypothe
e time of s
theses
ypothesized
paroscopic t
nts for techni
he training pr
script 1: Spe
ning tools to
script 2 and
rs by assessi
it is hypothe
4, 5 and 6 a
Chapter 3 is
predict simu
general surg
r incoming t
urves using k
practice as a
performing a
8 and 9 incl
eses and A
election
that not all s
technical ski
ical aptitude
rogram.
ecific surroga
predict tech
3: Furtherm
ing their lear
esized that m
are original r
a cross secti
ulated techn
gery program
trainees and
k-means clus
a performanc
advanced lap
lude the gen
Aims
surgical train
ills by the en
e before entry
ate markers
hnical aptitud
more, simulat
rning curves
medical stude
research man
ional cohort
ical perform
m directors op
chapter 5 is
stering. Las
ce enhancem
paroscopic sk
eral discussi
nees will be
nd of training
y into a surg
that assess v
de in laparos
ed technical
s over multip
ents will disp
nuscripts foc
t study asses
mance. Chapt
pinions on th
a prospectiv
stly, chapter
ment strategy
kills.
ion, limitatio
able to reach
g. Therefore
gical program
visual spatial
scopic skills.
l tasks may a
ple repetition
play similar
cusing on te
sing whethe
ter 4 is a Del
he simulatio
ve cohort stu
6 is a random
for senior g
ons, conclusi
h technical c
, screening i
m may benef
l ability may
.
also be used
ns. Within si
learning cur
chnical skill
r visual spat
lphi consens
on skills mos
udy, analyzin
mized contro
general surge
ion and futu
competence
incoming me
fit both the tr
y be appropr
to stratify d
imulated tec
rves for disp
2
ls and
tial tests
sus
st
ng
ol trial
ery
ure
in open
edical
rainee
riate
ifferent
hnical
parate
3
basic laparoscopic and open surgical skills, and that these will be correlated with their potential to
reach proficiency in subsequent, more complex technical tasks.
Aims
1: To evaluate whether previous surgical experiences, non-surgical experiences and 2D-
3D visual spatial tests correlate with baseline laparoscopic skills in the novice surgical
trainee.
2: To identify the current components used in the general surgery selection process at
different institutions
3: To solicit program directors’ opinions on the proportion of trainees who do not achieve
the minimum technical standards expected at the time of graduation
4: To establish a national consensus on the desired attributes of GS candidates, and the
technical skills that would be most indicative of future performance
5: To quantify different learning patterns among trainees for both basic and more
advanced laparoscopic and open skills
6: To assess whether background characteristics or experiences explain potential
differences in performance.
7: To determine whether trainees stay within their learning patterns across simulated tasks
of varying difficulty (basic and advanced) and type (minimally invasive and open)
8: To identify a subset of trainees who consistently fail to reach proficiency on simulated
tasks and determine the features of their learning curves that separate them from their
peers
During surgical training
4
Hypotheses
It is hypothesized that mental practice (MP) aimed at teaching the visual and kinesthetic cues for
the crucial operative steps of advanced laparoscopic surgery, specifically laparoscopic
jejunojenunostomy (JJ), will improve surgical performance and decrease stress levels
experienced by the surgeon during adverse situations. Furthermore, it is hypothesized that this
approach will improve surgeons’ non-technical skills thereby maintaining a competent level of
communication, leadership and decision-making during stressful situations.
Aims
1: To develop a MP script for the performance of an advanced laparoscopic procedure
2: To assess the effectiveness of MP on advanced laparoscopic technical skill performance
3: To determine whether MP is associated with differences in stress levels and improvement in
non-technical skills in a simulated crisis scenario.
Chap2
2.1
Each y
gradua
or her
Physic
remed
The se
intervi
residen
the Ca
CaRM
incom
assess
institu
Progra
The fo
(Canad
slightl
simila
separa
checkp
comm
pter 2: L
The curr
year in North
ate as compe
role is to en
cians and Su
diation of trai
election proc
iew for succ
ncy position
anadian Resi
MS system, w
ming trainees
ment forms
ution. The Un
am (Nationa
ormat of the
dian Residen
ly different (
ar across the
ate section to
point on the
ment within a
Literature
rent resid
h America m
etent surgeon
nsure trainee
urgeons. Furt
inees within
cess in Canad
essful candi
n in Canada,
ident Matchi
which is an o
’ application
the candidat
nited States u
l Resident M
documents i
nt Matching
(e.g. requirin
country. In N
o assess tech
surgical rota
a reference le
e review
dent selec
medical and s
ns. In Canad
s reach comp
thermore, thi
n their progra
da is compo
dates. To su
medical stud
ing Service (
online platfor
ns. Each surg
te is required
uses a simila
Matching Pro
inputted into
Service, 20
ng 4 referenc
North Amer
hnical aptitud
ation In Trai
etter(s), how
w
tion proc
surgical prog
da, the progra
petency in th
is individual
am (R. C. o.
sed of a stru
ubmit a struct
dents in thei
(CaRMS). E
rm, designed
gical program
d to input to
ar process ca
ogram).
o the CaRMS
14). Althoug
ce letters rath
ica the curre
de. At best, t
ining Evalua
wever neither
ess
grams endea
am director
heir field as
l is responsib
P. a. Surgeo
uctured writt
tured written
ir fourth year
Each surgical
d to collect i
m specifies w
complete th
alled The Na
S online plat
gh the each p
her than 3), t
ent applicatio
technical per
ation Report
r of these are
avor to selec
is an appoin
defined by t
ble to selecti
ons, 2015).
ten applicatio
n application
r enter a nati
l program is
information t
which object
he written ap
ational Resid
tform is outl
program’s re
the general f
on form doe
rformance m
s (ITER) or
e standardize
t trainees wh
nted position
the Royal Co
ion, evaluati
on followed
n for a surgic
ional match
required to
that will mak
tive and subj
pplication for
dent Matchin
lined in detai
equirements
format is ver
es not contain
may be a sing
a subjective
ed or mandat
5
ho will
n and his
ollege of
ion and
by an
cal
through
use the
ke-up
jective
r their
ng
il below
may be
ry
n a
gle
e
tory.
6
2.1.1 CaRMS written application
1. Curriculum vitae
Verification of Canadian citizenship and medical school attendance.
All candidates require Canadian permanent residency status or citizenship to apply
for a post-graduate medical training position in Canada.
2. Clinical clerkship In Training Evaluation Reports (ITERS)
These are institution-specific rating scale assessment tools used by faculty to score
the student’s performance within the clinical setting.
3. Examinations
Medical Council of Canada Examination
Medical Council of Canada Qualifying Examination (MCCQE)
part 1 is a mandatory examination for all medical students before
entering supervised practice in a postgraduate training program. In
Canada releasing the exam score is optional.
United States Medical Licensing Examination (USMLE)
These examinations are mandatory to be licensed medical doctor
within the United States of America (USA). Part 1 of 3 is
mandatory for application to medical school. These examination
results are required for applicants applying for residency training in
the USA, however optional for student applying in Canada.
4. Clinical electives
These are clinical rotations during the final year of medical school in either
the discipline or a related discipline to which the candidate is applying.
5. Scholarships and awards
7
The selection committees generally perceive both scholarly and
extracurricular achievements favorably.
6. Research experience and Publications/presentations
An interest in science and research with evidence of productivity also
increase the strength of the student’s application.
7. Work experience and Volunteer activity
Programs endeavor to recruit well-rounded applicants therefore
extracurricular activities contribute to assessing the applicant’s
experiences, interests and commitments outside of their university
degree(s).
8. Personal statement
A personalized one page description that explains the applicant’s
motivation(s) for applying to the program and their future goals.
9. Reference letters
Each applicant is required to submit 3-5 reference letters written by faculty
who can comment on the students’ clinical performance, ability to work
within the interdisciplinary heath care team and overall ability to excel
within the program.
Candidates then select the institutions to which they wish to apply and each institution reviews
the application as per their own selection criteria. If successful, the candidate is invited for a
formal in-person interview.
2.1.2 National interview process
The interview process differs between institutions across the country in terms of the number of
interviews required of each candidate, the size of each interview panel, and the interview
8
questions asked. However, the end goal in most cases is to assess the candidates for
characteristics that are not easily obtained from the paper application including: communication
skills, enthusiasm for surgery, program fit and interpersonal , problem-solving skills. To do so,
each institution hosts an interview day where invited candidates attend in person. However,
similar to the paper application, technical performance is not routinely incorporated into the
interview process.
2.1.3 Informal discussion
Both medical and surgical programs acknowledge that the combined scores of the written
application and the interview do not always adequately assess each applicant holistically.
Therefore, informal discussions between the faculty and residents who have worked directly with
the students are also part of the selection process. Many of the applicants will travel to the
institutions they are most interested in applying to and work clinically with the residents and
faculty during a 2-4 week onsite elective. Over this time period, the candidates overall clinical
performance may uncover either desirable or undesirable traits that are helpful for the selection
committee. Therefore, feedback and input from these encounters are encouraged.
2.1.4 The Canadian national residency match
After the institution has combined the scores of the written application, the interview and the
informal discussions, a final rank list of candidates is submitted from each institution to CaRMS.
The process of the combining the scores is usually quantitative however the weight of the scores
that contribute to the total score are program specific and differ across institutions. Each student
also creates their own rank list and both lists are entered into the CaRMS Roth-Peranson
algorithm ("CaRMS: The Match Algorithm," 2015). On “national match day”, successful
candidates are matched to a program and institution, which they are contractually obligated to
attend for their residency training (Canadian Resident Matching Service, 2014).
Theref
do not
neithe
the tec
2.1.5
Before
proces
potent
exami
residen
progra
below
2.1.5.
The ov
applic
institu
strong
extrap
succes
suppor
recruit
fourth
(Adus
recruit
by fac
fore, across
t undergo an
r the trainee
chnical skills
Evide
reside
e proposing
ss, a literatur
tial. Many st
nation score
ncy. The sel
am including
.
Over.1
verall rank s
ation, interv
ution’s ability
gest students
polated to me
ssful they wi
rted by the l
tment at a sin
year of resi
umilli et al.,
tment year, t
ulty assessm
Canada and
n assessment
nor the train
s needed in t
nce for the
ent perform
yet another
re review wa
tudies have e
es, interview
lection proce
g radiology,
rall institut
core for each
view and info
y to attract th
in the applic
ean that the h
ill be in the c
iterature. Fo
ngle program
dency, with
, 2000). Mor
the three low
ment in their
the United S
of technical
ning program
their surgica
e current se
mance
assessment m
as completed
evaluated the
scores and r
ess holds for
pediatrics, s
tional rank
h candidate
ormation dis
heir highest
cation cohor
higher the st
clinical envi
or example, t
m was comp
no significa
reover, of the
west ranked s
fourth year
States, stude
l aptitude. Th
m knows wh
al training.
election pa
metric to be
d of the curre
e ability of th
reference let
r all trainees
urgery and i
k score
is the institu
scussions. A
ranked cand
rt for their gi
tudent is rank
ronment. Ho
the rank scor
pared to over
ant relationsh
e five candid
students wer
of training (
ents who suc
he present se
hether or not
arameters a
added to the
ent assessme
he overall in
tters to predi
entering res
internal med
ution’s cumu
measure of
didates, whic
iven instituti
ked during t
owever, this
re of radiolo
rall clinical p
hip identified
dates that ma
re considered
(Adusumilli
ccessfully en
election proc
the trainee i
and their ab
e already len
ents and thei
nstitutional r
ict clinical p
sidency and c
dicine. Thes
ulative score
recruitment
ch are percei
ion. Intuitive
the selection
assumed rel
ogy residents
performance
d between th
atched durin
d the stronge
et al., 2000)
nter surgical
cess ensures
is adept to ac
bility to pr
ngthy selecti
ir predictive
rank score,
performance
comprises al
e are outline
from the wr
success is th
ived to be th
ely this has b
n process, the
lationship is
s over 7 year
e scores in th
he two measu
ng a single
est clinical r
). Similar fin
9
training
s that
cquire
redict
ion
during
ll
ed
ritten
he
he
been
e more
not
rs of
heir
ures
residents
ndings
were r
with re
Wilson
2.1.5.
Exami
offer t
averag
future
of the
certific
the Na
NBME
Admis
exami
and pr
studyi
exami
Howev
exami
1997 a
faculty
Part I
perfor
exami
(much
and hu
signifi
McLau
clinica
reported for
esidents’ ov
n, 2000).
Exam.2
inations scor
the advantag
ge. However
performanc
USMLE par
cation exam
ational Board
E part III (Sw
ssion Test sc
nations is th
ractice quest
ng and expe
nations.
ver, excellen
ned all 66 re
and demonst
y and the Or
Examination
rmance in all
nation score
h weaker than
uman sensiti
icant but poo
ughlin, & W
al performan
a cohort of p
erall rank sc
mination s
res are the m
ge of compar
r, these score
e on upcomi
rt 1 correlate
mination (de V
d of Medical
wanson, Cas
cores, which
he testing of k
tions. It is re
erience-acqui
nt exam scor
esident appli
trated no sig
rthopedic In-
n (Dirschl, C
l programs (n
es. Resident p
n most resid
vity. The co
or, at 0.17 (c
Wright, 2010)
nce, when as
pediatric resi
core during s
cores
most objectiv
rative assessm
es do not pre
ing standard
es well with
Virgilio, Yag
l Examiners
se, & Nunge
h predict USM
knowledge a
asonable tha
ired examsm
res do not ne
ication files
gnificant rela
-Training Ex
Campion, &
n=244) to th
performance
dents) to 5 (m
orrelation bet
clinical acum
). Despite th
sessing whic
idents, wher
selection (r=0
ve assessmen
ment of an in
edict future c
dized examin
performanc
ghoubian, K
(NBME) pa
ster, 1991).
MLE scores
acquired thro
at mastery of
manship may
ecessarily ren
from the Un
ationship betw
xamination (
Gilliam, 200
he Medical C
e was measu
much stronge
tween the M
men) and 0.16
e lack of stro
ch componen
re faculty rat
0.19, p=0.11
nt available t
ndividual w
clinical poten
nations. For i
e on the Am
Kaji, & et al.,
art II is predi
This trend is
(Julian, 200
ough textboo
f these skills
y prove bene
nder excelle
niversity of N
tween reside
OITE) and A
06). Wolosch
Council of C
ured by facul
er than most
MCC examina
6 (human se
ong correlati
nt of the sele
tings only w
1) (Borowitz
to the selecti
ith respect to
ntial. Instead
instance, suc
merican Boar
, 2010). In ad
ictive of per
s also true fo
05). Common
ok readings,
s, coupled w
eficial in futu
ent clinicians
North Caroli
nt clinical ev
American Or
huk et al. co
Canada (MCC
lty on a Like
t residents) f
ation and the
ensitivity) (W
ion between
ection proce
weakly correl
z, Saulsbury,
ion committ
o the nationa
d, they predi
ccessful com
rd of Surgery
ddition, succ
rformance on
or Medical C
n to these
, case-based
ith disciplin
ure written
s. Dirscjle et
ina from 198
valuations b
rthopaedic S
orrelated resi
C) Part 1
ert scale from
for clinical ac
ese scores w
Woloschuk,
n exam score
ess most influ
10
ated
, &
ee. They
al
ict
mpletion
y
cess on
n
College
learning
ed
t al.
83 to
y
Surgery
idency
m 1
cumen
was
es and
uences
overal
signifi
are the
fact th
In sum
exami
Althou
before
have n
2.1.5.
The pu
applic
excel c
differe
compa
perfor
residen
their in
relatio
(Komi
anesth
(Metro
faculty
asserti
contra
and fa
poor a
ll rank score
icant variabl
e most likely
hat exam sco
mmary, an ex
nations, but
ugh passing
e entering ind
no reliable cl
Inter.3
urpose of the
ation. Howe
clinically is
ent medical a
ared intervie
rmance asses
nts from a sp
nstitution ea
onship detect
ives, Weiss,
hesia residen
o, Talarico, P
y interviews
iveness and m
ast, Borowitz
aculty ratings
associations b
, performanc
le in the Unit
y predictor o
res do not ne
xcellent exam
does not nec
the final cer
dependent pr
linical perfor
rview scor
e interview i
ever, the cap
debated. Th
and surgical
ew scores fro
ssments of 5
pectrum of d
ach applicant
ted between
& Rosa, 19
nts’ interview
Patel, & We
that assesse
maturity, wh
z et al. found
s within the
between inte
ce on the US
ted States (S
of receiving a
ecessarily pr
m score may
cessarily pre
rtification ex
ractice, selec
rmance bene
es
is to assess p
ability of int
e predictive
disciples wi
om their form
1 residents,
different spe
t underwent
the their int
84). Similarl
w scores and
etmore, 2005
ed characteri
hich were av
d a weak cor
clinical setti
erview score
SMLE part 1
Stain et al., 2
a top 5 rank
redict clinica
y predict succ
edict who wi
xamination fo
cting trainee
efit (McGagh
personality a
terviews to i
utility of thi
ith predomin
malized selec
in their first
cialties inclu
two intervie
erview score
ly, Metro et
their clinica
5). In this stu
stics such as
veraged to de
rrelation betw
ing (r = 0.27
es and clinica
1 examinatio
2013). There
score at the
al performan
cessful comp
ill become a
for all residen
es based on e
hie, Cohen,
attributes tha
identify resid
is process ha
nantly negat
ction proces
t two clinical
uding medic
ews by facul
es and in-ho
al. demonst
al performan
udy, each app
s personality
etermine the
ween pediatr
7; p=0.02) (B
al performan
on has been r
efore, high sc
time of sele
nce (Stain et
pletion of hi
an excellent c
nts is a nece
examination
& Wayne, 2
at are not ass
dents who ar
as been expl
tive results. K
s to in-hospi
l years. The
ine, surgery
lty members
spital perfor
trated no rela
nce as measu
plicant parti
y aspects, ent
e final intervi
ric residents
Borowitz et a
nce, it has be
reported as th
cores on this
ection, despit
t al., 2013).
igher-level n
clinical resid
ssary millsto
ns scores will
2011).
sessed by the
re best suited
lored in man
Komives et a
ital rating sc
study incorp
and psychia
, with no sig
rmance score
ationship bet
ured by facul
cipated in 4-
thusiasm,
iew score. In
’ interview s
al., 2000). G
een proposed
11
he most
s exam
te the
national
dent.
one
l likely
e written
d to
ny
al.
cale
porated
atry. In
gnificant
es
tween
lty
-5
n
scores
Given the
d that
the in-
applic
2.1.5.
Refere
to asse
faculty
future
referen
post-g
0.12 w
referen
range
profes
obstetr
clinica
ability
One p
who d
reader
2000).
proces
2.1.5.
Many
perfor
-person inter
ant, rather th
Refe.4
ence letters a
ess the candi
y. Reference
clinical perf
nce letter sco
graduate train
was found, re
nce letter sco
of clinical c
ssionalism (r
rics and gyn
al performan
y and work e
otential reas
determined th
rs, calling int
. Overall, alt
ss, these lette
Sum.5
resid
of the existi
rmance, how
rview proces
han predictin
erence lette
are considere
idate’s perfo
e letters, how
formance ac
ores were co
ning in intern
espectively (
ores for gene
ompetencies
r = 0.15)(Bro
necology resi
nce in the cat
ethic (J. G. B
son for the po
hat the interp
to question t
though the u
ers have not
mmary of th
dent perfor
ing elements
wever they co
ss may be mo
ng future clin
ers
ed an import
ormance with
wever, have a
cross a range
orrelated to c
nal medicine
(Curry, Yarn
eral surgery
s including p
others & We
idents also s
tegories of c
Bell, Kanellit
oor predictiv
pretation of t
the reliability
use of referen
proven to b
he current s
rmance
s of the selec
ontinue to pe
ore beneficia
nical perform
tant compon
hin the clinic
also failed to
e of different
clinical perfo
e, only weak
nold, Bryant,
residents we
patient care (
etherholt, 200
howed no si
clinical judgm
tsas, & Shaff
ve value of r
the reference
y of these le
nce letters is
e predictive
selection p
ction process
ersist due to
al in promot
mance (Dubo
nent of the ap
cal setting as
o demonstrat
t medical and
ormance afte
k Pearson co
, Martin, & H
ere found to
(r=0.35), com
07). Furtherm
ignificant rel
ment and acu
ffer, 2002).
reference lett
e letters diff
etters for asse
s deeply root
of future cli
parameters
s do no adeq
their historic
ting the prog
ovsky et al.,
pplication pa
s perceived b
te predictive
d surgical sp
er the first an
orrelations of
Hughes, 198
be weakly c
mmunication
rmore, refere
lationship w
umen, patien
ters is report
fered signific
essment (Dir
ted in the tra
inical perfor
and their a
quately predi
c presence a
gram to the
2008)
ackage, and
by their supe
e value in ter
pecialties. W
nd third year
f r = 0.25 an
88). Similarl
correlated w
n (r=0.26) an
ence letter sc
with faculty-a
nt rapport, su
ted by Dirsc
cantly betwe
rschl & Ada
aditional app
rmance.
ability to p
ict future clin
and lack of n
12
are used
ervising
rms of
When
r of
d r =
y,
with a
nd
cores of
assessed
urgical
chl et al.
een
ams,
plication
predict
nical
new
predic
study t
by foc
assess
is an e
surgeo
further
medic
2.2
There
to ach
trainee
techni
Unfort
describ
This p
exami
Invasi
8.1% o
Simila
laparo
to train
did no
arthro
Alvan
simula
trainin
To the
ctive assessm
to improve t
cusing on tec
ed within th
essential com
on from a me
r ensuring th
ine.
Surgical
is increasing
ieve technic
es between 1
cal proficien
tunately, the
be how the r
phenomenon
ned the learn
ve Surgical
of residents
arly, Schijve
oscopic clip a
nees in ortho
ot reach comp
scopy, with
d, Sunil Aup
ation environ
ng pool is at
e best of my
ment tools to
their predicti
chnical skill.
he realm of se
mponent of su
edical doctor
hat the candi
l trainees
g objective e
al competen
1972 and 200
ncy after com
e authors of t
residents we
has been no
ning curves
Trainer – Vi
did not show
n et al. repor
and cut task
opedic surge
petence in sh
the latter com
plish, Harind
nment, these
risk for not
knowledge,
replace them
ive value, I h
Technical s
election. Op
urgical pract
r. Incorporat
idates selecte
’ variable
evidence tha
nce. Cuschier
02, and foun
mpleting 5 y
this study di
re assessed f
oted in the si
for 37 traine
irtual Reality
w any skill im
rted that 20%
after 30 tria
ery, Alvand e
houlder arth
mmonly con
derjit Gill, &
e results sugg
reaching tec
no studies t
m. Although
have chosen
skill can be o
perating on p
tice and one
ting technica
ed for surgic
e success
at even with
ri et al. perfo
nd that appro
ears of surgi
d not define
for proficien
imulation lab
ees performi
y (MIST-VR
mprovement
% of residen
ls (M. P. Sch
et al. reporte
hroscopy, and
nsidered to b
& Jonathan R
gest that app
chnical comp
o date have
h many areas
n to add to th
objectively m
patients and r
e of the key e
al skill into t
cal training a
in acquir
continued pr
ormed a long
oximately 5-
ical training
e ‘technical c
ncy, limiting
b setting as w
ng 6 distinct
R). The autho
t (Grantcharo
nts did not re
hijven & Jak
ed that after
d 25% did n
be less techn
Rees, 2011).
proximately
petence, even
investigated
s of the selec
he existing lit
measured an
reaching tec
elements tha
the selection
are best suite
ring techn
ractice, not a
gitudinal stu
10% of resid
(Cushchieri
competence’
g interpretatio
well. Grantc
t tasks on th
ors found th
ov & Funch
each proficie
kimowicz, 2
30 repetition
ot reach com
nically deman
Although m
5-35% of th
n with practi
d the implica
ction process
terature in th
d is currentl
hnical comp
at differentia
n may contrib
ed for this ar
nical skill
all trainees a
udy of surgic
dents did no
i, 2003).
nor did they
on of their fi
harov et al.
e Minimally
hat after 10 tr
-Jensen, 200
ency on the
004). With r
ns, 35% of tr
mpetence for
nding (Abtin
mostly limited
e surgical re
ice.
ations of a su
13
s require
his field
y not
petence
ates a
bute to
rea of
are able
cal
t reach
y
findings.
y
rials,
09).
respect
rainees
r knee
n
d to the
esident
ubset of
surgic
reason
frustra
difficu
increa
a non-
most i
compl
terms
identif
a surg
Addin
approa
United
testing
import
either
Grantc
adding
To thi
marke
part of
innate
2.3
2.3.1
For a l
al residents
nable to expe
ation on the p
ult to teach. F
ase frustration
-surgical pro
importantly,
lication rates
of the outco
fying trainee
ical residenc
ng a technica
ach to screen
d Kingdom,
g at the time
tance of test
surrogate m
charov, 2016
g a surrogate
s end, a syst
ers including
f the selectio
technical sk
Potentia
process
Self-s
long time, su
failing to rea
ect that these
part of surge
From the res
n, promote a
ogram or dro
recent evide
s (Birkmeyer
mes of patie
es that will b
cy.
al aptitude as
ning for inco
Ireland and
of selection
ting technica
markers or sim
6). Given the
e technical ap
tematic revie
g personal ch
on criteria fo
kill ability.
al predicto
election as
urgical discip
ach surgical
e individuals
eon-teachers
sident perspe
a sense of in
pping out of
ence suggest
r et al., 2013
ent care. The
be unable to r
ssessment tes
oming traine
Australasia
n for some of
al potential ra
mulated tech
e current No
ptitude test t
ew of the pub
haracteristics
or applicants
ors of tech
s a predicto
plines have t
proficiency
s absorb grea
s and co-resid
ective, strugg
adequacy, an
f clinical trai
ts that poor t
3), potentiall
erefore, there
reach techni
st to the exis
es that may
have all repo
f their surgic
ather than le
hnical tasks (
orth America
to the intervi
blished litera
s and cogniti
to surgical t
hnical ab
or of techn
thought of th
by the end o
ater program
dents as a re
gling to gain
nd increase t
ining all tog
technical ski
ly resulting i
e are several
ical proficien
sting selectio
have difficu
orted ongoin
cal programs
earnt skills fo
(Louridas, Sz
an selection p
iew process
ature was pe
ive tests that
training prog
bility for u
nical abiliy
hemselves a
of training. H
m resources,
esult of being
n technical p
the likelihoo
ether. Final
ill in staff su
in adverse co
l potential ad
ncy, prior to
on process m
ulties in the o
ng use of tec
s. These coun
for incoming
zasz, de Mo
process desc
would be a
erformed to i
t could poten
grams, and t
use during
as distinct fro
However, it
and engende
g perceived
proficiency m
od of transfe
lly, and perh
urgeons can i
onsequences
dvantages to
them embar
may be a feas
operating roo
chnical aptitu
ntries empha
trainees usi
ontbrun, Harr
cribed in sec
feasible app
identify surr
ntially be use
that may pre
g the sele
om medical
14
is
er
as
may
erring to
haps
increase
s in
rking on
sible
om. The
ude
asize the
ng
ris, &
ction 2.1,
proach.
rogate
ed as
dict
ection
15
specialties. Specifically, surgeons take pride in the ability to work well with their hands, often
considering themselves ‘doers’ rather than just ‘thinkers.’ Intuitively, one might expect that
medical students recognize this characteristic of these disciplines, and studies have shown that
students that apply to surgical specialties have a higher self-perceived confidence in dexterity and
their ability to ’work well with their hands‘ as compared to students entering other medical
specialties (Van Hove et al., 2008). Unfortunately, studies have shown that there is no correlation
between students’ subjective self- assessment, and their objective scores on dexterity tests and
simulated surgical task performance. Harris et al. had forty-eight trainees in surgery, psychiatry,
anesthesiology and medicine undergo objective testing of manual dexterity and hand eye
coordination. The authors found no difference in performance between surgical and no-surgical
trainees (Harris, Herbert, & Steele, 1994). With respect to performance on simulated surgical
tasks, Panait et al. compared basic virtual reality (VR) skills in students entering surgical training
to those displayed by residents after a year of internal medicine residency, and found that the
internists performed better on three out of four VR tasks (L. Panait et al., 2011). Cope et al.
assessed 22 interns, where seven of 10 interns interested in surgery rated themselves as naturally
dexterous and only 2 of 12 interns interested in non-surgical disciplines felt they had this ability.
However, no significant differences in performance of basic VR skills tasks were identified
between the groups, suggesting that higher self-perceived natural dexterity does not confer any
objective advantage in technical skill (Cope & Fenton-Lee, 2008). Given these findings, self-
selection cannot be relied upon to ensure that surgical trainees have a high potential for technical
skill performance.
The differences between self-perceived and objective technical aptitudes may be explained by the
relative inexperience of incoming trainees. Even by the end of medical school, students have
typically had limited opportunity to practice surgical skills in the operating room, and are
unlikely to have obtained an objective evaluation of their technical ability that would allow them
to make an informed decision concerning their skills. Therefore, it has become clear that surgical
programs cannot rely on students’ perceptions of their technical skill as a surrogate marker for
their future technical performance during training. Instead, if technical ability is going to be
incorporated into the selection process, then objective assessment of this domain will be essential.
2.3.2
This p
Montb
Ann Su
2.3.2.
Becom
fiscal r
during
often,
2013;
the tea
knowl
Johnso
time a
this im
Furthe
care an
surgic
whole
2007;
2013).
1200 p
burden
year (B
agree t
who ar
progra
likely
Surrog
portion of thi
brun S, Harri
urg. 2015 Ju
Back.1
ming a surge
resources. S
g these years
results in co
Eddleman, A
aching and m
ledge, techni
on, Marquez
and energy to
mpart operati
ermore, altho
nd decrease
al procedure
(Babineau e
Offner, Haw
. The cost of
per hour, (M
n of $53 mil
Bridges & D
that surgical
re ready to e
ams rely on a
to succeed.
gate marke
is chapter ha
is KA, Gran
une 15
kground
on requires p
urgical resid
s, trainees wo
ountless pers
Aoun, & Bat
mentorship o
ical skills, an
z, & Feldman
o deliver form
ive and non-
ough outside
length of sta
es, which im
et al., 2004;
wkes, Maday
f operating ro
Macario, 2010
lion attribute
Diamond, 199
l training is w
enter indepen
a structured
ers as predi
as been bee p
ntcharov T. C
prolonged tr
dency genera
ork within a
sonal sacrific
tjer, 2013; F
of surgical fa
nd surgical j
n, 2013). In
mal and info
-operative su
e of the oper
ay, they sign
mpacts both th
Chamberlain
yag, Seale, &
oom time in
0) resulting i
ed to the ext
99). Most su
worth the sa
ndent practic
selection pro
ictors of fu
previously pu
Can we predi
rainee and fa
ally involves
grueling, fa
ces and high
Franke et al.,
aculty to dire
udgment (Sa
order to fill
ormal teachin
urgical judgm
ating room r
nificantly inc
he surgeon e
n, Patil, Min
& Maines, 20
n the United
in an estima
tra operating
urgical reside
crifices as lo
ce. To ensur
ocess intend
uture techn
ublished as L
ict technical
aculty comm
s 5 to 8 year
ast-paced, hig
h burnout rate
2013). Duri
ect and foste
anfey, Holla
this role, sur
ng and techn
ment to the n
residents imp
crease operat
educator and
nja, & Korde
003; Sasor, F
States is rep
ated cumulat
g room time u
ents, surgica
ong as it pro
re this end go
ded to identif
nical ability
Louridas M,
l aptitude? A
mitment, as w
s of speciali
gh-stakes en
es (M. Arora
ing this time
er the acquisi
ands, & Gant
rgeon educa
nical skill tra
next generati
prove the ef
tive time and
d the health c
ears, 2012; H
Flores, Wood
ported to be a
tive nationw
used to teach
al faculty and
oduces safe, c
oal is reache
fy candidate
y
, Szasz P, de
A systematic
well as consi
zed training
nvironment, w
a, Diwan, &
e, residents r
ition of clini
tt, 2013; Stra
ators dedicate
aining, and t
ion of surgeo
fficiency of p
d the cost of
care system
Harrington et
den, & Thol
approximate
wide annual c
h residents e
d economist
competent su
ed, surgical
s who are m
16
e
review.
derable
and
which
& Harris,
rely on
ical
aus,
e both
through
ons.
patient
f
as a
t al.,
lpady,
ely $900-
cost
each
ts would
urgeons
most
17
The selection processes differ widely between countries and institutions
(Accreditation Council for Graduate Medical Education, 2013; CaRMS, 2016; R. A. C. o.
Surgeons). However, common to all programs is the desire to select a strong cohort of
professionals who are able to learn quickly, work well within the healthcare team, make safe and
appropriate clinical decisions, and have the ability to learn the necessary technical skills to
operate independently. While technical skill is not commonly a part of the selection process in
North America, growing evidence suggests that adding a technical component to the existing
selection process should be considered (Cushchieri, 2003; Grantcharov & Funch-Jensen, 2009;
Mattar et al., 2013; M. P. Schijven & Jakimowicz, 2004). This added element is pertinent to the
modern day trainee who is required to meet technical competency despite restricted resident work
hours, introduction of more complex surgical procedures and more diverse application of difficult
minimally invasive techniques (Biondi et al., 2013; Khatuja et al., 2014; Levine & Spang, 2014;
Richards et al., 2015).
Increasing evidence demonstrates that even with continued practice, not all surgical trainees will
achieve technical competence in the operating room by the end of training. Technical competence
has been defined as the ability to complete tasks or procedures safely and independently (Szasz,
Louridas, Harris, Aggarwal, & Grantcharov, 2015). A single longitudinal study suggested that 5
to 10% of trainees do not reach technical proficiency after completing a 5 year training program
(Cushchieri, 2003). Furthermore, a North American survey of fellowship program directors
showed that 21% of fellows were unprepared for the operating room with 66% unable to
operative independently for more than 30 consecutive minutes (Mattar et al., 2013). In the
simulation setting, a number of studies suggest that somewhere between 8.1 to 20% of residents
do not reach competence despite ongoing practice of the simulated tasks (Abtin Alvand, Sunil
Auplish, Harinderjit Gill, & Jonathan Rees, 2011; Grantcharov & Funch-Jensen, 2009; M. P.
Schijven & Jakimowicz, 2004). This evidence suggests that a proportion of the resident training
pool is at risk for not reaching technical competence. Given the individual and faculty
commitment as well as the fiscal resources required to train residents, adding a measure of
technical skill to the selection process to help identify these individuals and direct them early on
to other medical specialties. This approach will benefit the trainees, the educational system and
the public (Birkmeyer et al., 2013).
To thi
backgr
human
could
2.3.2.
Search
A syst
marke
perfor
Items
structu
condu
1946 t
Hospit
subjec
Proced
‘neuro
‘opera
‘intern
‘applic
Percep
‘techn
‘Perso
‘screen
referen
additio
Eligib
s end, a syst
round charac
n basic perfo
be used to su
Meth.2
h strategy an
tematic revie
ers of innate
rmance, eithe
for Systema
ure of the rev
ucted the sear
to August 30
tal in Toront
ct heading (M
dure’, Opera
osurg*’, ‘exp
ate*or operat
n or interns o
cant*’, ‘stud
ption/’, ‘exp
nical’, ‘techn
onnel Selecti
n*’, ‘accept*
nce search o
onal studies
ility criteria
tematic revie
cteristics, as
ormance test
upplement e
hods
nd criteria
ew was cond
abilities in s
er in the sim
atic reviews a
view (Moher
rch using the
0, 2013, with
to, Canada. T
MeSH) terms
ative/’, ‘surg
p Laparoscop
tive*’ AND
or interne or
dent*’, ‘novi
Suture Tech
nique*’ AND
ion’, ‘exp Ps
*’, ‘intake*’
of the bibliog
that were re
ew of the pub
s well as cog
s) that poten
existing surg
ducted of stu
surgical train
mulation settin
and Meta-An
r, Liberati, T
e online ME
h the assistan
The search w
s and keywo
gery or surge
py’, ‘Laparo
‘exp Educat
internes’, ‘t
ces’ AND ‘e
hniques’, ‘sk
D ‘School Ad
sychological
, ‘predict*’,
graphies of re
levant to thi
blished litera
nitive tests (
ntially predic
gical residenc
udies that hav
nees, and wh
ng or in the
nalyses (PRI
Tetzlaff, & A
DLINE, Psy
nce of a full-
was perform
ords combine
eries or surg
oscopes’, ‘lap
tion, Medica
trainee*’, ‘re
exp Psychom
kill or skills’
dmission Cri
Tests’, ‘Apt
‘determinan
etrieved arti
s review.
ature was pe
(including vi
ct innate tech
cy selection
ve evaluated
hether these a
operating ro
ISMA) state
Altman, 2010
ycINFO and
-time medica
med using the
ed with the A
gical or surgi
paroscop*’,
al/’, ‘residen
egistrar*’, ‘h
motor Perfor
, ‘performan
iteria’, ‘Self
titude/’, ‘sel
nt*’, ‘criterio
cles was per
erformed to i
isual spatial,
hnical ability
criteria.
d association
abilities corr
oom. The Pre
ement was us
0). One revie
Embase onl
al librarian f
e following f
AND functio
ically’, ‘Spec
‘minimal* a
nt* or residen
house officer
rmance’, ‘ex
nce’, ‘ability
f-evaluation
lect*’, ‘prese
on or criteria
rformed to id
identify train
, psychomot
y, and theref
ns between su
relate with te
eferred Repo
sed to guide
ewer (M.L.)
line database
from St. Mic
four sets of m
on: ‘Surgical
cialties, Surg
adj2 surg*’,
ncy or reside
r*’, ‘candida
xp Space
y or abilities’
Program’,
elect*’, ‘apti
a’. A manua
dentify any
18
nee
tor and
fore
urrogate
echnical
orting
the
es from
chael’s
medical
l
gical’,
encies’,
ate*’,
’,
itude*’,
al cross-
19
All original studies that explored the relationship between technical performance in the
simulation setting or the operating room, and innate abilities, background characteristics, and/or
previous operative and non-operative experiences, were considered eligible for inclusion. Studies
that evaluated medical students or surgical trainees (i.e. those that had not yet completed surgical
specialty training) were included involving any of the surgical disciplines. No restrictions were
placed based on the type of technical task evaluated (e.g. open, laparoscopic or endoscopic).
Eligibility was limited to those studies with published abstracts or full text manuscripts available
in English. Review articles, expert opinions, case reports and editorials were excluded.
Data extraction
The following data were extracted: 1) study features including year of publication, study design,
duration and statistical analyses, 2) details of study population including demographics and
sample size, 3) surrogate predictors of technical skill and assessment including participant
characteristics, visual spatial ability, psychomotor ability (excluding surgical simulation tasks
because surrogates designed to simulate operative movements were the focus of this review), and
depth perception, 4) type(s) of surgical setting(s) used for assessment, such as virtual reality
simulators, box trainers, porcine models or patient operations 5) technical skills assessment
measures such as global rating scales, procedures specific checklists, time, or computer generated
outputs including path length, path angle and error scores.
Assessment of methodological quality
The quality of each selected article was individually assessed by two reviewers (M.L. and P.S.)
using the Grading of Recommendations Assessment, Development and Evaluation (GRADE)
system (Guyatt, Oxman, Kunz, et al., 2008; Guyatt, Oxman, Vist, et al., 2008). Using this
framework, quality was assessed by initially stratifying studies by design (randomized trial versus
observational study), followed by ranking the evidence up or down based on five defined
categories including: limitations, inconsistency, indirectness, imprecision and publication bias
(Guya
betwee
Figure
2.3.2.
In tota
118 stu
subseq
random
study
(Dimit
2004;
et al.,
2007;
Madan
Buckle
att G. et al., 2
en the review
e 1: Flow dia
Resu.3
al 8035 citati
udies remain
quently revie
mized contro
(Dashfield, L
triou, Nighti
L. Enochsso
2003; Hassa
Hislop et al
n, Harper, Fr
ey, Frisby, &
2011). Any d
wers.
agram of sea
ults
ions were in
ned potentia
ewed, with 5
olled trials (B
Lambert, Ca
ingale, Khaz
on et al., 200
an et al., 200
., 2006; John
rantzides, &
& Darzi, 200
discrepancie
arch strategy
itially identi
lly eligible f
52 included i
Brandt & Da
ampbell, & W
zali, Hatzige
06; Gettman
07; Hedman,
nson et al., 2
Tichansky,
03; Shah, Pau
s in quality w
y
ified. After s
for inclusion
in the final r
avies, 2006;
Wilkins, 200
orgiades, &
et al., 2003;
Klingberg,
2004; Macmi
2008; Nomu
ul, et al., 200
were resolve
sequentially
n. Full text ve
review (Figu
Hedman et
01), 21 were
Prendiville,
; Gibbons, B
Enochsson,
illan & Cusc
ura et al., 20
03; Steele, W
ed by consen
screening by
ersions of th
ure 1). Of the
al., 2006), 1
cross-sectio
2009; Lars
Baker, & Ski
Kjellin, & F
chieri, 1999;
008; Nugent
Walder, & H
nsus discuss
y title and ab
hese studies w
e 52 studies,
was a pre-p
onal studies
Enochsson e
nner, 1986;
Fellander-Ts
; Madan et a
et al., 2012;
Herbert, 1992
20
ion
bstract,
were
2 were
post
et al.,
Grober
ai,
al., 2005;
; Shah,
2; Tang
21
et al., 2014; Tangchitnob, Solnik, Saad, Rad, & Ogunyemi, 2011) and 28 were prospective cohort
studies (Buckley et al., 2013; Buckley et al., 2014; A. G. Gallagher, Cowie, Crothers, Jordan-
Black, & Satava, 2003; Groenier, Schraagen, Miedema, & Broeders, 2014; Hedman et al., 2006;
Hoffer & Hsu, 1990; M. Keehner, Lippa, Montello, Tendick, & Hegarty, 2006; M. M. Keehner et
al., 2004; Kolozsvari et al., 2011; Masud, Undre, & Darzi, 2012; Marlies P. Schijven,
Jakimowicz, & Carter, 2004; Schueneman, Pickleman, Hesslein, & Freeark, 1984; Van Herzeele
et al., 2010; Van Hove et al., 2008; Wanzel, Hamstra, Anastakis, Matsumoto, & Cusimano, 2002;
Wanzel et al., 2003; White & Welch, 2012). When assessed using the GRADE system, the 2
included RCTs were rated as high quality evidence, 31 studies as low quality evidence, and the
remaining 20 as very low quality evidence. The most common reason for downgrading the
quality score was a lack of blinding of assessors when using a subjective assessment instrument
(20 of 52 studies). A detailed summary of the quality assessment organized by subjective or
objective assessment method can be found in Table 1.
Potential predictors of technical ability identified in these studies can be divided into: 1)
information generated from participant questionnaires such as background characteristics, non-
surgical experiences, and surgical experiences (Table 2); and 2) validated cognitive tests designed
to test innate visual spatial ability (Table 3), dexterity (Table 4), human basic performance
resources, and other related characteristics (i.e. depth perception, working memory).
Participant questionnaires
A total of 14 studies attempted to predict surgical performance based on responses to participant
questionnaires. Of the 23 potential predictors studied, only video gaming consistently showed a
significant correlation with initial technical skill. However, Paschold and Dimitriou both found
that gamers lost their initial technical advantage once non-gamers were given the opportunity to
practice (Dimitriou et al., 2009; Paschold et al., 2011). All the other characteristics collected by
participant questionnaires (e.g. age, handedness, experience typing, sports etc.) failed to
consistently predict technical performance (Table 2) (Banerjee, Cosentino, Hatzmann, & Noe,
2010b) .
22
Cognitive tests
Four different categories of cognitive tests, aimed at testing innate abilities perceived to be
important in acquiring surgical skill, were identified: 1) visual spatial, 2) dexterity, 3) human
basic performance resources and 4) other.
Visual spatial
A total of 38 studies using twenty-five different visual spatial tests (VST) were identified that
have been evaluated in terms of their ability to predict technical performance. Of the 25 VSTs, 2
have repeatedly shown a positive relationship with technical performance. Specifically: the
PicSOR test of perceptual skill (5 of 8 studies) (Buckley et al., 2013; Buckley et al., 2014; A. G.
Gallagher et al., 2003; M. Keehner et al., 2006; Kolozsvari et al., 2011; McClusky, Ritter,
Lederman, Gallagher, & Smith, 2005; E. Matt Ritter, McClusky, Gallagher, Enochsson, & Smith,
2006; D. Stefanidis, Korndorffer Jr, et al., 2006), and the mental rotation test (MRT) (6 of 9
studies) (Brandt & Wright, 2005; Deary, Graham, & Maran, 1992; Groenier et al., 2014; Hedman
et al., 2007; M. Keehner et al., 2006; Luursema, Buzink, Verwey, & Jakimowicz, 2010; D.
Risucci, Geiss, Gellman, Pinard, & Rosser, 2001; Wanzel et al., 2002; Wanzel et al., 2003).
When stratified by technical skill category, PicSOR demonstrated a positive relationship for
laparoscopic skills learned in the box trainer and virtual reality simulator.(Buckley et al., 2013;
Buckley et al., 2014; A. G. Gallagher et al., 2003; Kolozsvari et al., 2011; McClusky et al., 2005)
In contrast, MRT has demonstrated a positive relationship with open surgical skills.(Wanzel et
al., 2002; Wanzel et al., 2003) All other visual spatial predictors were either only evaluated in a
single study, or failed to show a relationship with technical performance in the majority of studies
(Table 3) (Buckley et al., 2013; Buckley et al., 2014; Deary et al., 1992; Dimitriou et al., 2009; L.
Enochsson et al., 2006; Groenier et al., 2014; M. Keehner et al., 2006; M. M. Keehner et al.,
2004; Kolozsvari et al., 2011; Luursema et al., 2010; McClusky et al., 2005; Murdoch,
Bainbridge, Fisher, & Webster, 1994; Neumann et al., 2005; Nugent et al., 2012; D. A. Risucci,
2002; E. Matt Ritter et al., 2006; Marlies P. Schijven et al., 2004; Schueneman, Pickleman, &
Freeark, 1985; Schueneman et al., 1984; Steele et al., 1992; D. Stefanidis, Korndorffer Jr, et al.,
2006; Tang et al., 2014; Van Herzeele et al., 2010; Wanzel et al., 2002).
23
Dexterity tests
A total of 19 studies using twenty different surrogate tests of dexterity were identified. An
adaptive tracking task (ADTRACK2) was significantly associated with technical performance in
2 of 2 studies, one testing performance on the reef knot and endoscopic sinus surgery (Dashfield
et al., 2001; Dashfield & Smith, 1998). The grooved pegboard test was significantly associated in
4 of 5 studies, however, in these studies, the significant association was either limited to the
initial trial of the task, or to a single task sub-score in the areas of laparoscopy and endoscopy
(Nugent et al., 2012; D. Stefanidis, Korndorffer Jr, et al., 2006; Van Herzeele et al., 2010). The
seventeen remaining dexterity tests were either only evaluated by single study or failed to find a
relationship with technical performance (Table 4) (Buckley et al., 2013; Buckley et al., 2014;
Hoffer & Hsu, 1990; Macmillan & Cuschieri, 1999; Masud et al., 2012; Murdoch et al., 1994;
Neumann et al., 2005; Nugent et al., 2012; Marlies P. Schijven et al., 2004; Schueneman et al.,
1985; Schueneman et al., 1984; Steele et al., 1992; D. Stefanidis, Korndorffer Jr, et al., 2006; Van
Herzeele et al., 2010; Wanzel et al., 2003).
Human basic performance resources
Three studies evaluated association between human basic performance resources (BPRs) and
technical performance. BPRs are a group of simple tests including: simple visual-hand response
speed, visual information processing speed, upper extremity neuromotor channel capacity, upper
extremity steadiness and grip strength. Overall, BPRs accurately predicted technical performance
in 62 to 75% of cases. They over predicted performance in 14 to 17% of cases, and under
predicted performance in 18 to 21% of cases in the areas of laparoscopic and endoscopy
(Gettman et al., 2003; Johnson et al., 2004; Matsumoto et al., 2006).
24
Other innate abilities
Twelve additional tests of innate ability, which did not fit into the aforementioned categories,
were identified. Two of these demonstrated a positive association with technical performance,
namely: tonic accommodation, defined as a stable parameter that the eye adopts in the absence of
stimulation; and abstract reasoning, which investigates an individual’s non-verbal reasoning and
is related to intelligence quotient (M. Keehner et al., 2006; Marlies P. Schijven et al., 2004; Shah,
Buckley, et al., 2003). Tonic accommodation was significantly correlated with the error score of
the right hand on a virtual reality simulator. However, after the 5th repetition, the relationship
was no longer significant (Shah, Paul, et al., 2003). Abstract reasoning was shown to have a
positive correlation with laparoscopic cholecystectomy performance on a virtual reality simulator
(Marlies P. Schijven et al., 2004). The remaining 10 tests failed to demonstrate an association
with technical performance. These included: information management (Dashfield & Smith,
1998), stereoscopic vision (Deary et al., 1992), verbal reasoning (Groenier et al., 2014), working
memory (Hedman et al., 2006), flow and mental strain (Hedman et al., 2006), organizational
planning (Van Herzeele et al., 2010), memory test (Hedman et al., 2006), flexibility of closure
(Luursema et al., 2010), personality test (Neumann et al., 2005) and vigilance endurance test
(Neumann et al., 2005).
25
Table 1: GRADE classification and assessment method of included studies organized by subjective or objective assessment method
Study design
Limitations Precision Consistency Directness Publication
bias Grade
*Studies n=52
Objective assessment
2 RCT not serious no imprecision Consistent direct not detected high 31,32
22 Obs not serious no imprecision Consistent direct not detected low
71,83,39,40,63, 32,46,67,69,77,73,82,51,52,80,74,54,34,72,37,5
8,60
5 Obs not serious imprecision Consistent direct not detected very low 64,44,59,70,35
Subjective assessment
2 Obs not serious no imprecision Consistent direct not detected low 43,53
13 Obs serious no imprecision Consistent direct not detected very low 42,33,75,38,41,66,47,68,48,84,81,55,79
2 Obs not serious imprecision Consistent direct not detected very low 36,56
Combination of objective and subjective assessment
2 Obs not serious no imprecision Consistent direct not detected low 61,62
4 Obs serious no imprecision Consistent direct not detected very low 49,50,65,57
RCT - Randomized controlled trial; Obs - Observational study (cross-sectional, cohort, case-series, pre-post quasi-experimental); *Studies - references within the body of the manuscript.
26
Table 2: Summary of background characteristics, surgical and non-surgical experiences as predictors of surgical performance collected by participant questionnaires
Count Potential predictors (n=23) Number
of studies
Total number of participants (number of
participants in significant studies)
Number of studies reporting a significant association
(percent; weighted percent)
Number of studies that failed to identify a significant
association (percent; weighted percent)
*Studies n=14
Background characteristics
1 Gender 9 493 (272) 3 (33.3; 55.2) 6 (66.6; 44.8) 71,49,60,38,58,79,
74,72,69
2 Handedness 6 651 (0) 0 6 (100.0; 100.0) 71,50,70,60,38,79
3 Age 5 548 (35) 1 (20.0; 6.4) 3 (80.0; 93.6) 71,70,60,79,72
5 Surgical career aspirations 3 401 (32) 1 (33.3; 8.0) (66.6; 92.0) 70,69,51
6 Self-reported motor skills 3 390 (43) 1 (33.3; 11.0) 1 (66.6; 89.0) 72,70,51
7 Glove size 3 57 (11) 1 (33.3; 19.3) 1 (66.6; 80.7) 71,74,72
9 Weight 2 46 (0) 0 2 (100.0; 100.0) 71,72
10 Vision 2 36 (0) 0 2 (100.0; 100.0) 71,74
11 Height 2 46 (0) 0 2 (100.0; 100.0) 71,72
12 Completion of surgical internship 1 326(0) 0 1 (100.0; 100.0) 70
Non-surgical experiences
13 Gaming experience (TV/video/computer) 9 673 (32) 7 (88.9; 95.2) 2 (11.1; 4.8) 69,46,50,51,70,60,
38,37,72
14 Musical instrument 7 604 (0) 0 7 (100.0; 100.0) 71,49,50,51,70,60,
38
27
15 Typing 3 175 (0) 0 3 (100.0; 100.0) 49, 50, 38
16 Sport 2 82 (0) 0 2 (100.0; 100.0) 71,38
17 Chopsticks 2 118 (67) 1 (50.0; 56.8) 1 (50.0; 43.3) 49, 50
18 Sewing 2 118 (0) 0 2 (100.0; 100.0) 49,50
19 Driving 2 100 (43) 1 (50.0; 43.0) 1 (50.0; 57.0) 51,38
20 Experience operating tools 2 118 (0) 0 2 (100.0; 100.0) 49,50
21 Billiards 1 21(21) 1 (100.0;100.0) 0 72
Surgical experiences
22 Laparoscopic experience 2 46(25) 1 (50.0;54.3) 1 (50.0; 45.65) 71,72
23 Endovascular experience 1 61(0) 1 (100.0;100.0) 0 46
*Studies - references within the body of the manuscript
28
Table 3: A summary of visual spatial tests as predictors of surgical performance
Description of innate ability
Number of studies
Total number of participants (number of
participants in significant
studies)
Number of studies
reporting a significant association (percent; weighted percent)
Number of studies that failed to
identify a significant association
(percent; weighted percent)
Number of studies stratified by type of technical skill
Number of studies reporting a significant
association stratified by technical skill category
(percent)
Number of studies that failed to
identify a significant association stratified
by technical skill category (percent)
*Studiesn=38
Count Visual spatial test (n=25)
1 Card rotation A picture of a 2-dimensional shape is set as the reference figure. Participants are then required to indicate whether they need to flip or rotate eight additional 2-dimensional shapes in order to match the reference figure. The test is completed in a set time and a score is generated on accuracy.
10 229 (68) 5 (50.0;30.0) 5 (50.0;70.0)
Open surgery
Laparoscopic skills
VR laparoscopy
VR endoscopy
1
2
6
2
0 (0.0) 1 (50.0)
3 (50.0)
1 (50.0)
1 (100.0) 1 (50.0)
3 (50.0) 1 (50.0)
75,73,$72,40,77,69,52,61,62
,64
2 Mental rotation test
Participants are shown two three-dimensional images and asked to compare the images and state whether they are the same image or the mirror image. The test is completed in a set time and a score is generated on accuracy.
9 389 (288) 6 (66.7;74.0) 3 (33.3;26.0)
Open surgery Laparoscopic skills VR laparoscopy
VR endoscopy
413
2
3 (75.0) 1 (100.0)
1 (33.3)
1 (50.0)
1 (25.0) 0 (0.0)
2 (66.7)
1 (50.0)
75,76,57,56,$45,77,64
,67,80
3 PicSOR Participants are required to move a spinning arrow on top of a cube, until the angle between the two objects is 90 degrees. The closer the approximate angle is to the actual angle the higher the score.
8 255 (206) 5 (62.5;80.8) 3 (37.5;19.2)
Laparoscopic skills VR laparoscopy
VR endoscopy
3
4
2
2 (66.7)
3 (75.0)
0 (0.0)
1 (33.3
1 (25.5)
2 (100.0)
63,73,39,74,72,69,
61,62
29
4 Cube comparison
Participants are asked to compare two three dimensional cubes with a letter on each surface and indicate whether they are the same cube or a different cube. The test is completed in a set time and a score is generated on accuracy.
8 181 (55) 3 (37.5;30.4) 5 (62.5;69.6)
Open surgery Laparoscopic skills VR laparoscopy VR endoscopy
2241
0 (0.0) 0 (0.0)
2 (50.0) 1 (100.0)
2(100.0) 2 (100.0) 2 (50.0) 1 (100.0)
75,56,73,74,$72,69
,61,62
5 Map planning Participants need to find the shortest route between two points while avoiding road blocks and passing along the side of the building. This task is scored on time and accuracy.
8 163 (74) 4(50.0;45.4) 4 (50.0;54.6)
Open surgery Laparoscopic
skills VR laparoscopy VR endoscopy
VR endovascular
22601
0 (0.0) 0 (0.0)
3 (50.0) 0 (0.0)
1 (100.0)
2 (100.0) 2 (100.0) 3 (50.0) 0 (0.0)
1 (100.0)
75,73,$72,65,69,52
,61,62
6 Surface development
Participants are given a 2-dimensional blueprint to fold in three-dimensional space and match to the appropriate 3-dimensional picture. This task is scored on time and accuracy.
5 164 (91) 3(60.0;44.5) 2 (40.0;55.5)
Open surgery Laparoscopic skills VR laparoscopy
203
1 (50.0) 0 (0.0)
2 (67.7)
1 (50.0) 0 (0.0)
1 (33.3)
75,57,61,62,64
7 Minnesota paper form board test
Participants are given a set of different parts and are required to choose which of the 5 arrangements could be made up of these parts. This task is scored on time and accuracy.
5 339 (157) 2 (40.0;46.3) 3 (60.0;53.7) Open surgery Laparoscopic skills
41
2 (50.0) 0 (0.0)
2 (50.0) 1 (100.0)
55,79,$72,75,56
8 Thurstone hidden figures test
Participants are given a reference image and are required to indicate whether the reference image is imbedded in the subsequent more complex image. Time and number of correct responses is the final score.
5 344 (73) 1 (20.0;27.2) 4 (80.0; 78.8) Open surgery VR endoscopy
41
1 (25.0) 1 (100.0)
3 (75.0) 0 (0.0)
42,36,77,55,79
9 Paper folding A piece of paper is folded in a specific way and then hole punched through. The participant uses this reference image to match which piece of paper would correspond to the paper when unfolded. Time and number of correct responses is the final score.
3 121 (48) 1 (33.3;39.7) 2 (66.7;60.0)
Open surgery Laparoscopic skills VR laparoscopy
1
1
1
0 (0.0) 1 (100.0)
0 (0.0)
1 (100.0) 0 (0.0)
1 (100.0) 75,68,64
10 Gestalt completion test
Participants are given a drawing of a fragmented object and are required to try and identify what it is. Time and number of correct responses is the final score.
3 191 (107) 1(33.3;56.0) 2 (66.6;44.0) Open surgery Laparoscopic skills
21
(0.0) 1 (100.0)
2 (100.0) 0 (0.0)
56,57,80
30
11 Guay’s visualization of viewpoints test
Participants are shown a picture of a cube with an image inside the cube. They are then shown the same cube in a different orientation and required to identify the corners of the cube where the new view was taken. Time and number of correct responses is the final score.
2 22(22) 1 (100.0;100.0) 0 VR laparoscopy 1
1 (100.0)
0 (0.0) 67
12 Rey-Osterrieth complex figure test
Participants are asked to refer to a complex line drawing and first draw it out by referring to the original image and then again by memory. Accuracy is the final score.
2 41 (20) 1 (50.0;48.8) 1 (50.0;51.2) Laparoscopic skillVR laparoscopy VR Endovascular
111
0 (0.0) 0 (0.0)
1 (100.0)
1 (100.0) 1 (100.0)
0 (0.0) 72,65
13 Space relations test
Participants are required to compare letters, numbers and/or objects quickly and accurately in a certain amount of time. Time and number of correct responses is the final score.
2 70 (37) 1 (50.0;52.9) 1 (50.0;47.1) Open surgery VR laparoscopy
11
1 (100.0) 0 (0.0)
0 (0.0) 1 (100.0)
81,54
14 Perceptual speed
Participants are given a two-dimensional pattern and asked to fold it and rotate the shape in order to match it to a three-dimensional image. Time and number of correct responses is the final score.
2 68 (15) 1 (50.0; 22.1) 1 (50.0;77.9) VR laparoscopy VR endoscopy
11
0 (0.0) 1 (100.0)
1 (100.0) 0 (0.0)
77,64
15 4 "magic eye" images
Participants are given four separate three dimensional images hidden within a 2-dimensional pattern and are asked to identify each image. Time and number of correct responses is the final score.
1 57 (0) 0 1 (100.0;100.0) Laparoscopic skills
1 0 (0.0) 1 (100.0) 38
16 Adapted Corsi Block Tapping Test
A monitor displays nine dice cubes in random position. A certain number of these dice are highlighted in a given order, the participant is then required to point out these dice in the same order. If more than three are correct the number of dice increases on the subsequent turn.
1 53 (0) 0 1 (100.0;100.0) VR laparoscopy 1 0 (0.0) 1 (100.0) 64
17 Choosing a path
The participant is given a diagrammed imagine of a city map. They must then plan routes between two set points
1 20 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 75
31
and avoid any roadblocks. This task is scored on time and accuracy.
18 Five porteus maze
Participants are required to trace their way through a complex maze, without crossing solid lines until they arrive at the exit point. This task is scored on time and accuracy.
1 140(0) 0 1 (100.0;100.0) Open surgery 2 0 (0.0) 2 (100.0) 55,75
19 Matrix reasoning
Participants are required to fill in a missing shape within a group of shapes and are given a number of choices. This task is scored on time and accuracy.
1 21 (0) 0 1 (100.0;100.0) Laparoscopic skills VR laparoscopy
1
1
0 (0.0) 0 (0.0)
1 (100.0) 1 (100.0
$72
20 Phase discrimination test
Participants are required to discriminate a one-dimensional pattern presented in two-dimensional noise. This exercise is scored on time and accuracy.
1 47 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 57
21 Shape memory test
Three shapes are presented to the participants, then hidden. The participant is then required to identify the same three shapes in the same order by memory. This task is scored on time and accuracy.
1 37 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 56
22 Touching blocks test
An image of three dimensional boxes, of difference shapes and sizes, are stacked onto one another. Fix grammar Each box is a given a number and the participant is required to state how many pieces are being touched by a given box. Time and number of correct responses is the final score.
1 107 (107) 1 (100.0;100.0) 0 Laparoscopic skills
1 1 (100.0) 0 (0.0) 80
23 Tube shape test
An initial image is shown in a transparent tube. A second image is shown from a different perspective, then the participant is required to choose from a few options which perspective the second image is being shown from. Time and number of correct responses is the final score.
1 58 (0) 0 1 (100.0;100.0) VR endoscopy 1
0 (0.0)
1 (100.0) 82
32
24 Snowy picture test
Participants are asked to identify a picture covered with a visual obstruction, as quickly as possible. Time and number of correct responses is the final score.
1 37 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 56
25 Soap carving A picture of a side view, top view and bottom view are given to the participant who must then visualize these views in three dimensions to carve out a cylinder-shaped piece of soap. A score is given for the accuracy of the final soap carving.
1 96 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 53
PicSOR - Pictorial Surface Orientation; VR – Virtual Reality; *Studies - references within the body of the manuscript; $more than one technical skill category contained in this study
Table 4: Summary of dexterity tests as predictors of surgical performance
Count Dexterity test (n=20)
Description of innate ability Number
of studies
Total number of participants (number of
participants in significant
studies)
Number of studies reporting a significant association
(percent; weighted percent)
Number of studies that failed to
identify a significant association
(percent; weighted percent)
Number of studies stratified
by type of technical skill
Number of studies reporting a significant association stratified by
technical skill category (percent)
Number of studies that failed to
identify a significant association stratified
by technical skill category (percent)
*Studiesn=19
1 Purdue pegboard
This task is completed with a board with two parallel rows of holes (25 in total). Using their hands, the participant is required to place cylindrical shaped pegs into the holes on the board as quickly as possible.
6 399 (56) 2 (33.3;14.0) 4 (66.7;86.0)
Open surgery Laparoscopic skills VR laparoscopy VR endovascular
4111
1 (25.0) 0 (0.0) 0 (0.0)
1 (100.0)
3 (75.0) 1 (100.0) 1 (100.0) 0 (0.0)
65,66,72,59,55,79
2 Grooved peg board
A square board with 25 holes (5 holes across and 5 holes down) is used. Using their hands, the participants are required to place the metal pegs into the holes on the board as quickly as possible.
5 95 (74) 4 (60.0;77.9) 1 (40.0;22.1)
Laparoscopic skills VR laparoscopy VR endoscopy
141
0 (0.0) 3 (75.0)
1 (100.0)
1 (100.0) 1 (25.0) 0 (0.0)
52,72,62,65,61
33
3 Crawford Small Parts Dexterity test
A board with holes for pins and screws is presented to the participants. Using tweezers, the pins are placed into their designated holes and collars are placed on the pins sticking out. The screws are then placed into their designated holes and a screwdriver is used to screw down each screw until flush with the board. Participants are asked to complete the task as quickly as possible.
4 98 (8) 1 (25.0;8.2) 3 (75.0;91.8) Open surgery VR laparoscopy
31
2 (66.7) 0 (0.0)
1 (33.3) 1 (100.0)
57,54,66,36
4 ADTRACK2 A screen displays a block bounded by two bars. The bars move from side to side and the participant uses a joystick to keep the block as close to the bars as possible. The closer the block stays to the bars, the higher the score.
2 31 (31) 2 (100.0;100.0) 0 Open surgery Endoscopy
11
1 (100.0) 1 (100.0)
0 (0.0) 0 (0.0)
83,33
5 Block Design Participants are given a pattern and are required to rearrange blocks with their hands in order to recreate the pattern. The final score is calculated from accuracy and speed.
2 261 (0) 0 2 (100.0;100.0) Open surgery 2 0 (0.0) 2 (100.0) 55,79
6 Porteus Maze Participants are required to trace through a maze from one end to the other, avoiding dead ends or backtracking. A time limit is set depending on the complexity of the maze.
2 261 (0) 0 2 (100.0;100.0) Open surgery 2 0 (0.0) 2 (100.0) 55,79
7 Tactual Performance
Participants are blindfolded and asked to place a number of cut out shapes into their corresponding positions on a form board. The participant completes the task three times; with their left hand, right hand and both hands together. Time is used for the final score.
2 261 (0) 0 2 (100.0;100.0) Open surgery 2 0 (0.0) 2 (100.0) 55,79
8 ADEPT Participants read off a computer screen while working within a dome with laparoscopic instruments and a camera. Four tasks are completed. The error plate detects excessive movement and a score is generated.
1 10 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 48
9 ADTRACK3 A screen displays a block bounded by two bars. The bars move from side to side and the participant uses a joystick to keep the block as close to the bars as possible. The task differs from ADTRACK2 in that the participant can adjust the difficulty. The higher the level and the closer the block stays to the bars, the higher the score.
1 16 (16) 1 (100.0;100.0) 0 Endoscopy 1 1 (100.0) 0 (0.0) 83
34
10 Bimanual coordination test
The participant uses two controls to navigate two points through a labyrinth. The right hand moves the points up and down and the left moves the points side to side. Accuracy and time generate a final score.
1 58 (0) 0 1 (100.0;100.0) VR endoscopy 1 0 (0.0) 1 (100.0) 82
11 Bennett hand tool dexterity test
Screws, nuts and bolts are secured tightly into a wooden frame. Using the tools provided the participant is required to take apart the units and reassemble them on the opposite side. Time is used for the final score.
1 8 (0) 0 1 (100.0;100.0) Open surgery Laparoscopic skills
2
1
0 (0.0) 1 (100.0)
2 (100.0) 0 (0.0)
66
12 Reaction time Participants hold down a button; then when a neighboring button lights up they are required to tap it. Cumulative response time is used for the score.
1 21 (0) 0 1 (100.0;100.0) VR laparoscopy
1
1 (100.0)
0 (0.0) 72
13 Double labyrinth test
The participant uses two leavers to control the position of two markings within a cylinder, which rotates at a constant speed. The right hand controls the right point and the left hand controls the left point. If a marking touches the side of the screen then an error is recorded.
1 58 (58) 1 (100.0;100.0) 0 VR endoscopy 1 1 (100.0) 0 (0.0) 82
14 Finger tap test Participants place their hand palm down over a board with their index finger overlaying a device that counts the number of finger taps. For 10 seconds at a time they are required to tap as quickly as possible for 3-6 repetitions. This test is completed with the left and right hand. The more finger taps completed in the allotted time the higher the score.
1 12 (21) 0 1 (100;100) Laparoscopic skills VR laparoscopy
1
1
0 (0.0) 0 (0.0)
1 (100.0) 1(100.0)
72
15 Gibson spiral maze test
Participants are given a circular maze and are required to trace between the lines. Score is determined by time and error.
1 10 (0) 0 1 (100.0;100.0) Open surgery 1 0 (0.0) 1 (100.0) 36
16 Minnesota rate manipulation test
The participant is required to place a number of discs onto a large board in a specific series. Once completed, the participant then flips over each disc in series moving along each row in a consecutive manner until all the discs have been flipped.
1 8 (0) 0 1 (100.0;100.0) Open surgery 1
0 (0.0) 1 (100.0) 66
17 Steadiness hole test
The participant is required to place a metal tip stylus into 9 progressively smaller holes without touching the edges. This task is scored on time and error.
1 8 (8) 1 (100.0;100.0) 0 Open surgery 1
0 (0.0) 1 (100.0) 66
35
18 Tremor Participants grab a needle with a laparoscopic instrument that is attached to an oscillator and hold it steady for 20 seconds. Steadiness is used as the final score.
1 21 (0) 0 1 (100.0;100.0) Laparoscopic skills VR laparoscopy
11
0 (0.0) 0 (0.0)
1 (100.0) 1(100.0)
72
19 Wire Loop Dexterity Test
The participant isrequired to pass a hand held loop over a wire that has three formed bends. The goal of the task is to move from one end of the wire to the other without making contact with the loop.
1 37 (37) 1 (100.0;100.0) 0 Open surgery 1 0 (0.0) 1 (100.0) 81
ADEPT - Advanced Dundee Endoscopic Psychomotor Tester; VR – Virtual Reality; ADTRACK - Adaptive Tracking Task; *Studies - references within the body of the manuscript
2.3.2
Train
resou
other
proce
daily
predi
select
of eff
abilit
studie
tests,
comb
Of th
small
Notab
groov
as a p
playin
short
the re
unwa
2003
gamin
game
Van H
advan
skills
manu
dexte
Dis2.4
ning resident
urces. Furthe
r selection cr
edures and in
practice (Bi
cting technic
tion process
fort has been
ties thought t
es that have
3 basic perf
bination of te
he surrogate m
l minority re
ble among th
ved pegboard
potential mar
ng experienc
-term advan
esult of their
atched hands
; Hislop et a
ng advantag
ers’ technica
Hove et al., 2
ntage can be
s training ma
ual dexterity
erity as quan
scussion
ts requires co
ermore, selec
riteria given
ncreasing inc
iondi et al., 2
cal aptitude
to benefit a
n dedicated t
to be require
investigated
formance res
ests has been
markers inve
egularly dem
hese are prev
d dexterity t
rker of super
ce may be m
tage. Gamer
r familiarity
s, as well as
al., 2006; Ro
ge was lost af
l skill impro
2008). Toge
e quickly ove
ay also overr
testing. Van
ntified using
onsiderable t
ction for tech
resident wor
corporation
2013; Khatuj
were possib
ll parties inv
to identifying
ed for trainee
d 23 participa
source tests a
n identified t
estigated by
monstrated a p
vious video
est. Althoug
rior innate te
more appropr
rs appear to d
with reactin
increased att
sser et al., 2
fter initial ba
oved slower w
ether, these f
ercome by m
ride any base
n Herzeele et
the grooved
trainee and f
hnical aptitu
rk hour restr
of more diff
uja et al., 201
le, this comp
volved. It is t
g an objectiv
es to reach c
ant character
and 12 other
to validly an
participant q
positive rela
game playin
gh some auth
echnical abil
riately consid
demonstrate
g to 2-dimen
tention and f
007). Howev
aseline testin
when compa
findings sugg
minimal techn
eline advanta
t al. reported
d pegboard te
faculty comm
ude has gaine
rictions, grow
ficult minima
14; Levine &
ponent could
therefore no
ve test to me
competent te
ristics, 25 vi
r innate abili
nd reliably pr
questionnair
ationship wit
ng experienc
hors have co
lity, evidenc
dered a learn
e superior tec
nsional feedb
faster reactio
ver, Van Ho
ng, and Dim
ared to non-g
gest that the
nical skills t
age associat
d a significan
est, and base
mitment and
ed increased
wing comple
ally invasive
& Spang, 201
d be added t
ot surprising
easure the un
echnical perf
isual spatial
ity tests, no
redict techni
res and dexte
th technical p
ce, and perfo
nsidered vid
ce suggests th
ned skill that
chnical abilit
back from a
on time (Gre
ove et al. rep
mitriou et al. f
gamers (Dim
initial gami
training. Sim
ed with high
nt correlatio
eline perform
d substantial
d relevance a
exity of surg
e techniques
14). Therefo
o the current
that a large
nderlying inn
formance. De
tests, 20 dex
single test o
ical aptitude
erity testing,
performance
rmance on th
deo game exp
hat video ga
t only confer
ty, and this m
monitor usi
een & Bavel
ported that th
further noted
mitriou et al.
ng performa
milarly, techn
her performa
on between m
mance on a v
36
fiscal
amongst
gical
s into
re, if
t
amount
nate
espite 52
xterity
r
.
, only a
e.
he
perience
ame
rs a
may be
ing
ier,
he
d that
, 2009;
ance
nical
ance on
manual
virtual
37
reality endovascular renal artery stenosis task. However, the predictive value of the dexterity text
was lost after participants were given the opportunity to practice this task (Masud et al., 2012).
Thus, it appears that some surrogate tests that have been presumed to reflect innate ability may in
fact be assessing learned skills, limiting their value as predictors of technical performance by
surgical trainees.
It would seem intuitive that tests of visuospatial ability would reliably predict technical ability,
given surgeons’ need to operate around complex structures in three-dimensional space. One
might expect this relationship to be particularly strong in the case of laparoscopic procedures,
where surgeons operate in three dimensions while relying on two-dimensional visual feedback.
Therefore it is particularly surprising that of the 26 visual spatial tests studied to date, only 3
(12%) (the Card Rotation Test, the Mental Rotation Test(MRT), the PicSOR test) have
consistently demonstrated a positive relationship with technical ability.
While there may be a perception that visuospatial ability is a single homogenous aptitude that
varies between individuals, the results of the present review suggest that different visuospatial
tests may assess distinct and discrete abilities, each of which may only apply to a specific subset
of surgical procedures. For example, while the MRT was positively correlated with technical
performance in the majority of studies, this association differed depending on the type of surgical
task. MRT results significantly predicted technical performance in all studies involving open
surgical tasks in a simulated environment. A positive correlation was seen between MRT scores
and performance of internal fixation of a mandible fracture (Wanzel et al., 2003), 4 flap z-plasty,
(Wanzel et al., 2002), and with tying of an open surgical reef knot (Brandt & Davies, 2006). In
contrast, when assessed as potential predictors of simulated endoscopic and laparoscopic tasks
performance, the results were inconsistent, with half of studies finding no predictive value
(Groenier et al., 2014; Hedman et al., 2006; M. M. Keehner et al., 2004; Luursema et al., 2010).
Similar discrepancies are noted with the predictive value of the PicSOR test to predict minimally
invasive performance. Although this test was developed specifically to test the visual spatial
abilities required for laparoscopy, one might expect it to also be equally effective in predicting
technical performance on endoscopic tasks, given the similar minimally invasive nature of both.
However, PicSOR has been successful in predicting simulated laparoscopic tasks but not
simulated endoluminal tasks in studies to date. Specifically, PicSOR has demonstrated a positive
correlation with a laparoscopic cutting task (A. G. Gallagher et al., 2003), manipulation and
38
diathermy task (McClusky et al., 2005), and laparoscopic peg transfer in the novice trainee
(Kolozsvari et al., 2011), but no correlation with the endobubble and gastroscopy tasks on the
virtual reality simulator (L. Enochsson et al., 2006; E. Matt Ritter et al., 2006). Therefore,
although visual spatial ability is likely important for technical performance, the surrogate tests
available seem to predict performance on specific subgroups of surgical tasks instead of
reflecting underlying abilities that transfer across surgical procedures in general.
Given that the purpose of this systematic review was to focus on aptitude testing for selection
into training, open and minimally invasive techniques have been discussed together. This is
appropriate for contemporary surgical practice, where a number of specialties (e.g general
surgery, urology) require that surgeons be proficient with a mix of both open and minimally
invasive techniques, and this is also reflected in training curricula. However, the literature does
suggest that different tests may be more appropriate for predicting aptitude for either open or
minimally invasive techniques. These tests may ultimately be able to provide some insight into
whether surgeons would benefit from biasing their independent practices following completion
of training toward either open or minimally invasive procedures. Furthermore, some surrogate
markers seem to be more appropriately suited for specific surgical specialties. For example MRT
has been shown to be predictive for z-plasty, which is pertinent for reconstructive surgery.
However, this same correlation has not been demonstrated for open abdominal surgery (Deary et
al., 1992; Schueneman et al., 1984). Similarly, PicSOR has demonstrated some predictive
potential for laparoscopic tasks and therefore may be more useful in surgical disciplines where
laparoscopic techniques are common (e.g. urology, general surgery or obstetrics and
gynecology). It is therefore, possible that a different set of surrogate tests may be required for
selection into different surgical specialties.
While several studies have examined the association between surrogate marker scores and cross-
section performance on one or more surgical tasks, there is limited evidence concerning potential
associations between natural aptitude and longitudinal performance in terms of either rate of
skills acquisition or performance over the longer term. Buckley et al. demonstrated that after
testing 86 novices, 12 high aptitude and 12 low aptitude scoring individuals correlated to fast
versus slow learners when completing simulated general surgery tasks. This suggests that
aptitude tests may be able to select out the gifted and the slow learner, however the surgical
performance of individuals at these extremes was not studied. In addition, no study has evaluated
39
whether these initial predictions in the simulation setting transfer to the real operating
environment. Therefore, further longitudinal study is needed to better delineate the value of
aptitude testing in terms of predicting both initial performance level as well as longer-term skills
acquisition and performance in the real operating room, before this testing can be used to select
candidates entering and progressing through surgical training.
Another potential benefit of aptitude testing might be to objectively identify trainees’ technical
strengths and weaknesses, allowing for the tailoring of training activities. However, to the best of
our knowledge, no studies to date have specifically evaluated this role for aptitude testing.
Furthermore, it is unclear whether the use of surrogate markers to identify trainees’ strengths and
weaknesses confers any benefit over the use of existing in-training evaluation tools (e.g. in-
training evaluation reports, objective structured assessments of technical skills, procedure-
specific checklists etc.). Thus, further study is required to determine whether these tests can be
used in this way, including optimal testing frequency (once versus several occasions), the need to
control for time in training to account for potential change in aptitude scores over time, and
benefits of surrogate marker testing versus in-training tools.
Ultimately, technical performance is likely a result of a complex interplay of numerous innate
abilities, in conjunction with subsequent exposures and experiences. These may vary between
individuals, potentially explaining the limited success to date in identifying individual surrogate
markers that reliably predict technical performance. In an attempt to account for these
complexities, other performance disciplines utilize a battery of tests in an attempt to
simultaneously assess a range of innate abilities, with the goal of selecting candidates based on
the cross-section of innate abilities believed to maximize performance in a given field. For
example, the United States military uses a testing battery, called the Armed Services Vocational
Aptitude Battery, to characterize new recruits across a broad range of innate abilities.(Mayberry
& Carey, 1997) This test battery evaluates arithmetic reasoning, word and mathematics
knowledge and paragraph comprehension. The results are then used to direct recruits toward
specific branches and roles within the military (e.g. pilot training, intelligence) that best reflect
their set of innate abilities. Perhaps motivated by this experience, Deary et al. (Deary et al.,
1992) and Schueneman et al. (Schueneman et al., 1985; Schueneman et al., 1984) developed and
evaluated test batteries for selection into surgery. They tested combinations of characteristics
including dexterity, visual spatial ability, and personality traits, but failed to find a combination
40
tests that predicted surgical performance in the operating room as assessed by staff surgeons.
More recently, Gettman et al., (Gettman et al., 2003) Johnson et al., (Johnson et al., 2004) and
Matsumoto et al. (Matsumoto et al., 2006) have tried a more sophisticated approach to predict
performance across a range of surgical procedures by combining the assessment of multiple
abilities into a single model, known as Basic Performance Resources (BPR). This model
involves measuring a range of innate abilities such as visual-hand response speed, visual
information processing speed and grip strength. The results are then analyzed using nonlinear
causal resource analysis that attempts to identify the point at which an individual’s performance
resources become insufficient to meet those required by a given surgical task. Resources become
insufficient when one or more of the above abilities is unable to meet the challenge of the
surgical performance. In the three studies of BPR reported to date, this method has accurately
predicted technical performance for 62 to 75% of participants (Cadeddu & Kondraske, 2007;
Gettman et al., 2003; Johnson et al., 2004; Matsumoto et al., 2006). Although not perfect, this
model is the first to attempt to quality the interplay of many abilities that likely work together to
produce an individuals end performance. BPRs also acknowledge that the reasons for weak
performance are likely due to a different underlying ability, depending on the performer.
Therefore, by testing multiple abilities the BPR approach may have a higher likelihood of
understanding where restrictions most often occur. However, given the limited number of studies
and the small sample sizes used, further BPR studies are required to optimize the combination of
innate abilities best suited for predicting technical performance for different surgical techniques
to increase the accuracy of predictions. To date, this method has not been used in open surgery
and may be worth pursuing. Furthermore, further study is required to confirm its value and
feasibility as a selection tool for surgical training.
Given the large number of surrogate markers evaluated to date, it may be beneficial for future
studies to focus on the surrogate tests that have reported positive associations. Based on the
studies identified through in this review, the majority have reported positive associations
between PicSOR and laparoscopic procedures. Thus it would appear to be reasonable to include
the PicSOR in studies evaluating associations with performance on laparoscopic tasks. Similarly,
the MRT has shown positive associations with z-plasty and mandibular bone fixations, therefore,
surgical procedures involving other bone fixation or flap reconstruction maneuvers may be more
likely to correlate with this specific visual spatial test. However, in our opinion, because of a lack
41
of any other consistently reported associations, there is insufficient evidence to recommend any
other test for studies that involve any other open or minimally invasive surgical task. With
respect to combination studies, future work may benefit from incorporating these potentially
promising individual surrogate tests, as part of a combination that is appropriate for a specific
surgical discipline. Of the combination studies identified in this systematic review, Deary et al.
used the MRT but did not find a significant correlation with performance in the operating room.
However, this work was done before the development of surgical simulation and a limitation of
this study was the use of subjective assessment tools. PicSOR was developed after both these
studies had been published and therefore was not used in either one. Given the advances in
surgical education with respect to simulation and objective performance assessment tools, future
combination studies may confer more promising results when incorporating these advances.
Alternatively, BPR offers the novel advantage of attempting to quantify the interplay of many
abilities to predict each individual’s end performance, and more work with this model may prove
beneficial.
A number of studies have investigated the relationships between personal characteristics and/or
cognitive test results, and surgical performance. However, to date, no single test has been
reported to reliably predict technical performance across the range of techniques and skills
required of surgical trainees. Visual spatial tests have demonstrated some promise, but only in
predicting performance on a specific subset of surgical tasks. It appears that strategies such as
BPRs, that assess multiple innate abilities, their interaction, and their relationship with technical
skill, may be more likely to ultimately serve as reliable predictors of future surgical performance.
However, studies of this nature are limited to date, and therefore more robust research is required
before implementing this method into the surgical trainee selection processes.
2.3.3 The role of simulated technical tasks to inform trainee selection
An alternative approach to surrogate markers in predicting technical aptitude may be to
incorporate simulated technical tasks into the selection process. The United Kingdom and Ireland
both incorporate simulated technical tasks into their selection process for advanced surgical
training. The surgical training system in these countries differs from that of North America and
42
instead of medical students directly entering 5 years of surgical residency, their programs are
separated into basic and advanced surgical training. After medical school students interested in a
surgical career enter basic surgical training, which has a focus in surgery but is a rotating
internship that teaches the foundation of all clinical practices and lasts two years (Beard, 2008;
A. G. Gallagher et al., 2008). At the completion of these two years students then apply for
specialty surgery training, which involves a written application, knowledge examination and
technical skills bell ringer exam (Evgeniou, Peter, Tsironi, & Iyer, 2013). If they are not
successful in entering specialty surgical training, the student then enters a non-surgical career
with no consequences.
The technical examination in these countries is a series of simulated technical tasks assessed by
the surgical faculty for the open tasks, and the virtual reality simulator for the laparoscopic tasks.
Examples of open tasks included in the technical skills exam in Ireland are simulated end-to-end
bowel anastomosis, or resection of an ingrown toenail. In contrast, examples of minimally
invasive skills tested on the virtual reality simulator are upper GI endoscopy, or core
laparoscopic skills (A. G. Gallagher et al., 2008). The skills exam was designed to ensure that
students have the technical aptitude required before entering specialty training. Unfortunately, no
long-term follow up data has been published quantifying whether this objective skills test
predicts clinical performance during training. However, given that many simulation studies have
demonstrated transfer of skills into the real operating room (Palter & Grantcharov, 2012; Zevin
B, Dedy NJ, Bonrath EM, & TP., 2013) and that the simulation tasks used in this test are directly
linked to skills performed in the real clinical setting, it may be reasonable that scoring well on
these skills may transfer to the clinical setting.
The inherent structure of the North American selection process does not allow for a two-tiered
scheme with a simulation exam after two year. However, incorporating technical skills during
the selection process may still be possible. A single program in the United States has
incorporated a technical task into their selection process at the time of the in-person interview.
On interview day, the Otolaryngology program at the Mayo Clinic School of Medicine, has their
applicants complete a simple suture microsurgical task. The students are oriented to the
microscope, instrument handling and basic microvascular knot tying technique. The students are
given 20 minutes to close a vertical incision in a nylon glove, with simple interrupted knots using
10-nylon suture. A plastic surgeon on faculty grades performance in real time using a global
rating
perfo
is a m
succe
Price
perfo
Howe
progr
learni
progr
may n
skills
stude
aptitu
the cu
2.4
Study
withi
have
curric
the fu
traini
appro
traine
g scale (Carl
ormance scor
moderate ass
essfully ente
, Van Abel,
ormed poorly
ever, of two
ram allow stu
ing occurs in
ram is that t
not be repres
s over time in
ent level may
ude was desi
urriculum de
Simulat
ying the lear
n a standard
difficulty w
culum and in
ull breadth o
ing. The foll
opriateness f
ee (e.g. junio
lson, Archib
res on this sc
ociation betw
red the prog
& Carlson, 2
y on the scre
methods of
udents to lea
n the true tra
they assess th
sentative of
nto the Cana
y be most fea
igned for me
esign are exp
tion mod
rning curves
dized curricu
with technical
ntroducing a
of simulation
owing eleme
for the techn
or vs. senior
ald, Sorom,
creening too
ween the glo
gram and fac
2014). Unfo
ening assess
incorporatin
arn technical
aining enviro
he applicant
overall perfo
adian system
asible. There
edical studen
plained below
alities for
of medical s
ulum may pro
l skills durin
a simulated te
n evaluable a
ents should b
ical task, 2)
trainee).
& Moore, 2
l and in-hosp
obal rating sc
ulty clinical
ortunately, th
sment and th
ng technical
l skills over
onment. The
t for a single
formance. To
m, assessing t
efore, a simu
nts. The com
w.
r technica
students perf
ovide eviden
ng residency
echnical task
and which sk
be considere
the cost of t
2010). Longi
pital clinical
cale (GRS) s
performanc
here is no da
heir clinical p
aptitude into
time, which
e disadvanta
moment in
o incorporate
technical ski
ulation curric
mponents, stru
al skill tra
forming sim
nce to identi
. However, b
k to a trainee
kills are mos
ed: 1) the typ
the model, an
tudinal follo
l performanc
scores of the
ce score after
ata comparin
performance
o selection, t
h is more rep
age of the Ot
time on inte
e the acquisi
ill progressio
culum to scr
ucture and e
aining
mulated techn
fy students w
before desig
e it is import
st suitable fo
pe of simula
nd 3) the exp
ow up of
ce suggest th
e residents w
r graduation
ng students w
e during resi
the UK and
resentative o
tolaryngolog
erview day w
ition of techn
on at the me
reen for tech
evidence sup
nical tasks le
who will exc
gning a simul
tant to under
r simulation
ation model a
perience lev
43
hat there
who
n (Moore,
who
dency.
Irish
of how
gy
which
nical
dical
hnical
pporting
earned
cel or
lation
rstand
n
and its
vel of the
2.4.1
A num
tasks
the op
high
2.4.1
Surgi
fideli
synth
basic
These
been
2004)
withi
medic
5 ‘low
to hig
High
senio
Robe
fideli
(Figu
1 Open
mber of diff
(i.e. those p
perative fiel
fidelity mod
Low1.1
ical simulato
ity synthetic
hetic models
technical sk
e models are
shown to be
). Furthermo
n the operati
cal students
w-fidelity m
gh fidelity m
fidelity mod
or resident) w
erts, & Duffy
ity models in
ure 2 b,c). Th
n simulatio
ferent simula
performed th
d). These ca
dels), and liv
w and high
ors designed
models and
are generall
kills such as
e generally r
e equivalent
ore low fidel
ing room, an
and junior tr
models were c
models when
dels, howeve
who have mo
y, 2014; Sidh
nclude cadav
hese models
on models
ation models
hrough a trad
an be stratifie
ve tissue mod
h fidelity si
for open sur
high fidelity
ly made from
knot tying o
eusable and
to high fidel
lity models h
nd therefore
rainees (Chi
chosen for th
used by jun
er, have been
ore experienc
hu, Park, Bry
veric porcine
are often no
s have been r
ditional surgi
ed into man-
dels (high fid
imulation m
rgical tasks c
y cadaveric t
m plastic, pap
or simple sut
are lower in
lity models w
have been sh
are an appro
ipman & Sch
he open task
nior trainees,
n shown to b
ce working w
ydges, MacR
e and canine
ot reusable an
reported for
ical incision
-made or non
delity model
models
can be separ
tissue or live
aper or felt an
turing techni
n cost (Figur
when teachin
hown to impr
opriate and e
hmitz, 2009;
ks given their
as described
be superior f
with real tiss
Rae, & Dubr
models and
nd costly for
the training
providing d
n-living tissu
ls).
rated into low
e tissue mod
nd are typica
iques (Hamm
re 2a). Low f
ng novice tra
rove technic
effective cho
; Jensen et al
r cost advant
d in section
for higher-le
sue (Shetty,
rowski, 2007
d intricate sy
r ongoing tra
of open surg
direct visuali
ue models (l
w fidelity an
dels. Low fid
ally used to t
moud et al., 2
fidelity mod
ainees (Grob
cal performa
oice for teach
l., 2008). In
tage and equ
2.5.1’
evel trainees
Zevin, Gran
7). Common
nthetic mod
aining curric
44
gical
ization of
low or
nd high
delity
teach
2008).
els have
ber et al.,
ance
hing
Chapter
uivalence
(e.g.
ntcharov,
n high
els
cula.
a.
Figur
cadav
Mont
2.4.1
Live
the op
requi
anim
teach
wher
suffic
allow
mana
Gastr
minim
2008
re 2: Simulat
veric porcine
tbrun)
Liv1.2
models such
pportunity to
re resources
als, and appr
hers’ time are
e other simu
cient include
ws senior lev
agement and
rointestinal a
mally invasiv
; S. o. A. G.
ted open tech
e, c. high fid
ve models
h as pigs or d
o mimic surg
s such as a lic
ropriate anes
e also costly
ulation mode
e: 1) the Adv
el surgical tr
repair of sta
and Endosco
ve procedure
a. E. Surgeo
b.
hnical skills
delity synthet
dogs are the
gical compli
censed anim
sthesia (Bad
y. Therefore,
els are not ad
vanced Traum
rainees to lea
ab wounds to
opic Surgeon
es that are ve
ons, 2015).
models a. lo
tic (picture p
most realist
ications such
mal laboratory
ddoo et al., 2
live models
dequate. Two
ma and Oper
arn rare but
o the chest a
ns resident co
ery difficult
ow fidelity s
provided cou
tic simulation
h as bleeding
y, specially
008). Opera
s are typicall
o examples w
rative Mana
life-saving o
and heart, an
ourses, desig
to simulate
c.
synthetic, b.
urtesy of Dr.
n models av
g. However,
trained pers
ative equipm
ly reserved f
where simul
agement (AT
operative tec
nd 2) the Soc
gned to teach
(Ali, Ahmed
high fidelity
. Sandra de
vailable, and
they are cos
onnel to care
ment and surg
for circumsta
lation may n
TOM) course
chniques suc
ciety of Ame
h advanced
d, Jacobs, &
45
y
afford
stly and
e for the
geon
ances
not be
e, which
ch as the
erican
& Luk,
2.4.2
Simu
visua
can b
opera
2.4.2
Box t
high
introd
that p
comp
& Gr
The m
of La
using
knot
show
Goul
Partic
by th
Helm
gener
2 Minim
ulation mode
alization of th
be stratified i
ative field, an
Lap2.1
trainers (BT)
fidelity mod
duction of th
projects onto
plete the task
rantcharov, 2
most widely
aparoscopic S
g the BT to s
tie and extra
wn to transfer
d, 2009).
cipation in F
e United Sta
mer, Yates, &
ral surgery tr
mally-inva
els for minim
he operative
into those th
nd those tha
paroscopic
) are used to
dels (Figure 3
he simulation
o a monitor.
k, and operat
2010).
recognized
Surgery (FL
imulate 5 ta
acorporeal kn
r to the real o
FLS is among
ates Accredit
& Osland, 20
raining prog
asive simul
mally-invasiv
e field throug
at rely on vi
t use a comp
box traine
o simulate ba
3). The BT i
n model, and
The surgeon
tes while vie
formalized t
LS) curriculu
sks: peg tran
not tie (Fried
operating roo
g the gradua
tation Counc
012). Althoug
grams in Can
lation mod
ve surgical ta
gh a laparosc
isualization o
puter-genera
ers
asic and adva
is a plastic b
d port site op
n inserts real
ewing 2-dim
training curr
um. FLS is de
nsfer, pattern
d et al., 2004
om (McClun
ation requirem
cil for Gradu
gh this cours
nada, it rema
dels
asks (ie. thos
cope, endosc
of an existin
ated virtual s
anced laparo
box with mul
penings to in
l laparoscopi
mensional fee
riculum that
esigned to te
n cut, placing
4). The skills
ney et al., 20
ments for ge
uate Medical
se is also rec
ains optional
se relying on
cope, or othe
ng three dime
space.
oscopic tasks
ltiple openin
nsert the lapa
ic instrumen
edback from
uses the BT
each basic la
g a ligation l
s learned fro
007; Stelzer,
eneral surger
l Education
commended
at the prese
n indirect
er similar de
ensional sim
s using eithe
ngs to allow t
aroscopic ca
nts into the b
the monitor
T is the Fund
aparoscopic
loop, intraco
om FLS have
Abdel, Sloa
ry trainees m
(ACGME) (
by a numbe
ent time.
46
vice)
mulated
er low or
the
amera
ox to
r (Palter
amentals
skills
orporeal
e been
an, &
mandated
(Brown,
er of
a
c
Figur
tasks
mode
2.4.2
Virtu
consi
as an
for in
task (
re 3: (a) the l
, using eithe
el such as (c)
Vir2.2
ual reality sim
idered a high
educational
ndependent s
(e.g. lifting a
laparoscopic
er a low fidel
) cadaveric p
tual reality
mulation for
h fidelity mo
l tool (Chou
study approp
and grasping
c box trainer
lity model su
porcine smal
y simulatio
surgical tec
odel. Virtual
& Handa, 2
priate to the s
g) or procedu
b
r can be used
uch as (b) a p
ll bowel with
on
hnical skill i
reality has a
006). An adv
student’s tra
ure (e.g. lapa
d for both ba
plastic penro
h attached m
is akin to pla
a number of
vantage of th
aining level.
aroscopic ch
asic and adva
ose drain, or
mesentery.
aying a vide
f advantages
he VR simu
The student
holecystectom
anced laparo
r a high fidel
o game, and
and disadva
lator is that i
t is able to se
my) best suit
47
oscopic
lity
d is
antages
it allows
elect a
ted for
48
his/her training level from a series of modules (Figure 4). The simulator can be programmed to
provide instructions for the task, and generates immediate feedback scores and graphs for the
student to monitor his/her progress. This interactive interface decreases the number of live
instructors required to teach these skills, while providing a low stakes, stress free environment in
which to learn these skills. From a research perspective, a major advantage of this system is that
that no personnel are required to watch and rate the procedures.
However, there are two major disadvantages to the VR: 1) a lack of realism and 2) high cost.
Although the VR simulation is categorized as a high fidelity model and is designed to imitate the
real surgical environment (e.g. abdomen or pelvis), the appearance is often not realistic.
Furthermore, the surgeon’s interaction with the simulated tissue, the way it moves or bleeds, and
the lack of tactile feedback also negatively affect the experience (Satava, 2001). Newer models
have been designed to improve realism by incorporating haptic feedback. However, despite these
advances, the technology has yet to meet real operating room standards. Additionally, the cost of
a VR simulator is substantial, with the initial acquisition cost ranging from 77,500 to 150,000
Canadian dollars, with further requirements for ongoing support, upgrades and maintenance to
keep the technology updated and running smoothly (Lowry, Porco, & Naseri, 2013; Orzech,
Palter, Reznick, Aggarwal, & Grantcharov, 2012).
Figur
2.5
Many
mode
categ
metri
comb
re 4: Virtual
Instrum
y assessment
els described
gories of asse
ics and learn
bination to op
reality simu
ments for t
t tools have
d above, and
essment tool
ning curve as
ptimize the a
ulator for lap
the assess
been develo
for procedu
ls, including
ssessment, ar
assessment g
paroscopic te
sment of
oped to asses
ures performe
task specific
re outlined b
goals of diffe
echnical skil
technical
ss technical p
ed in the rea
c checklists,
below. These
ferent technic
l training an
l skill
performance
al operating r
, global ratin
e tools can b
cal skills cur
nd assessmen
e for the simu
room. The m
ng scales, tas
be used alone
rricula.
49
nt.
ulation
major
sk
e, or in
50
Table 5: Task specific checklist applicable to both the placement of one interrupted suture and
intracorporeal knot tying
Instrument)tie)check)listNot)done)or)
complete)incorrectlyCompleted)correctly
Correct'handling'of'needle'driver 0 1
Correct'handing'of'forceps 0 1
Loading'the'needle'driver'at'the'tips'of'the'needle'driver 0 1
Loading'the'needle'at'90'degrees'to'the'needle'driver 0 1
Loading'the'needle'2/3'and'1/3 0 1
Holding'the'tissue'edge'with'the'forceps 0 1
Entering'the'tissue'at'90'degrees 0 1
Coming'through'both'black'dots 0 1
Following'the'curve'of'the'needle 0 1
Exiting'the'tissue'using'the'needle'driver 0 1
Exiting'the'tissue'by'following'the'curve'of'the'needle 0 1
Pulling'the'suture'through' 0 1
Leaving'a'short'tail 0 1
Placing'the'needle'driver'between'short'and'long'suture' 0 1
1st'double'throw 0 1
Square'knots 0 1
2nd'single'throw 0 1
3rd'single'throw 0 1
Hold'both'sutures'with'needle'driver 0 1
Cut'with'suture'scissors 0 1
Max'Total'20:
51
2.5.1 Task specific checklists
These checklists are created for specific open or minimally invasive tasks and are generally
formatted as a binary scale documenting whether a step of the procedure is omitted or completed.
Checklists are best suited to evaluate novice or junior trainees who are focused on learning the
steps of the procedure. These tools are easy to implement because the examiner requires minimal
training; the binary scale is relatively intuitive, and minimal surgical judgment is needed to score
performance (Regehr, MacRae, Reznick, & Szalay, 1998). Checklists can be used to provide
objective feedback both in the simulation setting and in the real operating room. However, a
disadvantage to checklists is that they do not measure performance nuances or technical quality,
and therefore are poorly suited to more advanced trainees (Hodges, Regehr, McNaughton,
Tiberius, & Hanson, 1999). Furthermore, checklists are procedure specific, and therefore cannot
be re-used for different operations (Table 5).
2.5.2 Global rating scales
GRS are used to provide an overall assessment of performance. They require the examiner
observe a trainee perform a procedure and uses the judgment of these individuals to rate
elements of their technical skill on a Likert scale in accordance with specific descriptive anchors.
Since the tool relies on the examiner’s judgment, a subjective variable is inherently incorporated
into the assessment. Therefore, GRS are more reliable in the hands of experienced examiners or
alternatively, trained personal.
GRS have been shown to differentiate levels of experience and can therefore be used for both
novice and more advanced trainees (Regehr et al., 1998). In addition, because these scales are not
procedure specific they can be used for a multitude of tasks or operations in different surgical
fields as well as within the simulation setting or the real operating room. The disadvantage of
GRS is that they require trained personal to watch the procedures either in real time or on video
to generate a score. Both are time consuming and resource heavy, which has been reported to
severely limit uptake. Table 6 is the Objective Structure of Technical skills (OSATS), which is
considered the gold standard GRS to assess technical performance (Martin J.A et al., 1997).
52
Table 6: Objective Structure of Technical skills - Global Rating Scale (Martin et al., 1997)
1 2 3 4 5
Respect for tissue Frequently used unnecessary force of tissue or caused damage by inappropriate use of instruments
Careful handing of tissue but occasionally caused inadvertent damage
Consistently handled tissue appropriately with minimal damage
Time and motion Many unnecessary moves Efficient time/motion but occasionally causes inadvertent damage
Economy of movements and maximum efficiency
Instrument handling Repeatedly makes awkward moves with instruments
Competent use of instruments although occasionally appears stiff or awkward
Fluid moves with instruments and no awkwardness
Knowledge of Instruments
Frequently asked for the wrong instrument or used an inappropriate instrument
Knew the names of most instruments and used appropriate instrument for the task
Obviously familiar with the instruments and their names
Use of assistant Consistently placed assistant poorly or failed to use assistant
Good use of assistant most of the time Strategically used assistant to the best advantage at all times
Flow or operation and forward planning
Frequently stopped operating or needed to discuss next move
Demonstrated ability for forward planning with steady progression of operative procedure
Obviously planned course of operation with effortless flow from one more to the next
Knowledge of specific procedure
Deficient knowledge. Needed specific instruction at most operative steps
Knew all important aspects of the operation
Demonstrated familiarity with all aspects of the operation
53
2.5.3 Task metrics
Task metrics are the most common method to evaluate technical performance because the data
are objective and easy to collect (Schmitz et al., 2014). Examples of these metrics include time,
instrument motion tracking and error scores (Figure 5) (A. G. Gallagher, Richie, McClure, &
McGuigan, 2001). Task metrics are used in the simulated setting with the virtual reality (VR)
simulator and laparoscopic box trainer (BT). The VR simulator assessment metrics have been
shown to be able to discriminate performance between experts, intermediate and novice trainees.
The BT tasks measure the time of the procedure or task in real-time and quality of the end
product in terms of an error score which are quickly determined and successfully discriminate
trainee level (A. G. Gallagher & Satava, 2002; van Dongen, Tournoij, van der Zee, Schijven, &
Broeders, 2007). However, these metrics do not necessarily capture quality. For example a
trainee may complete a task quickly and accurately resulting in a high score; but may also be
rough, have poor respect for tissue and have suboptimal technique, none of which are measured
in this scoring system. However, the evidence does suggest that when separating the metrics,
time is more accurate than motion tracking when assessing performance, and that summing
these metrics into an overall score is a better assessment of technical performance (D. Stefanidis,
Scott, & Korndorffer, 2009).
Penalty Scores
Secure knot = 0
Slipping knot = 10
Knot comes apart = 20
Time to complete =_______seconds Penalty = _____ ( + ) mm. from edge of pre-drawn dots
_____ ( + ) mm. gap in incision
_____ ( + ) security of knot
Figure 5: Task metrics for the laparoscopic knot tie task, using the laparoscopic box trainer
2.5.4
Unde
impo
traine
thresh
defin
by th
wher
progr
Sever
assoc
“sigm
follow
wher
result
Howe
more
6b) (Y
variat
up an
comb
4 Using
erstanding th
rtant topic in
ees may imp
hold (Cushc
ned as an imp
ree main fea
e the surgeo
ression along
ral graphical
ciated with su
moidal” curv
wed by a per
e performan
ting in a curv
ever, as the t
substantial
Yelle, 1979)
tion between
nd down fluc
bined with th
g learning
he progressio
n surgical ed
prove with de
hieri, 2003),
provement in
atures: 1.the
ns’ learning
g the learnin
l representat
uccessful lea
ve), with an i
riod of rapid
nce plateaus.
ve that begin
task being le
familiarizati
). It is impor
n attempts, a
ctuations, ref
he random in
curves to a
on of technic
ducation. It
edicated prac
, defined as c
n performanc
initial or sta
curve stabil
ng curve cont
tions have be
arning (Yell
initial famili
d improveme
The familia
ns with the s
earned becom
ion phase, an
rtant to note
and thus the
flecting the u
nter-attempt
assess tech
cal skill acqu
is becoming
ctice, a subs
competence
ce over time
art point; 2. t
lizes (Cook,
tinuum varie
een proposed
e, 1979). Th
arization pha
ent with steep
arization pha
steep slope o
mes more co
nd the full s-
that all indi
individual le
underlying s
variance (Fi
hnical skill
uisition by an
g increasingly
set of trainee
(Szasz et al
e. Graphicall
the rate of le
Ramsay, &
es from indiv
d to represen
he most popu
ase characte
p slope, and
ase is often s
of the rapid i
omplex for th
-shape is mo
ividuals will
earning curv
s-shaped dist
igure 6c).
l progressio
nalyzing lea
y apparent th
es may not m
l., 2014). Lea
ly, learning c
earning; and
Fayers, 200
vidual to ind
nt the perform
ular resembl
erized by slo
d finally a co
hort or absen
mprovemen
he individua
ore clearly de
experience
ve is characte
tribution of p
on
arning curves
hat although
meet the safet
arning curve
curves are de
3. expert lev
04). However
dividual.
mance over
es an s-shap
w improvem
onsolidation p
nt for a basic
nt phase (Figu
al, it demand
emonstrated
performance
erized by a s
performance
54
s is an
h all
ty
es are
escribed
vel,
r,
time
pe (a
ment,
phase
c task,
ure 6a).
ds a
d (Figure
e
series of
e
Figur
and (
additi
Analy
repor
cohor
ANO
(eg. P
delin
of lea
Schlu
inclu
for se
2.6
2.6.1
Once
deliv
and th
traini
Datta
The d
into s
contin
parad
studie
re 6: Exampl
b) a difficult
ional fluctua
ysis of learn
rted using tra
rts by splittin
OVA (Cook e
Pearson or S
eating indivi
arning curve
up, Nizard, &
ded in chapt
electing a no
Technic
1 Struc
e the simulat
er the curric
hen adopted
ing model, c
a, Chang, &
distributed p
shorter time
nuous practi
digm was fir
ed this phen
les of expect
t task. Actua
ations betwe
ning curves w
aditional stat
ng the group
et al., 2004;
pearman) ha
idual perform
s (LC-CUSU
& Porcher, 2
ter 5 describ
ovel analysis
cal skill tr
cturing curr
ion models a
ulum is an im
d into surgica
ompared to
Darzi, 2002
ractice mode
segments ov
ice that occu
st reported in
omenon in r
ted smoothe
al performan
en attempts
within the su
tistics. These
ps and comp
Joseph, Phil
ave also been
mance rather
UM) and cur
2008; Marlie
es these ana
, namely k-m
raining cu
ricula for t
and assessm
mportant nex
al education
a mass train
; Moulton et
el, also know
ver multiples
urs without re
n 1913 by a
reference to m
d learning cu
nce of an ind
(c).
urgical educa
e statistical m
aring perform
llips, & Rupp
n used along
r than group
rve fitting ar
s P. Schijven
lyses is deta
means cluste
urricula
technical sk
ment tools hav
xt step. Stud
have repeate
ing design, i
t al., 2006; P
wn as spaced
s days or we
est between
German psy
memory reca
urves for the
dividual learn
ation literatu
measures are
mance with
p, 2012). Al
g with regres
ped performa
re most comm
n et al., 2004
ail in the disc
ering, for thi
kill trainin
ve been sele
dies originati
edly demons
is superior fo
Palter & Gran
d repetition,
eeks. Massed
trials. The d
ychologist H
all and demo
e performanc
ning a basic
ure has been
e best suited
t-test, chi-sq
lternatively,
ssion analysi
ance, cumula
monly utiliz
4). The full t
cussion and
s study.
ng
cted, a teach
ing in the ps
strated that a
for learning (
ntcharov, 20
breaks up th
d practice is
distributed pr
Hermann Ebb
onstrated tha
ce of (a) a ba
task may di
commonly b
d to comparin
quare test or
correlation s
is. However
ative sum an
zed (Biau, W
text manuscr
outlines the
hing model t
sychology lit
a distributed
(Mackay, M
014).
he skills train
defined as
ractice learn
binghaus, wh
at memorizin
55
asic task
splay
been
ng
r
statistics
, when
nalysis
Williams,
ript
reasons
to
terature
practice
organ,
ning
ning
ho
ng
56
random characters was more successful when separate in time (Ebbinghaus, 1913 (Reprinted
Bristol: Thoemmes Press, 1999)). Subsequently, distributive practice has proven beneficial in
procedural memory, learning fine and gross motor skills (Kwon, Kwon, & Lee, 2015; T. D. Lee
& Genovese, 1989). Technical skills require both gross and fine motor movements. In the early
2000s, therefore, researchers compared distributed practice to mass practice in the area of
surgical skills to assess whether there would be similar benefits.
The studies comparing mass practice to distributive practice were studied using simulated
technical skills. Mackay et al compared massed practice to distributed practice on minimally
invasive tasks and found a significant difference between the groups, favoring distributed
practice (Mackay et al., 2002). Moulton et al compared the two practice methods for learning a
microsurgery simulated bench task and found similar findings. Students completing distributed
practice training outperformed massed practice when assessed with GRS and checklist scores
(Moulton et al., 2006). Furthermore, the deliberate practice has also demonstrated successful
transfer of skills learned in the simulation laboratory, to the real operating room (Palter &
Grantcharov, 2014). Therefore, when I designing the technical skills training curriculum for
detecting technical aptitude in medical students, a distributed practice-training model was used.
2.6.2 Mental practice as an adjunct for technical skill training
During surgical training, technical skills curricula using the models described above have gained
tremendous momentum(Willis & Van Sickle, 2015). However, technical performance adjuncts
that require minimal equipment and include focus on cognitive components are relatively new to
the field of surgical education. Therefore mental practice (MP), as a performance adjunct for an
advanced laparoscopic procedure, was studied in chapter 6 and was the first study in the surgical
education literature to assess the effect of MP on advanced laparoscopic skills.
2.6.2
MP is
rehea
1969)
(MI),
MP is
inclu
Naga
Ment
by Sa
impro
in the
prepa
onwa
under
were
repor
physi
meta-
condu
their
these
effect
tasks
effect
novic
Def2.1
s defined as
arsal of a phy
). Interchan
, mental rehe
s performanc
ding sports,
ano, 2015; Sp
tal practice a
ackett (1934)
oved signific
e 1950s, MP
are their athl
ards, a growi
rstand the ef
the first to c
rted that men
ical practice
-analysis in
ucted a meta
predecessor
authors stud
t of the expe
when comp
ts have redu
ce athletes.
finition and
the “cogniti
ysical action
ngeable termi
earsal (MR)
ce adjunct th
music and m
pahn, 2015).
as a performa
) and Perry (
cantly for dif
began being
etes for com
ing interest i
ffectiveness
conduct a me
ntally practic
(D. L. Feltz
1988 (D.L. F
a-analysis, w
s. In addition
died the type
erience and c
pared to pred
uced to almos
d scope
ive rehearsal
n in the absen
inology used
or visualizat
hat has been
medical reha
.
ance enhanc
(1939) who
fferent psych
g used in spo
mpetitions (R
in MP spurre
of this techn
eta-analysis,
cing a motor
z & Landers,
Feltz & Land
which include
n to the posi
e of task, the
concluded th
dominately p
st half and 3
l of a task wi
nce of any gr
d to describe
tion.
demonstrate
abilitation (B
cement techn
demonstrate
hology tasks
orts when So
Ryba T, Stam
ed researcher
nique on mot
, including 6
r skill is bette
, 1983). Thes
ders, 1988).
ed 100 studi
itive effect o
e retention in
hat: 1) the eff
physical task
) MP is mor
ithout physic
ross muscul
e this cogniti
ed to improv
Bar-Eli & Blu
nique was fir
ed that after M
s (Perry, 199
oviet Union
mbulova N, &
rs to conduc
tor performa
60 studies co
er than no pr
se results we
Subsequent
ies, and came
of mental pra
nternal from
ffect of MP i
ks, 2) approx
re effective f
cal moveme
ar movemen
ive process i
ve performan
umenstein, 2
rst reported i
MP, individ
92; Sackett,
coaches use
& C., 2005).
ct numerous
ance. Feltz a
omparing 146
ractice at all
ere again sup
tly, Driskell
e to the sam
actice on mo
the MP inte
is stronger fo
ximately 2 w
for elite athle
ent” or a “sym
nt (Richardso
is: mental im
nce in many
2004; Nagan
in the early 1
dual performa
1934). Subse
ed this techni
From the 19
studies to
and Landers
6 effect size
l but not as g
pported in a
et al. (1994)
me conclusion
otor performa
ervention and
or more cogn
eeks after M
etes as comp
57
mbolic
on,
magery
y fields
no &
1930s
ance
equently
ique to
960s
(1983)
s and
good as
revised
)
n as
ance
d the
nitive
MP the
pared to
2.6.2
There
perfo
1.
The p
respo
produ
imagi
after
found
then i
electr
EMG
2.
Symb
effect
being
moto
impro
3.
Atten
aspec
perfo
perfo
distra
4.
Me2.2
e are five the
ormance. Eac
Psychoneur
psychoneuro
onses in the m
uced in the b
ined skill. In
visualizing i
d increased a
instructed su
romyography
G probe over
Symbolic le
bolic learnin
tive because
g cognitively
rs sequence,
oved overall
Attention-a
ntion and aro
cts of psycho
ormance phy
ormance and
action by irre
Bio-informa
ntal practic
eories that at
ch theory is o
romuscular
omuscular th
muscle fiber
brain during
n support of
images of th
activity durin
ubjects to im
y (EMG) pro
the bicep de
earning theo
ng theory diff
e individual a
y ready. Befo
, sets task go
l performanc
arousal set th
ousal set theo
oneuromuscu
siologically,
cognitively
elevant stimu
ational theo
ce theories
ttempt to exp
outlined belo
theory
heory sugges
rs used when
MP transmit
this theory,
he ‘Eiffel Tow
ng visualizat
magine movin
obe over the
emonstrated
ory
ffers from Ps
actions are p
ore a physica
oals, and cog
ce (Martin, M
heory
ory combine
ular theory. A
, by helping
by selective
uli (R. S. Ve
ry
s
plain the pos
ow.
ts that vivid
n actually per
t impulses to
Jacobson rec
wer’ or ‘this
tion as comp
ng their arm
e bicep. In vi
increased m
sychoneurom
planned in ad
al response i
gnitively con
Mortiz, & Ha
es symbolic l
According to
the athlete t
ely attending
ealey, 1987)
sitive effect
d, imagined e
rforming the
o the muscle
corded ocula
s morning ne
pared to rela
and simulta
isualizing th
muscular tens
muscular theo
dvance and t
is executed,
nsiders altern
all, 1999).
learning theo
o this theory
to adjust his/
g to the task
.
of mental pr
events produ
e task. Thus,
es for the exe
ar movemen
ewspaper’ an
axation (Jaco
aneously held
he arm movem
sion (Jacobs
ory and state
thus the indi
advance pla
native soluti
ory with the
y, imagery se
/her arousal
at hand and
ractice on
uce neuromu
, the images
ecution of th
nts in particip
nd consisten
obson, 1930)
d an
ment alone,
on, 1931).
es that MP is
ividual benef
anning optim
ons resulting
physiologic
erves to imp
level for opt
preventing
58
uscular
he
pants
ntly
). He
the
s
fits from
mizes
g in
cal
prove
timal
In 19
conne
becau
input
in the
physi
Figur
The r
focus
its eff
therap
comp
Figur
Ve
979 Lang pro
ections betw
use it empha
ts: 1) image c
e imagined e
iologic respo
re 7 (Lang, 1
reason for br
s on these co
ffectiveness.
py for obses
plete interact
re 7: Adapte
erbalInstr.
oposed that i
ween these un
asizes the im
cue (e.g. wri
event. He the
onse which t
1979).
reaking up th
omponents an
For exampl
ssive compul
tive process
d from Lang
B
Pro
.
Physio
magery is co
nits play a sp
mportance of
itten script),
en describes
then feed bac
he imagery p
nd to begin t
le, it is hypo
lsive behavio
(Lang, 1979
g’s model of
rain
ocess
ologicalEv
oded in a sin
pecific and p
both input a
2) image aid
two output v
ck into the m
process into
to study thei
othesized tha
ors only pro
9).
f input and ou
VerbalR
vent
ngle, uniform
predetermine
and output cu
d (e.g. film,
variables, 1)
model to mod
input and ou
ir impact on
at individuals
cess the verb
utput variab
Report
m, abstract m
e role. Lang’
ues. He desc
script), 3) a
) verbal repo
dify the ima
utput cues is
the overall p
s who do no
bal cues rath
bles for emot
manner in wh
’s model is u
cribes three m
ctive particip
ort and 2)
ge.
for research
process of M
t respond to
her than the
tional image
59
hich the
unique
main
pation
h to
MP and
MP
ry.
5.
The t
ackno
MP. T
The e
of the
overa
to op
Comb
enhan
enhan
athlet
been
2.6.2
It has
positi
comp
1992)
Three
traini
incor
First,
perfo
calmn
perfo
breath
throu
Ahsen’s trip
triple code m
owledges the
The triple co
effects are id
e image to th
all performan
timize the m
binations of
ncement stra
ncing perfor
tes use MP a
incorporated
How2.3
s become cle
ive effect on
petitive sport
) swimming
e component
ing: 1. begin
rporating em
before enga
ormance (Wa
ness into me
ormance. Tw
hing control
ughout the bo
ple code mod
model has som
e interaction
ode model si
dentified as I
he individual
nce and allow
mental image
these theori
ategy being t
mance in sp
as an adjunct
d into surgic
w athletes
ear from hist
n motor perfo
ts have demo
, (Bar-Eli &
ts are recom
nning with a
motional imag
aging in MP
adey & Hant
entally practi
wo common r
l. Progressiv
ody and syst
del of image
me similarit
n of many pro
implifies the
ISM: I – the
l. The incorp
ws for perfo
ery for the in
es are gener
taught. In su
orts is most
t to improve
cal training.
incorporat
toric and ong
ormance, inc
onstrated the
Blumenstei
mmended for
relaxation ex
gery.
, beginning w
ton, 2008). I
icing a sport
relaxation te
e muscle rel
tematically r
ery (ISM)
ies to Lang’
ocesses that
e concept to f
image itself
poration of a
ormance coac
ndividual (Ah
rally accepte
urgical educa
commonly r
e performanc
te mental p
going researc
cluding com
e positive eff
in, 2004) and
athletes to s
xercise, 2. v
with a relaxa
t is hypothes
t translates th
chniques are
laxation invo
relaxing each
s bioinforma
support and
focus on onl
f, S-the soma
all three elem
ches to focu
hsen, 1984).
d depending
ation, the stu
referenced a
ce also clarif
practice int
ch that MP c
mpetitive spor
ffects of MP
d golf (Brou
successfully
vivid imagery
ation exercis
sized that as
hese associa
e progressive
olves maxim
h muscle gro
ational theor
d contribute t
ly three effec
atic response
ments into im
s on each co
.
g on the perf
udy of MP in
and therefore
fies how elem
to training
continues to
rts. Studies f
including di
uziyne & Mo
incorporate
y using all fi
se has been s
ssociating a s
ated feelings
e muscle rel
mally tensing
oup until the
ry in that it
to the succes
cts of the im
e, M – the m
magery enha
omponent sep
formance
n relation to
e understand
ments of MP
demonstrate
from many
iving, (Grou
olinaro, 2005
MP into spo
ive senses, 3
shown to im
state of contr
into true
laxations and
g the muscles
body is com
60
ss of
magery.
meaning
nces
parately
ding how
P have
e a
uios,
5).
ort
3.
mprove
rol and
d deep
s
mpletely
61
tranquil. Deep breathing control, also known as diaphragmatic breathing, involves breathing in
deeply through the nose to expand the abdomen and then releasing each breath slowly through
the mouth (Williams & Harris, 2006). Engaging in either technique prior to MP is
recommended.
Second, the key to successfully using MP is being able to use visualization to produce an image
that makes the individual feel like he/she is actually performing the sport (Hale, 1998; Holmes
& Collins, 2001). Obtaining the highest level of vivid imagery requires performers to
incorporate all five senses: sound, sight, touch, smell, and taste. Auditory imagery may be
hearing the edge of an athlete’s ski as it cuts against the underlying snow. Visual imagery may
be seeing a basketball move through the net of the hoop. Olfactory imagery may be the smell of
chlorine before a swimmer enters the pool. Tactile imagery may be adjusting the grip before
swinging a golf club. Kinesthetic imagery may involve feeling the optimal positions of
movement before successfully executing a task (Holmes & Collins, 2001).
Third, in addition to the five senses, incorporating emotion into imagery has also been shown to
strengthen the experience. Athletes may link emotions such as fear, anger, stress or anxiety to a
particular performance. Recognizing and transforming negative emotions into positive feedback
has proven beneficial (R. Vealey & Greenleaf, 2006). For example a soccer player may prepare
for a penalty kick by thinking through the anxiety and pressure that is inevitably present during
the execution of this task. They then are encouraged to visualize a successful goal and link it to
the positive emotions associated with victory, such as elation, or pride (R. Vealey & Greenleaf,
2006).
However, all three of these components can be incorporated into MP that focuses on different
aspects of enhanced performance. Recognizing the diversity of MP within sports, Hall et al.
developed a classification system, separating imagery into five domains (Hall, Mack, Paivio, &
Hausenblas, 1998; Short, Tenute, & Feltz, 2005).
1. Motivation-Specific (MS) – imaging focused on goal specific behavior such as winning a
race or receiving a medal.
2. Motivational General-Mastery (MG-M) – imagery focused on coping during difficult
performance moments, being mentally tough or working on confidence
3
4
5
Depe
above
MG-M
task t
2.6.2
Simil
the po
appli
acqui
impac
rando
practi
The a
score
Van W
were
sutur
(Sand
MI de
altern
. Motivatio
with sport
. Cognitive
double ax
. Cognitive
cup socce
ending on the
e may be inc
M and MG-A
the focus of
Me2.4
lar to sports,
otential for p
cation of MP
isition, focus
ct of MP in b
omized to thr
ice and men
authors conc
es as compar
Walsum, 200
randomly as
e closure, or
ders et al., 20
emonstrated
native to con
on General-A
ts competitio
e Specific (C
xel or slap sh
e General (C
er game.
e focus of th
corporated. C
A are the foc
the imagery
ntal practic
surgery is a
profound lon
P in surgery
sing on dom
basic open s
ree groups r
ntal rehearsal
cluded that c
red to continu
04). The sam
ssigned to ei
r MI of the s
008). Theref
d a positive im
ntinued phys
Arousal (MG
ons such as r
CS) – using im
hot.
CG) – image
he performan
Classically f
cus, whereas
may be con
ce in surge
a high-stakes
ng-term cons
is relatively
mains CS and
surgical skill
eceiving eith
l for placing
ombining M
ued physical
me research g
ither a textbo
ame procedu
fore, it becam
mpact on pe
ical practice
G-A) – image
relaxation, a
magery to pr
ry of the stra
nce training,
for top tear a
s when novic
ncentrated on
ery
s environmen
sequences fo
y new and ha
d CG. In 200
ls. The autho
her physical
simple inter
MP with phys
l practice alo
group compl
ook descript
ure. The MI
me apparent
rformance a
e on more co
ery that focu
anxiety, arou
ractice and p
ategies for c
a different c
athletes prepa
ce players ar
n CS and CG
nt with narro
or patients be
as been focus
4 Sanders et
ors had sixty
practice alo
rrupted sutur
sical practice
one (Sanders
leted an add
tion of an inc
group outpe
that of the t
and could the
ostly simulati
uses on emot
usal.
perfect sport
competitive e
combination
aring for a c
re learning a
G.
ow performa
eing operate
sed primary
t al. were the
y-five medica
one or a com
res into a po
e resulted in
s, Sadoski, B
ditional study
cision with s
erformed the
two cost effe
erefore be co
ion models.
tion in conju
ting skills su
events such a
n of the five d
competition M
a specific ski
ance margin
ed on. Howev
on technica
e first to stud
al students
mbination of p
orcine foot m
equal perfor
Bramson, W
y where stud
simple interr
e textbook gr
ective modal
onsidered as
62
unction
uch as a
a world
domains
MS,
ill or
s and
ver, the
al skill
dy the
physical
model.
rmance
Wiprud, &
dents
rupted
roup
lities,
an
63
The impact of MI on minimally invasive techniques was studied first for endoscopy and
for laparoscopy for junior level procedures. In 2009 Komesu et al. utilized the same
methodology as Sanders et al. but compared gynecology residents’ performance on cystoscopy
after being trained by either MP or a textbook description of the procedure. They concluded that
the MP groups outperformed the textbook group by 15.1% (p=0.03). Initial study in the areas of
laparoscopy demonstrated conflicting reports. Immenroth et al. demonstrated that MP did not
improve laparoscopic cholecystectomy performance in already practicing surgeons (Immenroth
et al., 2007) and Jungman et al. reported that MI did not show an advantage in performance of
laparoscopic knot tying (Jungmann et al., 2011). However, Arora et al. criticized these studies
and identified that an insufficient description of how MP was being delivered to the participants
was problematic(S. Arora et al., 2010). Insufficient descriptive methodology made it difficult to
understand the reasons for the discrepancies in the results and, furthermore, impossible to
replicate these studies. To rectify this deficiency, Arora et al. developed and validated an MP
protocol for surgery (
Figure 8).
Figur
The M
traine
2010)
contr
levels
the R
reside
re 8: Mental
MP protocol
ees performi
). The autho
rol group and
s being equa
RCT describe
ents, MP wa
Men
V
pro
Fa
MPof
Ment
Practice pro
was tested d
ing laparosco
ors reported t
d scored stat
al (S. Arora,
ed in chapter
as delivered t
Introductio
Relax
ntalImagery
Videowithex
cedurewith
amiliarizatio
procedurew
talImagery
otocol for sur
during a rand
opic cholecy
that the stude
tically superi
Aggarwal, S
r 6 using MP
to the partici
ontomental
xationexerc
yQuestionn
xpertsperfo
htalk‐overo
on/learning
withextern
Questionna
rgery, adopt
domized con
ystectomy on
ents random
ior despite b
Sirimanna, e
P as an adjun
ipants using
lpractice
cise
aire(pretes
ormingthe
ofMPscript
gMPscripts
naltalk‐outl
aire(post‐te
ted from (S.
ntrolled trial
n a virtual re
mized to the M
baseline char
et al., 2011).
nct for advan
g a similar pr
st)
ts
s
loud
est)
Arora et al.,
l (RCT) in n
eality simula
MP group ou
racteristics a
Therefore w
nced laparos
rotocol.
, 2010).
novice surgic
ator (S. Arora
utperformed
and technical
when conduc
scopy in seni
64
cal
a et al.,
the
l skill
cting
ior
2.6.2
Figur
acqui
steps
the op
from
Step
It is g
field,
be int
ration
collec
metho
defin
data d
1967)
the co
study
Step 2
When
kines
As de
sense
focus
Me2.5
Using the
re 8 requires
isition of tec
: 1.interview
peration, usi
the expert s
1: Interview
generally acc
who perform
terviewed ha
nale was rep
cting data th
odologically
ned as the mo
does not she
). The sampl
omplexity an
y outline in c
2: Record th
n creating an
sthetic cues,
escribed in th
es as possible
sed on visual
ntal practic
e protocol ou
the develop
chnical skills
w expert surg
ing both inte
cript.
w expert surg
cepted that th
m the desire
as not been d
orted to exp
hrough interv
y reasonable
oment to stop
ed any furthe
le size of exp
nd predictab
chapter 6, sat
he surgeon’s
n MP script,
external and
he sporting l
e in order to
l and kinesth
ce script de
utlined in
pment of MP
s has focused
geons, 2.reco
ernal and ext
geons
he scripts sh
ed operation
defined. Aro
lain their sam
views is com
to stop colle
p interviewin
er light on th
perts to reac
bility of the o
turation was
s visual and t
it is recomm
d internal im
literature on
make the im
hetic cues on
evelopmen
P scripts, whi
d primarily o
ord the surge
ternal imager
hould come f
routinely. H
ora et al. inte
mple size (S
mmonly done
ecting data w
ng new stud
he issue unde
ch adequate s
operative app
s met after in
tactile cues,
mended to in
magery, and e
MP, the scr
magery realis
nly because t
nt
ich in the se
on CS and C
eon’s visual
ry, and 3. ex
from intervie
However, the
erviewed thre
S. Arora et al
e in qualitativ
when saturat
dy participan
er investigati
saturation w
proach to the
nterviewing e
using both i
ncorporate fo
each are expl
ripts should i
stic. In surge
the laparosco
etting of surg
CG imagery,
and tactile c
xtraction of c
ews of exper
e optimal num
ee expert sur
l., 2010). Re
ve research,
tion has been
nts ‘when the
ion’ (Glaser
will likely dif
e procedure
eight expert
internal and
our compone
lained in det
incorporate
ery, howeve
opic surgica
gical educati
following th
cues at each
common the
rt surgeons i
mber of expe
rgeons but n
ecognizing th
it may be
n met. Satura
e collection o
.B. & Straus
ffer dependin
selected. In
surgeons.
d external im
ents: visual c
tail below.
as many of t
r, the image
al environme
65
on and
hree
step of
emes
in the
erts to
no
hat
ation is
of new
ss.A.,
ng on
the
magery
cues,
the five
ry
ent, from
a tech
exclu
Ment
(Glisk
the th
pitfal
the th
grab
intern
exper
that t
Step 3
The s
each
comm
third
2.6.2
Apart
opera
surge
2011)
The o
surgi
envir
maint
hnical perspe
uded(Holmes
tal practice o
ky, William
hird person p
lls performed
hird person,
tiny bites of
nal imagery
rt being inter
he stapler is
3: Extraction
script is then
surgeon’s na
mon visual a
(external im
Me2.6
tech
t from techn
ative room c
eons and thu
).
operating roo
cal care to p
ronment, ther
tained despi
ective, is les
s & Collins,
outcomes are
s, & Kihlstro
perspective.
d by surgica
“they will gr
f the bowel w
is when you
rviewed may
perpendicul
n of common
n compiled b
arration. The
and tactile cu
magery) perso
ntal practic
hnical skill
nical skills im
an result in a
s improve no
om is compr
patients. The
refore the co
te the inhere
ss reliant on
2001).
e improved w
om, 1996). E
For example
l residents w
rab the bowe
which increa
u see yoursel
y verbalize,
lar to the bow
n themes fro
by extracting
e themes are
ues extracted
on.
ce as a stra
ls
mprovement,
a significant
on-technical
rised of a tea
operating ro
ongruence of
ent stressors.
smell, taste
when incorp
External ima
e, the expert
when running
el and mesen
ases the risk o
lf execute the
“ I always ch
wel. Once I
m the expert
g common th
e generally o
d for each ste
ategy to de
, it has been
t reduction in
l performanc
am of profes
oom is consi
f the team is
. The combin
and sound, t
orating both
agery is the a
being interv
g the bowel,
ntery in the s
of injury if t
e task in the
heck multipl
am sure, I fi
t script
hemes from t
organized int
ep, in both th
ecrease stre
shown that
n the stress l
ce (S. Arora,
sionals that
idered a high
s essential to
ned element
thus these se
h external an
ability to vis
viewed may
which woul
same bite, so
they slip off”
first person
le times and
fire the staple
the transcrib
to the steps o
he first (inter
ess and imp
using MP in
levels experi
, Aggarwal,
work togeth
h stakes, hig
o ensure perf
ts that compr
enses were
nd internal im
sualize a task
reflect on co
ld be verbali
ometimes th
”. In contras
n. For examp
d from many
er slowly”.
bed documen
of the operat
rnal imagery
prove non-
n the simulat
ienced by no
Moran, et al
her to deliver
h stress
formance is
rise these hu
66
magery
k from
ommon
ized in
ey will
t,
ple the
angles
nt of
tion and
y) and
-
ted
ovice
l.,
r safe
uman
67
factors within the operating room are termed non-technical skills (Yule & Paterson-Brown,
2012). Non-technical skills include: teamwork, communication, situational awareness and stress
management and have been recognized as an important part of a functioning within the real
operating room (Helmreich, 2000). Deficiencies in non-technical skill, especially
communication within the operating room, have been identified as the reason for approximately
60% of perioperative complications (Greenberg et al., 2007). The physiologic release of cortisol
induced by stress has been shown to modify cognitive processes in memory, performance and
decision making (de Quervain, Roozendaal, Nitsch, McGaugh, & Hock, 2000; Johnston,
Driskell, & Salas, 1997; Kirschbaum, Wolf, May, Wippich, & Hellhammer, 1996; Wolf, 2003)
Therefore, utilizing MP as an adjunct to control stress may be a potential strategy to improve
non-technical performance.
Four main valid and reliable assessment tools have been developed to evaluate non-technical
skills of different team members within the operating room: surgeons, nurses or anesthetists
(Table 7). These assessments are GRS that rely on direct observation of the operating room
personnel and therefore have the same advantages and disadvantages to the technical skill GRS
described in section 2.5.2.
Of the available tools, NOTSS and OSANTS both assess the surgeon role within the operative
team. Given that the study described in chapter 6 assessed the effect of MP on the surgeon in the
operating room, NOTSS was used because it was the only surgeon role assessment tool
available at the time the study was completed.
68
Table 7: Assessment tools of non-technical skills in the operating room
Name of assessment tool
Operating room team member being assessed
Non-technical skills included
Scale Reference
Non-technical skills for surgeons (NOTSS)
Surgeon Situational awareness Decision making Communication and teamwork Leadership
4 point Likert scale
(Yule et al., 2008)
Objective structured assessment of nontechnical skills (OSANTS)
Surgeon Situational awaress Decision making Teamwork Communication Leading and directing Professionalism Managing and coordinating
5 point Likert scale
(Dedyetal.,2015)
Anaesthetists' non-technical skills (ANTS)
Anaesthetist Situational awareness, Decision making Task management Team working
4 point Likert scale
(Lyk-Jensen, Jepsen, Spanager, Dieckmann, & Ostergaard, 2014)
System for scrub practitioners' non-technical skills (SPLINTS system)
Nurses and technicians
Situational awareness Communication and teamwork Task managmenet
4 point Likert scale
(Mitchell et al., 2013)
Cha3
visu
perf
train
The s
demo
lapar
Louri
spatia
postg
3.1
Back
traine
select
guide
purpo
spatia
Meth
non-s
evalu
was a
Resid
and v
apter 3: P
ual spati
formanc
ning
systematic re
onstrated a p
oscopic box
idas M, Quin
al ability test
graduate surg
Abstrac
kground: Em
ees have the
ting resident
e the incorpo
ose of the pr
al test results
hods: First y
surgical expe
uated using th
assessed usin
dent perform
visual spatial
Predictiv
al ability
ce among
eview (sectio
ositive abilit
trainer. The
nn E, Grantc
ting on lapar
gical training
ct
merging evid
ability reach
ts that have t
oration of tec
esent study w
s are predict
year residents
eriences wer
he PicSOR,
ng the camer
mance on thes
l findings.
ve value
y testing
g residen
on 2.3.2) con
ty to predict
erefore the fo
charov TP. P
roscopic bas
g. Surg Endo
dence sugge
h technical c
the ability to
chnical abilit
was to evalu
tive of baseli
s were studie
re obtained u
cube compa
ra navigation
se technical
of back
g on lapa
nts enter
ncluded that
t laparoscopi
ollowing stu
Predictive va
seline perform
o.2015 June
ests that desp
competency
o reach techn
ty into selec
uate whether
ine laparosco
ed. Demogra
using a quest
arison (CC) a
n (LCN) task
tasks was co
kground
aroscopic
ring post
t the visual sp
ic skill on th
udy was cond
alue of backg
rmance amon
20.
pite dedicate
in minimally
nical compet
ction process
r background
opic skill for
aphic data an
tionnaire. Vi
and card rota
k and laparo
ompared and
experien
c baselin
tgraduate
patial test Pi
he VR simula
ducted and p
ground expe
ng residents
ed practice, n
y invasive te
tence is imp
ses is limited
d experience
r the novice
nd backgrou
isual spatial
ation (CR) te
oscopic circle
d correlated
nces and
ne
e surgica
icSOR
ators and
published as:
eriences and
entering
not all surgic
echniques. W
ortant, evide
d. Therefore,
es and 2D-3D
surgical trai
und surgical
ability was
ests. Technic
e cut (LCC)
to the questi
69
d
al
:
visual
cal
While
ence to
, the
D visual
inee.
and
cal skill
task.
ionnaire
Resu
signif
was a
cube
p=0.0
signif
perfo
Conc
techn
metri
techn
skills
innat
3.2
Train
envir
their
into t
strate
aviati
Wick
desig
reaso
progr
succe
(Hold
aircra
ults: Previou
ficantly bette
associated w
comparison
03) and Ang
ficant correl
ormance.
clusion: Wh
nical skill per
ics of surgica
nical perform
s will require
e abilities as
Introdu
ning program
ronments hav
desired prof
training, and
egies has bee
ion, sports a
kham & Dilw
gned to objec
oning, and ap
ram (J. A. Be
eed on enteri
dsworth, 198
aft training m
s experience
er LCN perf
with significa
test demons
le Path (rs(AP
ations were
ile identifyin
rformance is
al performan
mance of diff
e the develop
s well as thei
uction
ms that prepa
ve two main
ficiency stan
d 2) optimizin
en recognize
nd the armed
worth, 1987)
ctively asses
ptitude for un
ell, 1988). T
ing this diffi
88). Subsequ
methods are
e in observin
formance, an
antly better L
strated a mor
P) =-0.426, p
found betwe
ng selection
s appealing,
nce related to
ferent laparo
pment of a se
ir inherent in
are candidate
opportunitie
ndards: 1) sel
ng the in-tra
ed in a numb
d forces (Th
. For examp
s intelligenc
nderstanding
These tests ar
cult, high str
uently, a com
used to optim
ng laparoscop
nd experienc
LCC task res
re accurate L
p=0.01) score
een the visua
tests for inc
the surrogat
o a single tas
oscopic tasks
eries of evid
nteractions.
es for perform
es to maxim
lecting appro
aining learnin
er of high-st
he Air Force,
ple, the aviati
ce, coordinat
g angles and
re designed t
ress, resourc
mbination of
mize learnin
pic procedur
ce in navigati
sults. Reside
LCN Path Le
e when comp
al spatial tes
coming surgi
te markers ev
sk but do no
s. Predicting
dence-based
mance in hig
mize the likeli
opriate cand
ng environm
takes perform
, 2013; Unite
ion industry
tion, instrum
d bearings, be
to select the
ce heavy and
f didactic lea
ng for succes
res was asso
ing the lapar
nts who scor
ength score
pleting the L
sts (PicSOr, C
ical trainees
valuated cor
ot appear to r
g the acquisit
tests that me
gh-stakes, hi
ihood of gra
didates at the
ment. The im
mance profe
ed States Co
y relies on a b
ment interpret
efore grantin
students wh
d costly train
arning, simul
ssful applica
ociated with
roscopic cam
red higher o
(rs(PL) =-0.36
LCN task. N
CC or CR) a
that predict
rrelate with s
reliably pred
tion of techn
easure a num
igh pressure
aduates meet
e time of adm
mportance of
essions inclu
oast Guard, 2
battery of tes
tation, gener
ng entry in th
ho are most l
ning program
lation and re
ants.
70
mera
on the
6,
No other
and LCC
specific
dict
nical
mber of
ting
mission
both
uding
2013;
sts
ral
he
likely to
m
eal
71
Surgical practice is similarly a high-stakes, high-performance profession. While technical skill
represents only a portion of the expertise expected of practicing surgeons, it is an unavoidable
component of the field, and emerging evidence suggests that it may also be linked to patient
outcomes (Birkmeyer et al., 2013). Furthermore, it is increasingly important given the changing
training and practice landscape, including increasing reliance on more technically challenging
minimally invasive techniques, and reduced work hour restrictions in the training milieu
(Accreditation Counci for Graduate Medical Education, 2013; Crothers, Gallagher, McClure,
James, & McGuigan, 1999; Deziel et al., 1993). To account for these changes a number of
innovations have been introduced to enhance the surgical training environment, including the
use of validated simulation models and competency based training curricula ("CanMEDS 2015:
The next evolution of the CanMEDS Framework," 2013; Palter, Orzech, Reznick, &
Grantcharov, 2013;
Royal College of Surgeons of England Royal College of Surgeons of England, 2013; Zevin B et
al., 2013). Focusing on optimizing the selection process for surgical trainees will further
complement these changes. While a range of subjective and objective selection criteria are used
for admission into surgical training, these are generally limited to assessments of theoretical
knowledge, clinical skills, and professionalism. However, it has been well documented that
surgical trainees acquire technical skills at variable rates, and that some trainees may not reach
competence despite dedicating time, effort and repetitive practice. Grantcharov, Schijven, and
Alvand et al. have all reported this phenomenon, and have suggested that between 8 and 16
percent of contemporary surgical trainees fall within this group (A. Alvand, S. Auplish, H. Gill,
& J. Rees, 2011; Grantcharov & Funch-Jensen, 2009; M. P. Schijven & Jakimowicz, 2004).
Thus, while current criteria may effectively select candidates with desirable theoretical
knowledge and professional attitudes, finding reliable measures to select residents able to reach
competence in technical skills will undoubtedly further improve the selection process.
Several authors have suggested that previous experience with tasks requiring hand-eye
coordination, and/or superior innate visual spatial 2D-3D conversion ability, may predict the
acquisition of laparoscopic and endoscopic skills (A. G. Gallagher et al., 2003; D. Stefanidis,
Korndorffer, et al., 2006). However, the evidence correlating these surrogate markers to surgical
technical skills are limited and inconsistent (Maan, Maan, Darzi, & Aggarwal, 2012). Therefore,
the purpose of the present study was to evaluate whether previous surgical experiences, non-
surgi
the no
3.3
3.3.1
All in
consi
start o
traine
comm
speci
the st
traine
techn
(e.g.
reside
schoo
Resea
provi
3.3.2
The r
and n
to lap
2012
cal experien
ovice surgic
Materia
1 Partic
ndividuals en
idered eligib
of the Depar
ees consente
monly report
alty, and we
tudy. The su
ees at the aut
niques, prepp
laparoscopy
ency, and att
ol. On comp
arch Ethics B
ided informe
2 Demo
residents com
non-surgical
paroscopic p
; Ju, Chang,
nces and 2D-
al trainee.
als and M
cipants and
ntering their
ble to particip
rtment’s man
ed to particip
ted that they
ere therefore
urgical boot c
thors’ institu
ping and drap
y and microsu
tempts to ho
letion of the
Board appro
ed consent to
ographics
mpleted a de
experiences
rocedures, v
Buckley, &
-3D visual sp
Methods
d setting
first year of
pate in the pr
ndatory surg
pate in the stu
felt that lap
not intereste
camp is a ma
ution that pro
ping) and in
urgery). The
omogenize th
e boot camp
oval was obta
o participate
and partici
emographics
s thought to c
video games
& Wang, 2012
patial tests co
f surgical tra
resent study
gical boot cam
udy. Trainee
aroscopic sk
ed in the add
andatory intr
ovides teach
ntroductions
e boot camp
he varied exp
a bell ringer
ained prior to
in the study
ipant quest
sheet along
contribute to
and musical
2; Paschold
orrelate with
aining at the
y, and were in
mp. Thirty-s
es who decli
kills were no
ditional lapa
roductory co
hing in basic
to specialty-
takes place
posures of th
r examinatio
o data collec
y.
tionnaire
with a ques
o baseline su
l instrument
et al., 2011)
h baseline la
University o
ntroduced to
seven of a to
ined to partic
ot pertinent t
aroscopic exp
ourse for all
surgical ski
-specific sur
during the fi
he cohort du
n is used to
ction, and al
stionnaire qu
urgical skill i
s (Adams, M
).
aparoscopic s
of Toronto w
o the study a
otal of 57 eli
cipate most
to their surgi
posure provi
first-year su
ills (e.g. steri
rgical techniq
first two wee
uring medica
assess the tr
l participant
uantifying su
including ex
Margaron, &
72
skills in
were
at the
gible
ical
ided by
urgical
ile
ques
eks of
al
rainees.
ts
urgical
xposure
Kaplan,
73
3.3.3 Visuospatial testing
Participants then completed three previously validated visual spatial tests designed to evaluate
innate 2D-3D visual spatial ability. The intent was to combine visual spatial tests that have
demonstrated positive relationships with minimally invasive technical skills in previous studies,
and were therefore believed to measure the innate abilities necessary for incoming trainees to
excel in laparoscopy (Buckley et al., 2013; Buckley et al., 2014; A. G. Gallagher et al., 2003; D.
Stefanidis, Korndorffer Jr, et al., 2006). The following tests were used: the Pictorial Surface
Orientation (PicSOr) test (A. G. Gallagher et al., 2003), the cube comparison test
(CC),(Ekstrom, French, Harman, & Dermen, 1976) and the card rotation test (CR) (Ekstrom et
al., 1976). The PicSOr test evaluates 2D-3D conversion ability by requiring participants to orient
a rotating arrow at a 90-degree angle to one side of an underlying cube. Initially, students used
the PicSOr test in practice mode, receiving immediate feedback from the software concerning
the actual arrow to cube angle. No limitations were set for the duration of practice, and a single
instructor was available to answer questions during this time. Once participants felt ready to
proceed, the software was switched to experiment mode (Figure 9). During the experiment, 35
angle estimations were completed and students were instructed to complete the task as quickly
and accurately as possible. No time restriction was enforced. The scores were graded as
described by the creators of PicSOr (A. G. Gallagher et al., 2003). The students then completed
the CC tests and CR test paper tests, administered as directed by the test copyright holder’s
(Educational Testing Service, Princeton New Jersey) official test administration manual Figure
10 (Ekstrom et al., 1976). The participants started by performing three practice exercises before
starting the test, with a single instructor present to correct and explain the questions. Next, the
two-page test was administered and timed, with 3 minutes allotted per page, totaling 42 cube
comparison exercises and 20 rows of card rotation tests. The students were instructed to work
through the questions as quickly and accurately as possible. The tests were graded with the
answer key provided in the test administration manual, with negative marking for incorrect
answers.
Figur
Setup
degre
Figur
test (C
re 9: Pictoria
p to change b
ees to the un
re 10: Paper
CR) and (b)
al Surface or
between prac
derlying cub
tests used to
cube compa
rientation tes
ctice and exp
be.
o assess 2D-3
arison test (C
st (PicSOr) u
periment mo
3D visual sp
CC)
used to asses
ode. b. rotati
patial ability
ss 2D-3D pe
ing arrow or
included the
erception abi
iented to lie
e (a) card ro
74
lity. a.
90
otation
75
3.3.4 Laparoscopic skills familiarity session
Residents were exposed to basic laparoscopic techniques during a mandatory 2 hour boot camp
session. The purpose of this session was to ensure that all residents were familiar with the
laparoscopic instruments and basic simulated tasks. First, students watched a video illustrating
each of the three Fundamentals of Laparoscopic Surgery (FLS) exercises. Subsequently,
students attempted the three tasks: the peg transfer task, the laparoscopic circle cut (LCC) task,
and intracorporeal knot tying.(Peters et al., 2004) Additionally, residents had the opportunity to
attempt three basic LapSim virtual reality simulator (Surgical Science, Gothenburg, Sweden)
tasks including: laparoscopic camera navigation (LCN), the coordination task, and the lifting
and grasping task. Three surgical faculty and two senior surgical residents were present at the
laparoscopic session to answer questions and demonstrate the tasks for the first year trainees. No
formal training was offered during this session.
3.3.5 Assessment of laparoscopic baseline skill
Students were given a maximum of 300 seconds to complete the laparoscopic circle cut task in
the box trainer. Students were asked to cut out the circle as quickly and accurately as possible
(E. M. Ritter & Scott, 2007). The circle cut task was scored by a single grader using the
objective FLS scoring system, with lower scores representing better performance: final score
(max 300)= total time (seconds) + error (surface area of white gauze from the black line). Each
resident was then given three attempts at the LCN task on the virtual reality (VR) simulator and
the best score was retained for assessment. The LCN task was scored using three VR metrics:
total time, instrument path length and instrument angular path. LCN total score was generating
by adding these three components in equal weights.
3.3.6 Statistical Methods
One-way analysis of variance (ANOVA) was used to assess the relationships between previous
surgical and non-surgical experiences and the laparoscopic tasks. Hochberg post hoc analysis
was u
signif
corre
abilit
visua
data w
3.4
3.4.1
The g
with
repre
3 from
Thirty
comp
reside
into C
stand
used in the c
ficance occu
lation analys
ty tests and t
al spatial test
was analyze
Results
1 Demo
group includ
a median ag
sented in the
m plastic sur
y-three parti
pleted 1-2 jun
ency abroad
Canadian sur
dardized cour
ases where t
urred. Leven
sis was used
he two lapar
t scores. Val
s using SPSS
s
ographics
ded 37 first y
ge of 27 year
e sample (14
rgery, 3 from
icipants had
nior resident
and fellows
rgical reside
rse (Table 8)
the ANOVA
’s test was m
d to examine
roscopic skil
ues are prese
S 22.0 softw
year surgical
rs of age (ran
4 from gener
m vascular su
no previous
t years of su
ship training
ency training
).
A was signifi
met for all AN
the relation
ll test scores
ented as med
ware (SPSS I
residents, 2
nge 23-37 ye
ral surgery, 8
urgery, and o
surgical tra
urgery trainin
in vascular
g. None of th
icant in orde
ANOVA calc
nship betwee
s, because of
dian (range)
Inc., Chicago
27 men and 1
ears). Seven
8 from ortho
one each fro
aining outsid
ng, and 1 had
surgery in C
he participan
r to identify
culations. Sp
en the three i
f non-normal
) unless state
o, Illinois).
11 females, a
surgical spe
opaedic surge
om 5 differen
de of medical
d completed
Canada, befo
nts had comp
the level at
earman’s rh
innate visual
l distribution
ed otherwise
all right hand
ecialties were
ery, 4 from u
nt specialties
l school, 3 h
d a full surgic
ore being acc
pleted the FL
76
which
o
l spatial
n of
. All
ded,
e
urology,
s).
had
cal
cepted
LS
77
Table 8: Demographic and background characteristics of study participants
Background characteristic No. of
participants
Gender
Males 26
Females 11
Handedness
Right 37
Left 0
Surgical program
Plastic surgery 3
General surgery 14
Ear nose and throat surgery 1
Orthopaedic surgery 8
Urology 4
Neurosurgery 2
Cardiac surgery 2
Vascular surgery 3
Previous surgical training
None 33
1-2 junior years 3
Completed vascular surgery 1
Fundamental of laparoscopic surgery
Yes 0
No 37
78
Table 9: Previous surgical and non-surgical experiences
Type of previous experience No. of
participants
LCN
p - values
LCC
p - values
Observed laparoscopic procedures
None 0
0.04* 0.55 <10 8
10-20* 12
>20* 17
Drove the laparoscopic camera
None 2
0.55 0.01* <10 25
10-20 6
>20* 4
Musical instrument
None 13
0.86 0.70 Piano 11
String instrument 4
Wind instrument 5
Musical royal conservatory grade
None 19
0.37 0.14 < grade 5 9
> grade 5 9
Exposure to video games
None 17
0.43 0.38 >1-2 times/week 7
1-2 times/month 10
1-2 times/year 3
*Indicates level of significance after post-hoc analysis with Hochberg for alpha <0.05
NB: LCN - laparoscopic camera navigation, LCC - laparoscopic circle cut
79
3.4.2 Participant questionnaire
Of all the baseline surgical and non-surgical experiences assessed, only previous laparoscopic
experience was significantly predictive of baseline surgical skill on entering residency. Students
who had observed >10 laparoscopic procedures scored significantly higher on the LCN task
(mean 63.0 points compared to 57.9 points; p=0.04), when compared to those who had observed
between 0 and 10 procedures. Furthermore students who reported laparoscopic camera
navigation experience in more than 20 operative cases performed the LCC task significantly
faster and with more accuracy (lower score) than students who had less laparoscopic experience
(means score of 240.3 points compared to 323.8 points; p =0.01). No relationship was seen
between performance on either technical task and video game or previous music instrument
experience, whether assessed by instrument played or Royal Conservatory of Music grade level
completed (Table 9).
3.4.3 Correlation of visual spatial skills and laparoscopic baseline performance
Residents who scored higher on the CC test demonstrated more accurate LCN path length (rs(PL)
=-0.36, p=0.03) and angle path (rs(AP) =-0.426, p=0.01) scores during the LCN task. However,
participants’ time to complete the LCN task was not significantly associated with their CC test
scores (rs(time) =-0.04, p=0.84). No significant correlations were identified between LCN metrics
and PicSOr (rs(PL) =0.19, p=0.25, rs(AP) =0.25, p=0.141, rs(time) =-0.27, p=0.14) or CR (rs(PL) =-
0.06, p=0.74, rs(AP) =-0.06, p=0.74, rs(time) =-0.07, p=0.70) test scores. In addition, no significant
correlations were observed between LCC time, error and total score when compared to all the
three visual spatial tests (Table 10).
When examining the relationship between individual visual spatial tests a significant correlation
was seen between CC and CR tests (rs =0.33, p=0.05), suggesting that these tests measure
related visual spatial abilities. However, no correlation was seen between the PicSOr and CC or
CR tests (rs =-0.01, p=0.93 and rs =-0.07, p=0.69 respectively).
Table 10
Name of t
Laparo
naviga
Laparo
*Indicates
3.5
Recen
techn
techn
to rea
to im
chara
corre
Few s
basel
found
0: Correlation
echnical task
oscopic camera
ation
Time
Path Length
Angular Pat
Combined S
oscopic circle c
Time
Error
Combined s
s statistical sign
Discuss
nt evidence
nically comp
nical aptitude
ach technica
mplement. Ho
acteristics an
late with bas
studies have
ine laparosc
d that those w
n of 2D-3D
PicSO
a
0.27
h 0.19
h 0.25
Score 0.22
cut
-0.08
0.13
core -0.05
nificance for an
sion
has challeng
ponent surgeo
e into the ex
l competenc
owever, desp
nd visual spa
seline simula
e investigated
copic skills in
who had ove
innate abilit
Or p - value
7 0.11
9 0.25
0.14
2 0.19
8 0.66
0.44
5 0.77
n alpha <0.05
ged the tradit
ons given de
isting selecti
cy during trai
pite investiga
atial tests, on
ated laparos
d the relation
n novice trai
er 20 hours o
ty tests with
Name of
Card rotati
test
-0.07
-0.06
-0.07
-0.08
0.009
-0.18
-0.16
tional assum
edicated time
ion process t
ining is not o
ating a broad
nly previous
copic perfor
nships betwe
inees. In a st
of experience
laparoscopic
f visual spatial
ion p - valu
0.70
0.74
0.66
0.64
0.96
0.29
0.35
mption that a
e and practic
to identify c
only appeali
d range of po
laparoscopic
rmance.
een previous
tudy of 25 m
e assisting w
c surgical sk
test
ue
Cube
comparis
n test
0.04
-0.36
-0.43
-0.42
0.19
0.11
0.14
ll trainees w
ce. Incorpora
candidates w
ing, but coul
otentially pr
c experience
s surgical ex
medical stude
with laparosc
kill
so p - value
0.84
0.03*
0.009*
0.01*
0.26
0.52
0.4
will become
ating a meas
who are more
ld be relative
redictive bac
e seemed to
xperience and
ents, Benarje
copic operati
80
sure of
e likely
ely easy
ckground
d
ee et al.
ions had
81
significantly faster performance on an intracorporeal knot tying task (Banerjee, Cosentino,
Hatzmann, & Noe, 2010a; Buckley et al., 2014). Similarly, in the present study, residents who
had previously navigated a laparoscopic camera in more than 20 cases scored better on the LCC
task compared to trainees with less experience. Additionally, those who had observed more than
10 hours of laparoscopic surgery performed significantly better on the LCN task. These findings
suggest that exposure to laparoscopy during medical school may be one way to gauge initial
performance on basic laparoscopic tasks.
Other non-surgical characteristics that can be obtained through a questionnaire such as: gender,
handedness, typing performance, musical instrument ability, sewing, self-perceived dexterity,
and interest in surgery and sports, have repeatedly been shown to not be predictive of surgical
skill in a large number of studies including the present study (Cope & Fenton-Lee, 2008;
Lokuge, 2012; Madan et al., 2008). However, in contrast to the present study, increased video
game experience has been previously shown to correlate with superior baseline laparoscopic
performance (Hislop et al., 2006; Nomura et al., 2008; Paschold et al., 2011; Van Hove et al.,
2008). Paschold et al. reported that video game experience correlated with better VR
performance in two tasks (grasping and retracting tissue, and applying clips on the cystic duct
and artery) (Paschold et al., 2011). In the present study, the low number of participants in each
video game subgroup may explain the discrepancy in these findings. Nevertheless, even when
gamers exhibit an initial advantage in performance, Van Hove et al. reported that their
performance equalizes with that of non-gamers following even modest amounts of practice (Van
Hove et al., 2008). If the technical gaming advantage can be quickly learned, quantifying
gaming experience is of little utility for surgical selection. Therefore, while non-surgical
information obtained through questionnaires or curriculae vitae, may have some value in
highlighting extracurricular accomplishments and interests among candidates for surgical
training, this information does not appear to predict initial technical performance in the
contemporary training milieu.
While it is generally accepted that visual spatial 2D-3D conversion ability is required to perform
laparoscopic and endoscopic procedures, there is considerable variability in the reported
correlations between the results of formal visual spatial testing and performance on these
procedures. As a result, the reliability of formal visual spatial testing is questionable. In the
present study, residents who obtained a higher score on the CC test performed better on the VR
82
LCN task in two of three metrics, but not on the LCC box-training task. Furthermore, despite a
significant relationship between participant’s performance on the CR and CC task, no
correlation was identified between CR test scores and performance on either of the studied
laparoscopic tasks. In keeping with these inconsistencies, Enochsson et al. found that the CR test
did not reliably predict simulated gastroscopy performance in the novice trainee. Additionally,
the authors found that expert gastroscopists performed worse on the CR test when compared to
novice trainees (L. Enochsson et al., 2006). In contrast to these findings, Arora et al. reported a
positive correlation between medical students’ CC, CR and PicSOr test results, and their
performance on simulated endoscopic sinus surgery (H. Arora et al., 2005; Westman et al.,
2006). The PicSOr test, while designed specifically to test the perceptual ability thought to be
required for laparoscopy, has also demonstrated inconsistent associations with surgical
performance. Gallagher et al. reported positive correlations between surgical trainees’ PicSOr
results and their performance on the LCC task, and Kolozsvari reported similarly positive
correlations with laparoscopic peg transfer performance in the novice trainee. In contrast
Stefanidis et al. demonstrated no correlation between PicSOr test results and a range of
laparoscopic simulator tasks (D. Stefanidis, Korndorffer, et al., 2006). This is in keeping with
the results of the present study, where the PicSOr test failed to predict performance on any
simulated laparoscopic tasks. Similarly Thus, it would appear that although visual spatial ability
is thought it be essential for laparoscopic performance, the performance scores generated by the
PicSOR, CC and CR tests do not appear to accurately and consistently predict baseline technical
skill performance in novice trainees entering surgical residency. As a result, these tests in their
current form do not appear to be sufficiently reliable for use during trainee selection.
We acknowledge several limitations to the present study. First, due to the structure of the final
bell ringer examination, there was only enough time for a single attempt at the LCC.
Recognizing that baseline skill assessment can vary substantially in novice trainees between
consecutive attempts, using either the best or average score of three trials may have improved
the accuracy of assessment by reducing the effect of test-retest variance. Second, we were
limited by the relatively small size of our study cohort, potentially increasing the risk of a type II
error in our statistical analyses, most notably with respect to comparisons made using ANOVA.
Finally, it is possible that participants who were entering surgical specialties with limited or no
laparoscopic procedures may have been less motivated to perform well, biasing the results.
83
However, it would appear that many of these residents declined to participate in the study in the
first place, given the large number of non-participants who cited this reason. Nevertheless,
despite these limitations, we believe that the concurrent assessment of multiple different
potential predictors of baseline laparoscopic skills in novice trainees in our study provides
valuable, novel evidence concerning the reliability of these assessment tools.
The findings of the present study suggest that trainees who have previously had the opportunity
to observe and participate in laparoscopic cases have better baseline laparoscopic skills.
However, surgical technical skills, and the aptitude to acquire them, do not appear to transfer
from other life exposures or non-surgical experiences. Furthermore, formal tests of visual spatial
ability have demonstrated inconsistent associations with technical skills. The execution of any
given technical task is likely to be the result of a complex matrix of aptitudes that combine in
different ways and result in varied performance levels. Therefore, isolated formal tests of visual
spatial ability likely oversimplify this matrix of aptitudes, resulting in the previously described
inconsistent relationships with technical performance. Given these findings, instead of relying
on surrogate markers, future work may be best directed toward assessing the predictive value of
performance on higher-fidelity tasks that more closely replicate actual surgical tasks. Examples
might include performance of tasks in the simulation laboratory that have been validated and
shown to transfer into the operating room.
While the use of selection tests for incoming surgical trainees that predict future technical skill
performance would be beneficial in optimizing the technical competence of graduating
surgeons, surrogate markers such as non-surgical experience and visual spatial tests do not
appear to be reliable predictors. Given the poor predictive ability of the range of factors and tests
evaluated in the present study, and the conflicting data in the literature, it is reasonable to
conclude that novel approaches are required to identify reliable predictors of technical skill
performance in surgical trainees, and to identify those candidates most likely to reach technical
competence in the contemporary training environment.
Cha4
resi
The c
Amer
4.1
Back
surge
study
progr
comp
techn
Meth
ended
of > 0
thresh
Resu
respe
felt th
excel
surge
subcu
Conc
comp
in the
are ap
apter 4: O
idents - a
content of th
rican Colleg
Abstrac
kground: Su
eons, howeve
y was to 1) id
ram directors
petence and 3
nical skills th
hods: Delphi
d questionna
0.8 with 80%
hold for con
ults: The first
ectively. Onl
hat 5-15% of
llent (alpha =
ery. Technica
uticular sutu
clusion: PDs
petence in te
e selection p
ppropriate fo
Optimiz
a nationa
his chapter ha
ge of Surgeon
ct
urgical progr
er selection p
dentify the c
s’ (PDs) opi
3) establish
hat would be
i consensus m
aire, formula
% of respons
nsensus.
t and second
y 2 program
f residents h
= 0.92). The
al skills felt
re) and lapar
s indicate tha
chnical skill
rocess. Cons
or inclusion
ing the s
al consen
as been subm
ns Accredite
ams strive to
processes va
urrent Canad
nions on the
consensus o
e most indica
methodology
ated as a 5 po
ses in agreem
d rounds wer
ms reported a
had difficulty
top traits fo
to be most a
roscopic task
at 5-15% of
l. Despite thi
sensus amon
into the sele
selection
nsus
mitted for rev
ed Education
o recruit train
ary between
dian general
e proportion
n the desired
ative of futur
y was used.
oint Likert sc
ment (4 - agr
re completed
assessing tech
y reaching co
or success in
appropriate w
ks (coordina
graduating r
is, objectivel
ng PDs sugg
ection proces
n of gene
view and ha
n Instituted
nees who wi
institutions.
l surgery (GS
of trainees w
d attributes o
re performan
An open-en
cale, was ad
ree and 5 – st
d by 14 and
hnical skill,
ompetence in
GS included
were open ta
ation, graspin
residents had
ly assessing
gests that bas
ss.
eral surg
as been peer
ill graduate
. The purpos
S) selection
who have dif
of GS candid
nce.
nded, followe
dministered.
trongly agre
11, of a pote
however, th
n this area. C
d: work ethi
asks (one-han
ng and cuttin
d difficulty r
this domain
sic open and
gery
reviewed by
as competen
se of the pres
processes 2)
fficulty achi
dates, and th
ed by a close
A Cronbach
ee) determine
ential 17, GS
he majority o
Consensus w
c, passion fo
nded tie and
ng).
reaching
n is rarely inc
laparoscopi
84
y the
nt
sent
) solicit
ieving
he
ed-
h’s alpha
ed the
S PDs,
of PDs
was
or
d
cluded
ic skills
85
4.1.1 Background
Surgical programs strive to recruit trainees who will graduate as competent surgeons. To
structure entry into surgical post-graduate training programs in North America, national match
systems are used to pair final year medical students to specialty programs (e.g. general surgery
(GS) or neurosurgery) (CaRMS, 2016; NRMP, 2016). Therefore, unlike many countries,
medical students are admitted directly into a surgical specialty program without completing an
internship or advancing from basic to advanced surgical training (Condon et al., 2013; Erasmus,
2012; Anthony G. Gallagher, Leonard, & Traynor, 2009; Anthony G. Gallagher et al., 2008;
Levitt & Klein, 1991). Thus, Canadian and American programs are in a unique position because
they are selecting candidates into specialty, without having the opportunity to assess their
independent performance in the clinical environment or their acquisition of technical skill in the
operating room.
Intuitively, however, students who apply to enter surgical training likely enjoy working with
their hands and may self-select as better technicians. It has been reported that students who
apply to surgical specialties have a higher self-perceived confidence in their manual dexterity
and ability to “work well with their hands,” as compared to their medical colleagues (J. Y. Lee,
Kerbl, McDougall, & Mucksavage, 2012; Van Hove et al., 2008). However, when comparing
these two groups with objective technical skill assessment metrics, the incoming surgical
trainees do not outperform the internist (Cope & Fenton-Lee, 2008; Panait et al., 2011). Self-
selection cannot be relied upon to ensure that surgical applicants have a high potential for
technical performance and therefore it may be appropriate that surgical programs are given the
responsibility to make this assessment instead.
In the current North American system, technical skill is not routinely a component of the
selecting process. This may be due to the strong belief, supported by Ericsson learning theory,
that ongoing practice and mentorship will eventually translate into expert performance (K. A.
Ericsson, 2007). However, with work-hour restrictions, increasing complexity of surgical
techniques and increased patient safety concerns, the feasibility of this model has been
challenged (Kothari & Ponce, 2014). It has been reported that trainees are not reaching their
expected technical milestones by the end of training, which are then reflecting in their
performance at the fellowship level (Antiel, Thompson, Camp, Thompson, & Farley, 2012).
Unite
indep
30 m
benef
able r
Howe
McIn
There
the G
propo
traine
techn
4.2
Rese
The U
4.2.1
All C
progr
contr
Monk
proce
match
weigh
asked
evalu
reach
ed States fell
pendently pe
inutes durin
ficial for trai
reach techni
ever, there is
nnes, & Sing
efore, the pu
GS selection
ortion of trai
ees; and 3) e
nical skills th
Method
earch ethic
University o
1 Curre
Canadian GS
rams are stru
rast to indivi
key (Palo Al
ess across th
h system (Ca
hted score fo
d whether ap
uated during
hing compete
lowship PDs
erform a lapa
g a major pr
ining progra
cal compete
s a lack of ev
gh, 2013).
urpose of the
process at di
inees who do
establish nati
hat would be
ds
cs
f Toronto Et
ent selectio
program dir
uctured unde
dual hospita
lto, CA), wa
e country. A
aRMS- Cana
or each comp
pplicants’ cli
the selection
ence in these
s reported th
aroscopic ch
rocedure (Ma
ms to adjust
nce within th
vidence to g
e present stud
ifferent insti
o not achieve
ional consen
e most indica
thics Review
on practice
rectors (PDs
er the umbrel
al programs.
as used to ide
Although the
adian Reside
ponent of th
inical knowl
n process, an
e three doma
at a significa
olecystectom
attar et al., 2
t the GS sele
he restriction
uide this asp
dy was to: 1)
itutions; 2) s
e the minimu
nsus on the d
ative of futur
w Board appr
es
s) were invit
lla of a Univ
An online q
entify the cu
written appl
ent Matching
e application
edge, decisio
nd what perc
ains by the ti
ant proportio
my or operat
2013). Given
ection proces
ns of the cur
pect of the se
) identify the
solicit progra
um standard
desired attrib
re performan
roved this st
ed to partici
versity with
questionnaire
urrent compo
lication is st
g Service), P
n at their ins
on-making a
cent of traine
ime of gradu
on of GS fel
te unsupervi
n these repor
ss to recruit
rrent training
election proc
e current com
am directors
ds expected o
butes of GS c
nce.
tudy.
pate. In Can
a recognized
e, administer
onents used i
tandardized b
PDs were ask
stitution. In a
and technica
ees they felt
uation.
llows could n
sed for more
rts, it may be
applicants w
g environme
cess (Kenny
mponents us
’ opinions o
of graduating
candidates, a
nada, all train
d medical sc
red using Su
in the GS sel
by the nation
ked to provi
addition, PD
al skill were
had difficul
86
not
e than
e
who are
ent.
y,
sed in
n the
g
and the
ning
chool, in
urvey
lection
nal
de the
Ds were
lty
87
4.2.2 Delphi consensus methodology
A Delphi questionnaire was administered to gain consensus on which candidate-specific
attributes are important for residents to succeed in GS training. In addition, consensus was
sought on the simulated technical skills (both open and laparoscopic) that are most likely to be
indicative of a trainee’s aptitude to acquire more complex surgical skills and thus future
performance.
The Delphi methodology was originally developed in the 1950s by the RAND Corporation to
evaluate trends in technology on warfare, but continues to be widely used to create public
policy, clinical guidelines or to formulate training recommendations by aggregating the opinions
of experts, where little empirical evidence is available (Elissen, Struijs, Baan, & Ruwaard, 2015;
Loeffen et al., 2015; RAND, 1976; B. Zevin, Levy Js Fau - Satava, Satava Rm Fau -
Grantcharov, & Grantcharov). This methodology is comprised of four essential components: an
expert panel, the promotion of anonymous responses, multiple rounds of questions, and
statistical feedback to encourage convergence of responses until an acceptable consensus is met
(RAND, 1976).
4.2.3 Expert panel
Canadian general surgery PDs were invited by email to participate in this Delphi process. This
group of individuals was selected to participate due to their unique expertise with trainee
selection, as acquired through leadership in this process at their respective institutions.
4.2.4 Anonymity
A strength of the Delphi technique is that it protects against bias by prohibiting face-to-face
contact amongst the panel members, thus decreasing dominant verbal opinions, seniority or in-
person arguments which have been reported to sway the panel (Cuhls, 2005; Murphy et al.,
88
1998). In the present study, individual anonymous opinions were encouraged through an online
questionnaire, limiting the risk of interaction between panel members (Murphy et al., 1998).
4.2.5 Rounds of questions
The Delphi process calls for a minimum of two rounds of questions, with the first open-ended
and the second closed-ended (Cuhls, 2005; Powell, 2002). Open-ended questions encourage
responses from the expert panel without directing their opinions to multiple-choice answers
(Powell, 2002). In the present study, the first round of open-ended questions was then
supplemented with literature in the field to ensure completeness for the second-round of closed-
ended questions. In the second round, responses to closed-ended questions are solicited using a
Likert scale to allow for statistical feedback to panelists, and to encourage convergence of
responses to create consensus (Jairath & Weinstein, 1994). In our study, the closed-ended
questions were formulated on a 5-point Likert scale (1- strongly disagree, 2 – disagree, 3 -
somewhat agree, 4 – agree, 5 - strongly agree).
4.2.6 Consensus
Consensus for each section of the Delphi is calculated with a Cronbach’s alpha, which is a
statistical measure of internal consistency or homogeneity of expert responses. A Cronbach’s
alpha of ≥0.8 has previously been reported to be an acceptable benchmark for consensus and
was therefore used as the cutoff for this study (Graham, Regehr, & Wright, 2003). Subsequent
rounds are performed until consensus is met. The present study met consensus after the second
round.
4.2.7 Recommendations for incorporation into selection
Once consensus was achieved, each questionnaire item was assessed for whether the agreement
among experts was found to be positive, neutral or negative. Only items that reached positive
agree
was a
agree
either
attrib
Satav
4.3
4.3.1
Of th
round
count
proce
follow
the w
progr
perce
were
above
was d
interv
script
of 14
interv
statio
The m
clinic
ement were r
achieved wh
e) on the Lik
r 1 (strongly
butes or skill
va, & Grantc
Results
1 Curre
he 17 GS pro
ds of the que
try, with at l
ess was cons
wed by an in
written applic
rams: person
eived dedicat
then ranked
e the program
determined b
view score (w
ted interview
4 assess clini
view scenari
ons and refer
majority of P
cal knowledg
recommende
hen >80% of
kert scale. Ne
y disagree) or
s into selecti
charov, 2012
s
ent selectio
ogram directo
estionnaire, r
east one GS
sistently repo
n-person inte
cation, howe
nal statement
tion to GS 5
d and in-pers
m specific cu
by combinin
weighted 40
w scenarios t
cal knowled
ios. Finally,
rence letters.
PDs felt that
ge (range 0-
ed for incorp
f experts rate
egative agree
r 2 (disagree
ion. Neutral
2).
on process
ors across C
respectively.
program res
orted to occu
erview. All p
ever the weig
t 5-25%, cur
-20%, and re
on interview
utoff. Follow
g the written
0-75%). Ten
to assist with
dge using tran
only 2 of 14
.
less than 5%
10%), that 5
poration into
ed the attribu
ement was e
e) on the Lik
agreement i
anada, 14 an
. Excellent r
sponding fro
ur in two stag
programs rep
ghted score o
rriculum vita
eference lett
ws offered to
wing the inte
n application
n of 14 progr
h decision m
nscripts, refe
4 programs a
% of trainees
-10% of trai
o the selectio
ute or skill as
established w
kert scale, op
included all
nd 11 partici
representatio
om each prov
ges: review
ported using
of the compo
ae 10-35%, r
ters 10-40%
o those candi
erview, each
n score (weig
rams reporte
making at the
ference letter
assess techni
s have difficu
inees has dif
on process. P
s a 4 (agree)
when >80%
pposing inclu
other respon
ipated in the
on was achie
vince (Table
of written ap
g a structured
onents varie
research invo
. Candidate’
idates who a
h applicant’s
ghted 25-60%
ed using refe
e time of can
rs, curriculum
cal skill thro
ulty reachin
fficulty in de
Positive agre
or 5 (strong
of response
usion of thos
nses (B Zevi
first and sec
ved across th
e 11). The se
pplications,
d review pro
d between
olvement 10
’s application
achieved sco
final rank sc
%) and the
rence letters
ndidate select
m vitae and
ough simulat
g competenc
ecision-maki
89
ement
gly
s were
se
in, Levy,
cond
he
election
cess for
0-15%,
n scores
res
core
s and
tion. Six
ted
ce in
ing
90
(range 0-20%), and that 5-15% (range 0-15%) in technical skill by the time of completion of
training (Table 12).
Table 11: Participating Canadian General Surgery Programs
Participating Programs (n=14) Location
McMaster University* Hamilton, Ontario
University of Montreal* Montreal, Quebec
University of Manitoba* Winnipeg, Manitoba
Dalhousie University Halifax, Nova Scotia
Memorial University St. John, Newfoundland
Sherbrooke University Montreal, Quebec
University of Alberta Edmonton, Alberta
University of British Columbia Vancouver, British
Columbia
University of Calgary Calgary, Alberta
University of Ottawa Ottawa, Ontario
Queens University Kingston, Ontario
University of Saskatchewan Saskatoon, Saskatchewan
University of Toronto Toronto, Ontario
Western University London, Ontario
91
* Programs that only participated in the first round
4.3.2 Delphi consensus methodology
The overall consensus for the questionnaire was excellent, with a Cronbach’s alpha of 0.92.
Internal consensus was reached for each of the four sections of the questionnaire, namely: 1)
desired candidate attributes, 2) open surgical skills, 3) virtual reality (VR) simulation skills and
4) laparoscopic box training skills, with a Cronbach’s alpha of 0.87, 0.96, 0.92 and 0.80
respectively.
92
Table 12: General Surgery program director's responses to clinical knowledge, decision-making and technical skill during selection
and at the time of graduation
During selection At the time of graduation
Does your program assess the following areas during selection? What percent of trainees do not reach competency in the
following three areas by the time of graduation?
Yes No If yes, how? 0-1% 2-4% 5-10% 11-15% 16-20%
Clinical knowledge 6 8 Transcripts, reference letters, curriculum vitae, interview
clinical scenarios 5 7 2 0 0
Decision making 10 4 Interview scenarios, references 4 1 9 0 1
Technical skill 2 12 Technical skills station, reference letters 3 2 6 4 0
93
Table 13: Desired candidate attributes for selection into General Surgery
Attributes Cronbach's alpha = 0.87 Median (IQR) Consensus
Work ethic 5 (5,5) positive
Passion for General Surgery 5 (5,5) positive
Professionalism 5 (4,5) positive
Ability to work in a team 5 (4,5) positive
Sound judgment 5 (4,5) positive
Ability to make decisions 5 (4,5) positive
Assimilates information to formulate an opinion 5 (4,5) positive
Independence 4 (4,5) positive
Ability to multitask 4 (4,5) positive
Technical skill 4 (4,5) positive
Ability to accept criticism 4 (4,5) positive
Ability to think quick on his/her feet 4 (4,5) positive
Collaborative skills 4 (4,5) positive
Cool under pressure 4 (4,5) positive
Humility to learn 4 (4,5) positive
Communication 4 (4,5) positive
Confidence 4 (4,4) positive
Trainability 4 (3,4) neutral
Leadership 4 (3,4) neutral
IQR: interquartile range
94
Table 14: Appropriate simulated surgical skills for selection into General Surgery
Open Skills Median (IQR) Consensus
Cronbach's alpha = 0.96
*One handed ties 4 (4,5) Positive
*Interrupted subcuticular suturing 4 (4,4) Positive
*Running subcuticular suturing 4 (4,4) Positive
Horizontal mattress 4 (4,4) Neutral
Vertical mattress 4 (4,4) Neutral
Nevus removal 4 (4,4) Neutral
Simple interrupted suture 4 (3,5) Neutral
Two handed tie 4 (3,4) Neutral
Hand sewn bowel anastomosis 4 (2,5) Neutral
Chest tube insertion 3 (3,5) Neutral
Virtual Reality Simulation Skills
Cronbach's alpha = 0.92
*Coordination 4 (4,4) Positive
*Grasping 4 (4,4) Positive
*Cutting 4 (4,4) Positive
Camera navigation 4 (3,4) Neutral
Instrument navigation 4 (3,4) Neutral
95
Clip applying 4 (3,4) Neutral
Lifting and grasping 4 (3,4) Neutral
Bowel handling 4 (3,4) Neutral
Precision Task 4 (3,4) Neutral
Hand eye coordination 4 (3,4) Neutral
Fine dissection 4 (2,4) Neutral
Catheter insertion 4 (2,4) Neutral
Colonoscopy 4 (2,4) Neutral
Seal and cut 3 (2,4) Neutral
Gastroscopy 3 (2,4) Neutral
Laparoscopic Box Trainer Skills
Crobach's alpha = 0.80
Peg transfer 3 (3,4) Neutral
Laparoscopic circle cut 3 (3,4) Neutral
Extracorporeal Knot tying 3 (2,4) Neutral
Endoloop placement 3 (3,4) Neutral
Intracorporeal knot tying 4 (2,4) Neutral
Camera navigation 4 (3,4) Neutral
Running intracorporeal suture 4 (2,4) Neutral
Cholecystectomy on porcine bowel model 4 (2,4) Neutral
* inclusion into selection of technical skill; IQR: interquartile range
Si
pr
ag
ju
al
th
th
cu
4
Se
em
w
sk
gr
th
op
ad
at
nu
ha
(c
Th
di
(<
re
cl
ed
w
ixteen of the
rocesses. Th
greement) in
udgment, abi
lthough lowe
he 10 open ta
he one-hande
utting. All ot
Disc4.4
electing can
mpirical evid
written applic
kill. With the
raduating res
his, technical
pinion from
ddition to se
ttributes are
umber of tec
anded tie, su
coordination
he specific c
ifferences in
<5%), decisi
eported by P
linical know
ducation sys
will have alre
e 18 individu
he top 7 (high
ncluded: wor
ility to make
er on the des
asks and 15 V
ed tie, subcu
ther skills re
cussion
didates for s
dence to gui
cation, follow
e current me
sidents had d
l skill is rare
PDs regardi
lection, a De
beneficial fo
chnical skills
ubcuticular in
, grasping an
components
n the proport
on-making (
Ds. It appea
wledge, which
tem and scre
eady repeated
ual personali
hest median
rk ethic, pass
e decisions a
sired candida
VR tasks lis
uticular interr
eached neutra
surgical train
de the proce
wed by an in
ethods of sele
difficulty rea
ely incorpora
ing desired c
elphi consen
or success in
s suitable for
nterrupted su
nd cutting).
included in
ion of reside
(5-10%) and
ars that the le
h may be att
eening proce
dly demonst
ity attributes
and smalles
sion for surg
and assimilat
ate attributes
ted, 3 of eac
rupted suture
al consensus
ning is a chal
ess. Current s
n-person inte
ecting reside
aching techn
ated into the
candidate att
nsus was con
n GS residen
r incorporati
uture and co
current train
ents who had
d technical sk
east amount
tributed to th
ess. To quali
rated academ
s met the crit
st interquarti
gery, profess
te informatio
s list, also m
ch met the cr
e, continuou
s (Table 14).
llenging and
selection in C
erview, with
ents, the maj
nical skill at
selection pr
tributes and t
nducted. The
ncy. Furtherm
ion into selec
ntinuous sut
nee screening
d difficulty r
kill (5-15%)
of trainees a
he considerat
ify for applic
mic excellen
teria for incl
le range – si
sionalism, ab
on to formula
met the criteri
riteria for inc
us suture and
.
d important r
Canada cons
no formal a
jority of PDs
the complet
rocess. In ord
technical ski
e results sugg
more, consen
ction. These
ture) as well
g and selecti
reaching com
by completi
are reported
tion of this d
cation into re
nce by retain
lusion into th
ignifying gre
bility to work
ate an opinio
ia for inclusi
clusion into
d coordinatio
responsibility
sists of a stru
assessment o
s reported th
ion of surgic
der to establ
ills that are a
gest that a nu
nsus was rea
e included ba
l as basic lap
ion processe
mpetence in
ion of surgic
to not reach
domain with
esidency trai
ning knowled
he selection
eatest level o
k in a team,
on. Technica
ion (Table 1
selection, na
on, grasping
y that lacks
uctured revie
f incoming t
hat 10-15% o
cal training.
ish a collect
appropriate f
umber of dif
ached concer
asic open tas
paroscopic sk
es may expla
clinical know
cal training,
competency
hin the existin
ining, candid
dge in order
96
of
sound
al ability,
3). Of
amely
and
ew of a
technical
of
Despite
tive
for
fferent
rning a
sks (one-
kills
ain the
wledge
as
y in
ng
dates
to score
97
in the top tier of students on written exams at the undergraduate level, as well as on The Medical College
Admission Test (MCAT), which has been reported to predict training examination scores ("Admission
requirements for Medical School at the Univeristy of Toronto," 2015; "Admission requirements
University of British Columbia ", 2015; "Medical School Admission Requirement Dalhousie ", 2015;
Ronai, Golmon, Shanks, Schafer, & Brunner, 1984). Furthermore, all candidates will have successfully
completed 2-3 years of medical school, and thus will have demonstrated their academic potential in terms
of learning and retaining clinical knowledge. However, the same screening processes are not available to
assess decision-making and technical skill. In the current system, both of these domains have a more
prominent focus in residency as compared to medical school, and therefore it is understandable that
potential deficits are not uncovered until after admission. However, PDs report that fewer students have
difficulty with decision-making as compared to technical skill. Perhaps this is also a result of the
screening process currently in place, as decision-making is already incorporated in the interview
component of the selection process at a majority (10 of 14) of institutions. Furthermore the interview
score has been reported as a independent predictor of successfully completing GS training in a single
study (Alterman, Jones, Heidel, Daley, & Goldman, 2011). In contrast, only 2 of 14 programs assess
technical skill during selection. As a result, many trainees enter surgical programs with minimal or no
screening in this domain. Given the notable proportion of trainees who PDs believed to have difficulty
reaching competence, and that technical skill is an intrinsic part of surgical practice, it may be worth
considering the incorporation of technical skill testing into selection.
A large number of candidate attributes were thought to be necessary for success in GS, however the
current selection system poorly differentiates these qualities. In this study, 16 of 18 attributes listed by
the PDs during the first round of the Delphi gained positive consensus during the second round.
Currently, these attributes e.g. work ethic and professionalism, are evaluated at the time of interview with
a subjective opinion rather than an objective measure. In an attempt to measure this qualitative area
objectively, Bell et al. had 535 candidates across the Unites States complete the TriMetrix Personal
Talent Report questionnaire: a questionnaire designed to assess behavioral motivators and student
attributes. The authors then compared the candidate behavioral profiles’ to the final ranks lists at different
institutions and found that individuals with diverse attributes can be similarly attractive candidates for
surgical training (R. M. Bell, Fann, Morrison, & Lisk, 2011; Richard M. Bell, Fann, Morrison, & Lisk,
2012). Therefore, it seems that the current interview system is not able to separate candidates with
specific attributes, but instead selects for a large range of different qualities. This may be partly due to
98
the subjective assessment method used in the current selection system to evaluate these qualities, coupled
with the lack of empirical predictive evidence to inform which of these qualities actually increase the
likelihood of success during residency training. Therefore to begin clarifying this somewhat vague area
of selection, gaining expert consensus on the desired candidates attributes is an important first step.
Further research is required to objectively assess these qualities and narrow the list with long- term
supportive evidence.
Consensus was also gained on the technical skills that may be most appropriate for incorporation into the
selection process, however, controversy remains as to how technical aptitude is best assessed in candidate
trainees. To date, most studies have focused on the use of surrogate markers such as tests of dexterity and
visual spatial ability to predict technical skill performance, based on the belief that these markers
represent the building blocks required to excel in technical tasks ("Lafayette Instrument evaluation
Dexterity Tests," 2015; D. Stefanidis, Korndorffer Jr, et al., 2006). However, the results of studies to date
have been disappointing. Although most surgeons would agree that high dexterity is of benefit in the
operating room, scoring well on these tests has not correlated with technical performance (Hoffer & Hsu,
1990; Marlies P. Schijven et al., 2004; Schueneman et al., 1985; Schueneman et al., 1984; D. Stefanidis,
Korndorffer Jr, et al., 2006). Visual spatial ability has also been widely investigated in GS, especially
since the advancements of in laparoscopic and endoscopic techniques. However, in both simulated
laparoscopic and endoscopic tasks, the results are inconsistent (Ekstrom et al., 1976; A. G. Gallagher et
al., 2003) (L. Enochsson et al., 2006; A. G. Gallagher et al., 2003; Groenier et al., 2014; E. Matt Ritter et
al., 2006; D. Stefanidis, Korndorffer Jr, et al., 2006). Therefore, although measuring aptitude through
surrogate tests seemed to be a logical approach, it appears that technical performance is more complex
than these tests are able to measure. Therefore, altering the approach to assess technical aptitude by
incorporating simulation tasks that are directly transferable to the operating room is a worthwhile
endeavor.
Assessing performance on simulated surgical tasks has been reported to differentiate trainees’ surgical
aptitude. Grantcharov et al. plotted the individual learning curves of trainees performing basic VR
simulator tasks, and noted that 8.1% of trainees lagged behind their peers and did not show any skill
improvement (Grantcharov & Funch-Jensen, 2009). Similarly, Schijven et al. reported that 20% of
residents did not reach proficiency on the laparoscopic clip and cut task after 30 trials (M. P. Schijven &
Jakimowicz, 2004). These numbers are consistent with PD reports of 10-15% trainees having difficulty-
reaching competency by the end of residency training. However, the evidence is limited to whether
pe
lo
th
in
G
fo
su
(M
in
W
th
se
(D
as
nu
na
co
un
pe
ne
op
th
co
im
in
4
Se
in
erformance o
ongitudinal d
hat this may
nterrupted in
GRS. This sco
ollow up from
uturing task
Moore et al.,
ntroducing th
We acknowle
he questionn
econd. Irresp
Day & Bobe
s long as bet
umber retain
ational conse
oncerning ta
ncertainty am
erformance.
egative opin
pinions conc
hese tasks ha
onsensus aro
mportant firs
nto the overa
Conc4.5
electing the
ndicate that 5
on a technic
data from the
be the case.
nstrument tie
ore was then
m this group
at the time o
2014). Ther
he open and
edge two ma
aire, and PD
pective of thi
va, 2005). It
tween 7 and
ned in our stu
ensus on the
ask selection
mong the exp
In the case o
nions. Specif
cerning tasks
ave not been
ound 6 skills
st step in und
all selection p
clusion
best candida
5-15% of gra
al skill durin
e Otolaryngo
In this study
s under the m
n incorporate
p suggested t
of selection w
refore, in ord
laparoscopic
in limitation
Ds from three
is, the Delph
t has been re
10 members
udy. As a res
e topic of trai
fell within t
pert panel as
of uncertaint
fically in the
s that could p
tested for pr
that would
dertaking fur
process.
ates for admi
aduating resi
ng selection
ology progra
y, applicants
microscope,
ed into their
that incorpor
was predictiv
der to assess
c skills outli
ns to our stud
e additional p
hi consensus
eported that t
s are retained
sult, we beli
inee selectio
the category
s to whether
ty, responde
case of the p
potentially b
redictive val
be most usef
rther longitu
ission into re
idents had d
is predictive
am at the Ma
s to the progr
with direct
total selectio
rating the sim
ve of residen
s whether GS
ned in this s
dy. First, PD
programs co
s methodolog
the expert pa
d (Day & Bo
ieve that our
on into GS. S
of neutral a
this type of
ents are gene
present study
be used to eli
lidity. Never
ful to assess
udinal study o
esidency is o
ifficulty reac
e of future pe
ayo Clinic (R
ram were as
assessment f
on score (Ca
mulated sim
nt technical p
S can benefi
study should
Ds from three
ontributed to
gy is robust
anel will con
obeva, 2005)
r results rema
Second, a lar
agreement. T
f assessment
erally relucta
dy, PDs may
iminate train
rtheless, the
s in candidate
of their pred
of upmost im
ching compe
erformance.
Rochester, M
ked to perfo
from a facul
arlson et al.,
mple interrupt
performance
t from a sim
d be consider
e programs d
o the first rou
to a moderat
ntinue to pro
), which is lo
ain valuable
rge majority
This may refl
is predictive
ant to expres
be reluctant
nees from en
establishme
es for GS tra
dictive value
mportance. A
etence in tec
Promising
Minnesota) su
orm simple
lty member u
2010). Long
ted microsco
e during train
milar process,
red.
did not respo
und but not t
te degree of
oduce useful
ower than th
e and represe
y of response
lect consider
e of future
ss strong pos
t to express s
ntering GS w
ent of a natio
aining repres
when incorp
Although PD
chnical skill
99
uggests
using a
gitudinal
opic
ning
,
ond to
the
f attrition
results
he
ent a
es
rable
sitive or
strong
when
onal
sents an
porated
Ds
at the
100
completion of surgical training, assessment of this domain is rarely incorporated as part of the selection
process. Consensus among PDs suggests that a number of both open (one-handed surgical tie, interrupted
and running subcuticular sutures) and laparoscopic (coordination, grasping and cutting) skills are
appropriate for inclusion. However, further assessment of whether these tasks are predictive of in-training
performance is required before they can be recommended as a robust selection metric.
C5
s
Th
su
5
B
al
Th
no
w
va
M
cu
on
on
cl
w
pe
R
pr
ta
ta
ta
si
Chapter 5
election
he content o
urgical educa
Abst.1
Background:
lthough evid
herefore the
ovice learner
whether indiv
ariable diffic
Methods: Six
urriculum th
ne month pe
ne-handed ti
lustering, wa
was used to c
erformers.
Results: Top
roficiency in
asks but not f
asks includin
asks, particip
ignificantly b
5: Practi
curricul
of this chapte
ation in simu
tract
: Emerging l
dence to iden
purposes of
rs, to determ
viduals’ learn
culty.
xty-five stud
hat included f
eriod. The tas
ie (HT) and s
as used to str
ompare the p
performers
n all tasks. M
for all laparo
ng all laparos
pants in high
better initial
ce does
la in mod
er has been s
ulation.
literature sug
ntify differen
f the present
mine whether
ning patterns
dents were tra
forty repetiti
sks included
simulated la
ratify the stu
performance
(22-35% of
Moderate per
oscopic tasks
scopic tasks
her performan
performanc
not alwa
dern sur
submitted fo
ggests that n
nt learning pa
study were
r these were
s were consi
ained and co
ions of the fo
d peg transfe
aparotomy cl
udents into fo
e between th
participants)
rformers (25-
s. Low perfo
(PT task 7.8
nce clusters
e, as well as
ays make
rgical tra
r review and
not all surgic
atterns prior
to assess pat
associated w
istent across
ompleted a p
following lap
r (PT), circle
losure (LC).
our performa
he learning cu
) and high p
-37% of part
ormers (8-15
8%; CC task
demonstrate
s a significan
e perfect
aining
d has been p
al trainees re
r to entry into
tterns of sur
with baseline
a range of o
pre-determin
paroscopic an
e cutting (CC
A data mini
ance learnin
urves and fu
erformers (3
ticipants) rea
5%) failed to
k 9.4%; and I
ed innate abi
nt sustained p
t - the ne
eer reviewed
each technic
o surgical tr
rgical skills a
e characteris
open and lap
ned, standard
nd open tech
C), intracorp
ing techniqu
ng clusters. S
urther differe
32-42% of pa
ached profic
o reach profi
IS task 15.6%
ilities that co
performance
eed for
d by an expe
cal competen
aining is lim
acquisition a
stics, and eva
paroscopic ta
dized training
hnical tasks
poreal knot t
ue, k-means
Statistical an
entiate poor
articipants) r
ciency for all
iciency four
%). For lapa
onferred
e advantage
101
ert in
nce,
mited.
among
aluate
asks of
g
over a
tie (IKT),
alysis
reached
l open
of five
aroscopic
across
al
ac
an
C
ra
de
us
tie
5
A
w
20
co
su
20
th
ac
Je
te
as
pe
w
H
te
&
fo
an
Th
ll repetitions
cross all the
nd lacked ev
Conclusion: W
ange of surgi
emonstrating
se of laparos
er performer
Back.2
A common pa
with continue
008; Gladwe
ontests this b
ubset of stud
003; Grantch
hat 5% - 17%
chieving com
ensen, 2009;
echnical com
s surgical jud
erformers (G
with diligent p
However, stud
echnical skill
& Funch-Jens
or a successf
nd surgical t
he selection
s. In contrast
skill types a
vidence of pr
Whilst most
ical tasks, lo
g highly vari
scopic techn
rs may benef
kground
aradigm in s
ed practice an
ell, 2008 ) H
belief and su
dents unable
harov & Fun
% of trainees
mpetence wi
M. P. Schij
mponents of t
dgment and
Grantcharov
practice, ulti
dies have als
ls and fail to
sen, 2009; M
ful surgical c
raining prog
process for
t, the majorit
and complex
rogression to
students wi
ow performin
iable perform
iques in surg
fit both stud
urgical educ
nd adequate
However, em
upports the n
to reach com
nch-Jensen, 2
have an inn
th minimal p
ven & Jakim
the operation
non-technic
& Funch-Jen
imately reac
so identified
o reach comp
M. P. Schijve
career, identi
grams.
surgical trai
ty of modera
ities studied
owards a plat
ll reach prof
ng trainees fa
mance score
gical practice
ents and the
cation is that
mentorship.
erging evide
notion that tra
mpetence (A
2009; M. P.
nate technica
practice or e
mowicz, 200
n, allowing t
al performan
nsen, 2009;
ching a level
d a smaller su
petence even
en & Jakimow
ifying these
inees in Nort
ate or low pe
d. Low perfor
teau phase.
ficiency with
failed to reac
s across the
e, screening
ir training pr
t all surgical
.(K.A. Erics
ence from th
ainees acqui
Alvand, Aupl
Schijven &
al ability that
ffort (top/hig
4). These stu
them to focu
nce. In contr
M. P. Schijv
of technical
ubgroup of t
n with contin
wicz, 2004)
individuals
th America d
erformers rem
rmers’ learn
h continued p
ch proficienc
entire learni
potential ca
rograms.
trainees wil
son, Krampe
he minimally
ire technical
lish, Khan, G
Jakimowicz
t allows them
gh performe
udents quick
us their atten
rast, most tra
ven & Jakim
l proficiency
trainees (8-2
nued practice
Given that t
early may b
does not rou
mained in th
ning curves w
practice and
cy with lapar
ing curve. G
andidates to
ll reach techn
e, & Tesch-R
y invasive sim
l skills at var
Gill, & Rees,
z, 2004). Rec
m to rapidly
ers) (Grantch
kly and effec
ntion on othe
ainees (63-7
mowicz, 2004
y that is acce
20%) who str
e (low perfor
technical ski
enefit both p
utinely includ
hese categori
were widely
d mentorship
roscopic task
Given the incr
identify the
nical compe
Römer, 1993
mulation lite
riable rates, w
, 2011; Cush
cent studies
acquire skil
harov & Fun
ctively learn
er competenc
0%) are mod
4). They imp
eptable and s
ruggle to lea
rmers).(Gran
ill is a requir
prospective t
de screening
102
ies
variable,
p across a
ks,
reasing
lowest
etence
3; G.,
erature
with a
hchieri,
suggest
lls,
nch-
the
cies such
derate
prove
safe.
arn
ntcharov
rement
trainees
g for
te
id
cl
st
fo
(G
re
In
un
di
Fu
G
pr
pr
pe
Th
tr
pa
ba
w
(b
co
cu
5
5
M
pa
de
echnical abil
dentified at th
linical enviro
tratifying tra
or basic lapa
Grantcharov
ealistically m
n residency,
nknown whe
ifficulty, or w
urthermore,
Given that ma
ractice, an un
rogression (f
erforming in
hus, the ove
ainees acros
atterns amon
ackground c
whether traine
basic and adv
onsistently f
urves that se
Meth.3
.3.1 Pa
Medical stude
articipate in
eemed eligib
ity. This ma
his early stag
onment (Lou
ainees into hi
roscopic sur
& Funch-Je
model the div
surgical trai
ether trainee
whether they
there are few
any surgical
nderstanding
from basic to
ndividuals w
rall purpose
ss a range of
ng trainees fo
haracteristic
ees stay with
vanced) and
fail to reach p
eparate them
hods
articipants
ents complet
the study. A
ble for inclus
ay be becaus
ge of trainin
uridas, Szasz
igh and low
rgical tasks,
ensen, 2009;
versity of the
nees general
s, classified
y are able to
w reported d
disciplines c
g of the learn
o more adva
ill remain di
of this study
f surgical task
for both basic
cs or experie
hin their lear
type (minim
proficiency o
from their p
ting their firs
All students w
sion. To ens
se of controv
ng, and uncer
z, de Montbr
performers h
or by assess
M. P. Schijv
e surgical tra
lly progress
as poor perf
overcome th
data concerni
combine min
ning progres
anced) is nee
isadvantaged
y was to asse
ks. Specifica
c and more a
nces explain
rning curve p
mally invasiv
on simulated
peers.
st or second
willing to co
sure that part
versy as to w
rtainty wheth
run, Harris, &
has been don
ing perform
ven & Jakim
aining enviro
from basic t
formers on b
hese shortco
ing variabili
nimally inva
ssion for both
eded before i
d throughout
ess the diffe
ally, the goa
advanced lap
ned potential
patterns acro
ve and open)
d tasks, and d
year at the U
ommit to one
ticipants we
whether low p
her they are
& Grantchar
ne by assess
mance on a sin
mowicz, 2004
onment.
to more adva
basic tasks, s
omings given
ity in perform
asive and op
th types of sk
it can be dete
t their trainin
erences in tec
als were to: 1
paroscopic a
l differences
oss simulated
); and 4) iden
determine th
University o
e month of te
ere at the beg
performers c
truly disadv
rov, 206). To
sing individu
ngle simulat
4). However
anced techni
struggle acro
n diverse tec
mance for op
en technique
kills over the
ermined whe
ng.
chnical perfo
1) quantify d
and open skil
s in performa
d tasks of va
ntify the sub
he features o
of Toronto w
echnical skil
ginning of th
can be accur
vantaged in t
o date, attem
ual learning c
ted operation
r, these may
ical skills. It
oss tasks of v
chnical exper
pen surgical
es in their cl
e continuum
ether initiall
ormance by
different lear
lls; 2) assess
ance; 3) dete
arying difficu
bset of traine
of their learn
were eligible
lls training w
heir technica
103
rately
the
mpts at
curves
n
not
t is
variable
riences.
skills.
linical
m of skill
y low
novice
rning
s whether
ermine
ulty
ees who
ning
to
were
l skills
104
learning curve, the following exclusion criteria were applied: 1) previous involvement in a dedicated
technical skills training program, or 2) participation in more than two suturing workshops offered during
the pre-clerkship medical school curriculum. Ethics review board approval was obtained through the
University of Toronto, and informed consent was obtained from each participant prior to the start of the
study.
5.3.2 Sample Size
An a priori sample size calculation was performed by modifying the equation commonly used for studies
that estimate the proportion of a population with specified precision for a binary outcome (Merril, 2015).
The primary outcome measure for this study was the proportion of participants demonstrating
improvement in technical skill following training. Consequently, participants could be found to be either
responders or non-responders for the primary outcome measure. The literature suggests that 8-20% of
surgical trainees are non-responders. Therefore, the proportion of non-responders in the overall
population (p) was assumed to be 15% for the purposes of the sample size calculation (Grantcharov &
Funch-Jensen, 2009; M. P. Schijven & Jakimowicz, 2004; Sir Alfred, 2003). To ensure a 95% likelihood
that the study sample included at least 4 non-responders (ie. assuming a sample size of n and precision of
d, that n(p – d) = 4), the desired precision was defined as d = p – 4/n. Using the equation for estimating
population proportions with a given precision n where n = sample size, z = 1.96 (for 95%
confidence interval), p = 0.15, and d = p-4/n, a desired sample size of 64 participants was calculated.
5.3.3 Demographic questionnaire
At the start of the study each participant completed a questionnaire assessing a range of demographic
characteristics and previous non-surgical experiences that may potentially be associated with technical
skill performance. Demographic characteristics included age, gender and handedness, whereas non-
surgical experiences included interest in surgery, self-perceived technical ability, and involvement in
music, sports and video games.
5
5
St
la
tim
ad
in
te
re
5
Th
re
di
ta
us
5
E
ta
po
fi
th
re
.3.4 Tr
S.3.4.1
tudents com
aparoscopic s
mes per wee
dvanced skil
nstructional v
est, each stud
epetitions of
I.3.4.2
he student to
ecommendat
ifferences be
aught the ski
sing a task-s
D.3.4.3
ach repetitio
asks, the insi
ortable video
eld. For the
he field of vi
epetition num
raining cur
Structure o
mpleted a one
skills. The p
ek. The first
lls. At the be
video for eac
dent then com
f each task w
Instructors
o instructor r
tions in the l
etween traine
lls over the s
specific chec
Data collec
on was video
de view of t
o recording d
purposes of
ew of the ca
mber in a de-
rriculum
of skills cu
e month distr
program com
two weeks w
eginning of e
ch task, and
mpleted four
were perform
s and feedb
ratio was ma
iterature.(Du
ers on the stu
study period
cklist.
ction
o recorded to
he training b
device was u
f assessment
amera to allo
-identified m
rriculum
ributed pract
mprised a mi
were dedicat
each two we
then comple
r more repet
ed, for a tota
back
aintained at 4
ubrowski &
udents’ tech
d, and instruc
o allow for su
box was capt
used to captu
and analysis
ow each reco
manner.
tice training
inimum of fo
ted to basic s
ek block, the
eted a baselin
titions of eac
al of 40 repe
4:1 for each
MacRae, 20
hnical perform
ctors gave st
ubsequent b
tured by the
ure only the
s, a paper co
orded attemp
program de
ourteen train
skills and the
e students fi
ne performa
ch task. At ea
etitions per ta
training ses
006) To mini
mance, only
tandardized f
linded asses
laparoscopi
students’ ha
ontaining a u
pt to be tied t
signed to tea
ning sessions
e last two w
rst watched
ance test. Fol
ach subsequ
ask.
sion consiste
imize the im
y three differ
feedback aft
ssment. For t
ic camera. F
ands and sim
unique code w
to an individ
ach both ope
s scheduled 2
weeks to more
a detailed
llowing the b
uent session,
ent with
mpact of the
rent instructo
ter each repe
the laparosco
or the open t
mulated opera
was placed w
dual participa
105
en and
2-3
e
baseline
seven
ors
etition
opic
tasks, a
ative
within
ant and
5
Th
th
Su
Th
un
Th
si
ta
Pe
ho
cl
Pr
5
Fo
A
(M
in
th
si
Th
re
B
w
co
ra
w
T.3.4.4
he peg trans
he basic open
urgery (FLS
he HT was p
nderlying bo
he circle cut
imulated ope
asks were pe
erformance
ollow frame
lose the incis
ritts, & Harv
A.3.4.5
or the HT (b
A modified O
Martin et al.,
nstruments’ c
his task(Mart
imulated LC
he open task
earranged an
efore rating
were calibrate
oefficient of
ating period,
with ICCs ran
Technical
sfer (PT) was
n task. The p
S) manual.("F
performed us
oard.
t (CC) and in
en laparotom
rformed as o
Guidelines,"
with a 15cm
sion. This m
vey, 2015).
Assessmen
basic open ta
Objective Stru
, 1997). This
categories in
tin et al., 199
in 300 seco
k videos wer
nd assessed in
was begun,
ed to achieve
f >0.75 (Dow
the raters’ I
nging from 0
skills
s used as the
peg transfer w
FLS Manual
sing a silk ti
ntracorporea
my closure (L
outlined in th
" 2014) The
m incision do
model was mo
nt of techni
ask), students
ucture of Te
s modified sc
ncluded in th
97). During
onds. Each re
re assessed a
n a random o
these indivi
e excellent in
wning, 2004)
ICC was re-c
0.78 to 0.92 o
e basic laparo
was perform
l Skills Writt
ie suture pas
al knot tie (IK
LS) was used
he FLS manu
open LC wa
own the mid
odified from
ical skills
s were asked
echnical Skil
cale exclude
he original O
g the advance
epetition was
at the comple
order. A tota
duals were t
nter-rater rel
). To ensure
calculated ev
over the ratin
oscopic task
med as outlin
ten Instructio
sed through
KT) were us
d as the adva
ual.("FLS M
as simulated
ddle of the fe
m the ACS/A
d to tie as ma
lls (mOSATS
ed the ‘use o
OSATS asses
ed open task
s scored usin
etion of the s
al of three ex
trained on th
liability, def
that calibrat
very two wee
ng period.
k, and the on
ned in the Fu
ons and Perf
a Penrose d
sed as the adv
anced open t
Manual Skills
by fastening
elt. A 0-mono
APDS surgica
any square k
S) scale was
f assistance’
ssment tool,
k sessions, st
ng a mOSAT
study, to allo
xpert blinded
he mOSATS
fined as an in
tion was mai
eks. Excelle
ne-handed tie
undamentals
formance Gu
drain fastened
vanced lapar
task. The adv
s Written Ins
g a felt sheet
ofilament su
al skills curr
knots as poss
s used to sco
’ and ‘knowl
as they were
tudents were
TS.
ow for all the
d raters asse
instruments
ntraclass cor
intained ove
ent reliability
e (HT) was u
of Laparosc
uidelines," 2
d with Velcr
roscopic task
vanced lapar
structions an
t to an under
uture was use
riculum(Cofe
sible in one m
ore each repe
ledge of
e not applica
e asked to pe
e videos to b
ssed the ope
s and their ra
rrelation (IC
er the three-m
y was mainta
106
used as
copic
2014)
ro to an
ks, and a
roscopic
nd
rlying
ed to
er, Lee,
minute.
etition
able to
erform a
be
en tasks.
atings
C)
month
ained,
A
th
("
ch
th
do
5
St
pr
of
Th
M
tw
m
de
pr
th
re
N
m
pr
in
us
re
de
sc
All laparoscop
he FLS scori
"FLS Manua
heck the reco
hat the FLS s
one with the
P.3.4.6
tudents were
roficiency as
f these score
he scores pu
M. Ritter & S
wo standard
mean only. C
eviations bel
roficiency sc
he IKT. Thes
espectively, a
No previously
methodology
roficiency sc
nstitution per
sing the asso
epetitions we
eviations bel
coring metho
pic tasks we
ng system. T
al Skills Writ
orded time s
scoring of tim
e open tasks,
Performan
e trained to 4
s quickly as
es during sub
ublished by R
Scott, 2007).
deviations b
onsequently
low (slower
cores used w
se time score
as per the va
y published p
described by
cores for the
rformed each
ociated mOS
ere used to c
low (slower
odology, hig
ere scored in
Time and err
tten Instructi
scores, and to
me and error
was not nec
ce goals
40 repetition
possible. On
bsequent rep
Ritter et al. w
The publish
below (slowe
y, the cutoff f
than) the me
were: 54.8 se
es were then
alidated FLS
proficiency s
y Ritter et al
open tasks (
h basic and a
SATS instrum
calculate the
than) the me
gher scores re
real-time to
ror scores we
ions and Per
o fill in any u
r can be obje
cessary.
ns for each ba
nce proficien
etitions.
were used as
hed cutoffs fo
er than) the e
for the PT sc
ean, to keep
conds for th
n normalized
S scoring sys
scores were
l. for setting
(E. M. Ritter
advanced tas
ment by a sin
mean and st
ean set as th
epresented b
o remain con
ere collected
rformance G
unrecorded v
ectively mea
asic and adv
ncy was reac
s the proficie
for the CC an
expert mean,
core in the p
it consistent
e PT; 98.0 s
d for analysis
stem method
identified fo
FLS task pe
r & Scott, 20
sk five conse
ngle blinded
tandard devi
he proficiency
better perform
nsistent with
d for each ta
Guidelines," 2
values to shi
asured, blind
vanced task,
ched, empha
ency cut offs
nd IKT tasks
, while the c
resent study
t with the ot
seconds for t
s to 103.4, 72
dology (Vass
or the open t
erformance t
007). Two ex
ecutive time
d rater. The c
iation for eac
y score for e
mance for al
the validate
sk as outline
2014). The v
ield against
ded rating in
with the goa
asis was plac
s for the lapa
s were define
cutoff for PT
y was defined
ther FLS task
the CC, and
2.1, and 93.8
siliou et al., 2
tasks. Conse
targets was u
xpert surgeo
es. Each repe
completion t
ch task, with
each task (Ta
ll tasks.
ed methodolo
ed in the FLS
videos were
missing data
a random or
al of reachin
ced on maint
aroscopic tas
ed by Ritter
T was reporte
d as two stan
ks. The spec
113.0 secon
8 seconds
2006).
equently, the
used to estab
ons from the
etition was ev
times for all
h two standa
able 15). Pe
107
ogy of
S manual
used to
a. Given
rder, as
ng
tenance
sks (E.
et al. as
ed as the
ndard
cific
ds for
blish
study
valuated
10
rd
er the
108
Table 15: Proficiency scores for open tasks
One handed tie
mOSATS
Laparotomy closure
mOSATS
Repetition Expert 1 Expert 2 Expert 1 Expert 2
1 22 21 30 25
2 23 24 26 24
3 25 23 25 29
4 24 22 29 27
5 21 25 26 24
Mean 23 23 27.2 25.8
SD 1.6 1.6 2.2 2.2
Group mean 23 26.5
Group SD 1.5 2.2
Proficiency score 20.0 22.2
SD: standard deviation; mOSATS:modified objective structure of
technical skills; Proficiency score = Group mean - 2(SD)
5.3.5 Data analysis and statistical methods
A data mining technique, k-means clustering, was used to stratify the trainees into four categories. Data
mining is a computational process of finding patterns in various types of data using a combination of
methods from artificial intelligence and machine learning (Alpaydin, 2014). K-means clustering is a
technique used to partition n observations into k clusters, where observations within a cluster display
greater similarity to one another than to those in other clusters (Tan, Steinbach, & Kumar, 2006). The
surgical education literature has previously separated trainees into four performance categories therefore
k=4, was selected for this analysis (Grantcharov & Funch-Jensen, 2009; M. P. Schijven & Jakimowicz,
2004). While the number of clusters with this analytic approach is defined a priori, the number of
ob
cl
cl
m
pr
w
V
T
le
re
C
pe
ch
C
th
(t
ei
A
A
5
5
Si
du
m
bservations i
luster groupi
lusters, with
mean. This an
rocess befor
were conduct
Vienna Austr
o ensure the
earning curve
eached, or su
Chi-squared a
erformance c
haracteristics
Consistency o
he performan
o minimize
ight repetitio
All statistical
Armonk, NY)
Resu.4
.4.1 Pa
ixty-five me
ue to schedu
males and 36
in each clust
ings are dete
the goal of a
nalysis is ent
e finalizing p
ed using R s
ia).
e identified c
e was graphe
urpassed the
analysis was
clusters betw
s had a signi
of each parti
nce scores fo
skewing due
ons (highest
analyses we
).
ults
articipants
edical studen
uling conflict
females.
ter varies an
ermined thro
achieving th
tirely objecti
placement in
statistical sof
clusters accur
ed on a tradi
set proficien
then used to
ween tasks, a
ificant relatio
cipant’s perf
or their last 1
e to outlier v
score minus
ere performe
nts completed
ts. The mean
d is determin
ough a proce
he lowest sum
ive, and is ab
nto the appro
ftware (versi
rately repres
itional x and
ncy threshol
o determine
and to assess
onship with
formance at
10 repetition
values), and c
s lowest scor
ed using SPS
d the study.
n age of the p
ned by the re
ss of initial a
m of squares
ble to accoun
opriate clust
ion 3.2.2; R
sented techn
d y-axis to ev
d for each ta
whether the
s whether on
technical sk
the conclusi
ns for each ta
calculating t
re).
SS version 23
Sixty-six stu
participants
elationships
and updated
s of the Eucl
nt for variati
ter for that ta
Foundation
nical perform
valuate whet
ask.
same traine
ne or more de
kill.
ion of trainin
ask, excludin
the performa
3 (IBM SPS
udents were
was 24 year
between ob
d assignment
lidean distan
ion in each i
ask. K-mean
for Statistic
mance levels,
ther each gro
ees fell withi
emographic
ng was quan
ng the highes
ance range o
SS Statistics;
enrolled, bu
rs (SD = 2.2)
bservations. T
t of observat
nce from the
individual’s
ns clustering
cal Computin
, each cluster
oup approach
in the same
and backgro
ntified by exa
st and lowes
of the remain
IBM Corpo
ut one droppe
), and there w
109
The
ions to
cluster
learning
analyses
ng,
r’s mean
hed,
ound
amining
st score
ning
oration,
ed out
were 29
A.
B.
110
C.
Fi
pe
Pr
5
B
di
as
ac
2,
igure 11: Le
eg transfer, (
roficiency sc
.4.2 Le
ased on long
istinct cluste
ssignments w
cross differe
, moderate p
earning curve
(b) circle cut
cores are dem
earning cur
gitudinal per
ers, correspo
were determi
nt skills. For
performers as
es for cluster
t, and (c) int
marcated wi
rves patter
rformance ov
nding to fou
ined indepen
r each task, t
s cluster 3 an
rs 1 (top per
tracorporeal
th a dashed
rns for lapa
ver all repeti
ur learning cu
ndently for e
top performe
nd low perfo
rformers) to 4
knot tie task
line.
aroscopic a
itions, partic
urves, for ea
each task, wi
ers were lab
ormers as clu
4 (low perfo
ks, ordered fr
and open sk
cipants were
ach basic and
ithout consid
eled as clust
uster 4 (Figu
ormers) on th
from basic to
kills
effectively
d advanced t
deration of th
ter 1, high pe
ure 11 and F
he laparosco
o advanced.
separated in
task. Cluster
heir perform
erformers as
Figure 12).
111
opic (a)
nto four
r
mance
s cluster
A
B
Fi
ha
.
.
igure 12: Le
anded tie and
earning curve
d (b) laparot
es for cluster
tomy closure
rs 1 (high pe
e tasks, orde
erformers) to
red from bas
o 4 (low perf
sic to advanc
formers) on
ced.
the open (a)
112
) one
113
Learning curves for cluster 1 (22 to 35% of participants) were characterized by rapid achievement of
proficiency for all open and laparoscopic tasks, which was then reliably maintained on subsequent
repetitions. Learning curves for participants in cluster 2 (32 to 42% of participants) were characterized by
quick learning, eventually meeting the performance of their cluster 1 peers and reaching proficiency for
all tasks over the training period. Cluster 3 learning curves (25 to 37% of participants) were characterized
by slower learning, with proficiency attained for the open tasks but not for all the laparoscopic tasks.
Specifically, cluster 3 performance remained just below the proficiency threshold for both PT and IKT
tasks. Cluster 4 learning curves (8 to 15% of participants) were characterized by failure to reach
proficiency in four of five tasks, including all laparoscopic tasks (Table 16).
114
Table 16: Comparing clusters 1-4: start points, end points and repetitions to proficiency
Simulated surgical task Cluster 1 Cluster 2 Cluster 3 Cluster 4
Comparing clusters 1-4
Kruskal-Wallis H Test
p-value The clusters that are significantly
different
Laparoscopic tasks
Peg transfer Number of students 23 21 16 5
Start point score (points) mean (range)
75 (60-89) 46 (17-65) 29 (27-56) 17 (0-40) 49.12 *<0.001 all 4 clusters
End point score (points) mean (range)
108 (103-113) 105 (99-109) 95 (85-103) 90 (83-96) 47.78 *<0.001 all 4 clusters
Proficiency reached repetition (range)
16 (8-26) 26 (20-33) - -
Circle cut
Number of students 15 25 19 6 Start point score (points) mean (range)
54 (34-73) 43 (23-57) 22 (0-36) 25 (17-39) 44.09 *<0.001 cluster 1 and 2 cluster 2 and 3
End point score (points) mean (range)
91 (85-97) 88 (72-107) 82 (69-87) 71 (57-82) 33.82 *<0.001 all 4 clusters
Proficiency reached repetition (range)
5 (3-6) 8 (4-11) 16 (8-24) -
Intracorporeal knot tie
Number of students 14 27 14 10 Start point score (points) mean (range) 43 (11-70) 25 (1-60) 13 (0-36) 1 (0-2) 39.68
*<0.001 all 4 clusters
115
End point score (points) mean (range)
95 (87-102) 91 (77-100) 89 (64-100) 78 (55-93) 17.28 *0.001 all 4 clusters
Proficiency reached repetition (range)
29 (17-35) 35 (25-40) - -
Open technical skills
One handed tie (mOSATS)
Number of students 20 23 16 6 Start point score (points) mean (range) 9 (6-14) 8 (7-12) 7 (7-8) 7 (7-9) 7.09 0.069
none
End point score (points) mean (range)
23 (19-25) 22 (19-24) 20 (18-24) 17 (14-20) 28.56 *<0.001 cluster 2 and 3 clusters 3 and 4
Proficiency reached repetition (range)
17 (13-28) 24 (21-30) 34 (28-38) -
Laparotomy closure (mOSATS)
Number of students 14 26 20 5 Start point score (points) mean (range) 13 (9-21) 12 (9-17) 12 (8-17) 11 (8-14) 1.96 0.581
none
End point score (points) mean (range)
27 (21-30) 24 (19-29) 24 (18-29) 23 (20-27) 9.00 *0.029 cluster 1 and 2
Proficiency reached repetition (range)
16 (3-29) 27 (13-39) 26 (15-38) 36 (34-38)
mOSATS: modified Objective Structure of Technical Skills; *significance p<0.05; start point - average of first 3 repetitions; end point - average of last 3 repetitions.
116
Innate technical ability seemed to contribute significantly to laparoscopic performance, but not to
open skills. In all three laparoscopic tasks, students in higher clusters had significantly better
mean performance at the start of the learning curve as compared to peers in the next lower group
(mean initial performance for clusters 1 through 4 on the PT task of 75, 46, 29 and 4 points
respectively, p<0.001; on the CC task of 54, 43, 22 and 25 points respectively, p<0.001; on the
IKT task of 43, 25, 13 and 1 points respectively, p<0.001). The only exceptions were clusters 3
and 4 for the CC task, which were characterized by similar performance (Table 16). For the
open tasks, starting performance scores were similar across all clusters (mean initial performance
for clusters 1 through 4 on the HT task of 9, 8, 7 and 7 points respectively, p=0.069; on the LC
task of 13, 12, 12 and 11 points respectively, p=0.581).
The innate performance advantage for laparoscopic tasks seen in higher performing clusters
persisted over the duration of the study. The associated mean laparoscopic learning curves
remained distinctly higher in comparison to those of lower performing clusters throughout the
40 repetitions for all three tasks (Figure 11). This resulted in significantly different mean training
endpoints between clusters (mean final performance for clusters 1 through 4 on PT task of 108,
105, 95 and 90 points respectively, p<0.001; on CC task of: 91, 88, 82 and 71 points
respectively, p<0.001; on IKT task of 95, 91, 89 and 78 points respectively, p=0.001) (Table 16).
5.4.3 Associations between background characteristics and performance
clusters
Only two significant associations were identified between participant demographic
characteristics, non-surgical experiences, and performance clusters across all five technical tasks
(Table 17). Both significant associations were noted with respect to the peg transfer. Specifically,
males were more likely to be in higher performance clusters compared to females (cluster 1 had
65% males and 35% females as compared to 45% vs. 55% overall, p=0.039), and students who
played video games were more likely to be in a higher performance cluster compared to students
who did not play video games (cluster 1 had 73% video gamers and 27% non-gamers as
compared to 52% and 48% overall, p=0.021).
117
Table 17: Demographics and non-surgical experiences and their association with performance clusters
No. of participants
n=65
Medical school yearOne 49 (75)Two 16 (25)
Sex
Male 29 (45)Female 36 (55)
Handedness
Right 59 (92)Left 5 (8)
Yes 44 (69)No 20 (31)Missing 1
Yes 46 (71)No 19 (29)
Work well with my handsStrongly Agree 45 (70)Neutral 19 (30)Missing 1
Video Games
Yes 33 (52)No 31 (48)
Surgery primary interestYes 25 (38)No 40 (62)
Chi‐square Test was used throughout the table; *sign
Played a musical instrument
Involvement in team sports
Number of participants (percent) stratified by demographic or non‐surgical experiences
p‐value p‐value p‐value p‐value p‐value
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
17 (26) 17 (26) 13 (20) 2 (3) 11 (17) 19 (29) 15 (23) 4 (6) 9 (14) 23 (35) 9 (14) 8 (12) 13 (20) 17 (26) 15 (23) 4 (6) 12 (18) 17 (26) 15 (23) 5 (8)6 (9) 4 (6) 3 (5) 3 (5) 4 (6) 6 (9) 4 (6) 2 (3) 5 (8) 4 (6) 5 (8) 2 (3) 7 (11) 6 (9) 1 (2) 2 (3) 2 (3) 9 (14) 5 (8) 0 (0)
15 (23) 9 (14) 3 (5) 2 (3) 7 (11) 14 (22) 7 (11) 1 (2) 7 (11) 11 (17) 8 (12) 3 (5) 5 (8) 11 (17) 9 (14) 4 (6) 4 (6) 12 (18) 11 (17) 2 (3)8 (12) 12 (18) 13 (20) 3 (5) 8 (12) 11 (17) 12 (18) 5 (8) 7 (11) 16 (25) 6 (9) 7 (11) 15 (23) 12 (18) 7 (11) 2 (3) 10 (15) 14 (22) 9 (14) 3 (5)
20 (31) 20 (31) 16 (25) 4 (6) 14 (22) 23 (35) 17 (26) 6 (9) 12 (18) 26 (40) 13 (20) 9 (14) 18 (28) 21 (32) 16 (25) 5 (8) 14 (22) 23 (35) 19 (29) 4 (6)3 (5) 1 (2) 0 (0) 1 (2) 1 (2) 2 (3) 2 (3) 0 (0) 2 (3) 1 (2) 1 (2) 1 (2) 2 (3) 2 (3) 0 (0) 1 (2) 0 (0) 3 (5) 1 (2) 1 (2)
16 (25) 13 (20) 14 (22) 1 (2) 12 (19) 14 (22) 15 (23) 3 (5) 9 (14) 18 (28) 10 (16) 7 (11) 14 (22) 17 (27) 8 (13) 5 (8) 8 (13) 17 (27) 15 (23) 4 (6)6 (9) 8 (13) 2 (3) 4 (6) 3 (5) 10 (16) 4 (6) 3 (5) 5 (8) 9 (14) 3 (5) 3 (5) 6 (9) 5 (8) 8 (13) 1 (2) 6 (9) 8 (13) 5 (8) 1 (2)
16 (25) 15 (23) 11 (17) 4 (6) 11 (17) 17 (26) 13 (20) 5 (8) 9 (14) 17 (26) 13 (20) 7 (11) 11 (17) 20 (31) 11 (17) 4 (6) 9 (14) 20 (31) 16 (25) 1 (2)7 (11) 6 (9) 5 (8) 1 (2) 4 (6) 8 (12) 6 (9) 1 (2) 5 (8) 10 (15) 1 (2) 3 (5) 9 (14) 3 (5) 5 (8) 2 (3) 5 (8) 6 (9) 4 (6) 4 (6)
18 (28) 13 (20) 13 (20) 1 (2) 13 (20) 15 (23) 14 (22) 3 (5) 11 (17) 18 (28) 9 (14) 7 (11) 14 (22) 18 (28) 8 (13) 5 (8) 10 (16) 16 (25) 17 (27) 2 (3)5 (8) 7 (11) 3 (5) 4 (6) 1 (2) 10 (16) 5 (8) 3 (5) 2 (3) 9 (14) 5 (8) 3 (5) 5 (8) 5 (8) 8 (13) 1 (2) 3 (5) 10 (16) 3 (5) 3 (5)
16 (25) 8 (12) 5 (8) 4 (6) 9 (14) 12 (18) 10 (15) 3 (5) 6 (9) 14 (22) 8 (12) 5 (8) 9 (14) 11 (17) 7 (11) 6 (9) 4 (6) 11 (17) 14 (22) 4 (6)6 (9) 13 (20) 11 (17) 1 (2) 6 (9) 12 (18) 10 (15) 3 (5) 8 (12) 12 (18) 6 (9) 5 (8) 11 (17) 12 (18) 8 (12) 0 (0) 10 (15) 14 (22) 6 (9) 1 (2)
9 (14) 8 (12) 7 (11) 1 (2) 6 (9) 10 (15) 8 (12) 1 (2) 5 (8) 8 (12) 7 (11) 5 (8) 7 (11) 9 (14) 5 (8) 4 (6) 6 (9) 9 (14) 9 (14) 1 (2)14 (22) 13 (20) 9 (14) 4 (6) 9 (14) 15 (23) 11 (17) 5 (8) 9 (14) 19 (29) 7 (11) 5 (8) 13 (20) 14 (22) 11 (17) 2 (3) 8 (12) 17 (26) 11 (17) 4 (6)
nificant p‐value <0.05
0.822 0.717 0.516 0.480 0.715
0.047 0.113 0.639 0.217 0.139
0.021* 0.899 0.882 0.100 0.049
0.050
0.033 0.260 0.897 0.265 0.672
0.968 0.885 0.223 0.145
0.039* 0.296 0.556 0.153 0.494
0.310 0.864 0.672 0.533 0.395
0.279
Performance cluster Performance cluster Performance cluster Performance cluster Performance cluster
0.256 0.937 0.338 0.227
Laparoscopic Peg Transfer Task Laparoscopic Circle Cut Taks Laparoscopic Knot Tie Task Open One Handed Tie Task Open Laparotomy Closure Task
118
5.4.4 Individual performance differences between tasks
A significant association was seen between individuals’ cluster assignments across tasks (basic
vs advanced) within a given skill type (laparoscopic vs open), indicating that individuals were
likely to remain within the same or similar performance cluster across multiple skills of the same
type. However, this phenomenon was not demonstrated across the two skill types (e.g. low
performers in laparoscopic tasks were not necessarily low performers in open tasks) (Table 18).
Within a given skill type, individuals who were top performers (cluster 1) in the basic tasks
stayed in the upper tier (either cluster 1 or 2) for all basic and advanced tasks. Cluster 2
individuals maintained their grouping or moved up or down a cluster depending on the task.
However, poor performers, stayed in the lower tier (either cluster 3 or 4) for all tasks. Only one
individual did not follow this trend, and jumped from one extreme to the other (cluster 4 to
cluster 1) between tasks (Table 18).
119
Table 18: Consistency of performance clusters for laparoscopic and open technical skill
Participants that stayed within:
Clusters 1-2
Cluster 2-3
Clusters 3-4
p-value
Consistency of task clusters 1 - 4, between tasks
Laparoscopic peg transfer Laparoscopic circle cut 95% 66% 76% <0.001
Laparoscopic knot tie 71% 66% 38% 0.013
Laparoscopic circle cut Laparoscopic knot tie 63% 77% 42% 0.084
Open hand tie
Open Laparotomy closure
73% 74% 44% 0.008
Consistency of task clusters 1 - 4, across surgical techniques
Open skills Laparoscopic skills
Hand ties Peg transfer 60% 65% 19% 0.411 Circle cut 18% 64% 28% 0.694 Knot tie 62% 59% 24% 0.312
Laparotomy closure Peg transfer 61% 73% 38% 0.125 Circle cut 65% 70% 44% 0.622 Knot tie 63% 68% 42% 0.194
120
5.4.5 Concerning learning curve features for individuals unable to reach
proficiency in laparoscopic tasks
For laparoscopic tasks, cluster 3 individuals characterized by slower learning did not reach
proficiency in two of three laparoscopic tasks, while cluster 4 individuals did not reach
proficiency in any of the laparoscopic tasks. When comparing the two groups, the biggest
difference was performance stability between repetitions. Cluster 3 hovered below the
proficiency threshold but demonstrated an encouraging trend towards a plateau phase, indicative
of more stable performance (mean range in scores across last 10 repetitions PT=10.36; CC= 8.9
and IKT 13.8). In contrast, cluster 4 participants demonstrated highly variable scores between
repetitions, with no convincing trend towards a true plateau phase, indicative of more unstable
performance (mean range in scores of last 10 repetitions: PT=13.2; CC= 20.0 and IKT=23.77).
The wide variability in cluster 4 participants’ performance is a noteworthy and concerning
feature of their learning curves that may translate into a negative impact in the real operating
room. Figure 13 represents the IKT learning curve for representative individuals from clusters 3
and 4. In the last 10 repetitions per knot, the cluster 3 individual’s normalized score was between
103 and 154 points (equivalent to a performance time of 1.43 to 2.34 minutes), whereas the
cluster 4 individual scored between 174 and 456 points (performance time of 2.54 to 7.36
minutes). The performance times seen in cluster 4 (frequently >4 minutes per knot) are likely to
be disruptive to the progression and flow of a laparoscopic operation. Therefore, although
individuals in both clusters do not stabilize above the proficiency threshold, cluster 4’s
inconsistent non-proficient performance would likely be more problematic in the real operating
room.
Figur
and f
variab
5.5
This
to rea
Jakim
pract
both
done
surgi
advan
(basic
sugge
great
re 13: Learni
from cluster
bility in perf
Discuss
study was un
ach technica
mowicz, 2004
ice due to la
laparoscopic
in this study
cal training.
nced, laparo
c to advance
est that lapar
er barrier to
ing curves fo
4 (low perfo
formance tim
sion
ndertaken as
l proficiency
4). However
ack of suffici
c and open te
y, may be a m
Therefore, t
scopic and o
ed) but not ac
roscopic tech
reaching pro
or a represen
ormer) on the
mes and asso
s recent liter
y(Grantcharo
r, there has b
ient evidence
echniques of
more realisti
trainees wer
open tasks. P
cross surgica
hnical skills
oficiency as
ntative indiv
e laparoscop
ociated norm
ature sugges
ov & Funch-
been no impl
e on the lear
f variable di
ic reflection
e stratified i
Performance
al technique
were more c
compared to
vidual from c
pic intracorp
malized score
sts that not a
-Jensen, 200
lementation
rning pattern
fficulty. Ass
of the funda
nto perform
e clusters we
es (open vers
challenging
o open skills
cluster 3 (mo
oreal knot ti
es.
all surgical tr
09; M. P. Sch
of selection
ns of trainees
sessing a var
amental skill
mance cluster
ere consisten
sus laparosco
to acquire a
s. Low perfo
oderate perfo
ie, demonstr
rainees may
hijven &
n curricula in
s across a ran
riety of tasks
l sets require
rs for basic a
nt between sk
opic). Our re
and presented
orming traine
121
ormer)
rating
be able
n clinical
nge of
s as was
ed for
and
kill types
esults
d a
ees in
122
the laparoscopic tasks demonstrated weak innate ability with a sustained disadvantage
throughout their learning curves resulting in an inability to reach proficiency within the extensive
training window. Furthermore, laparoscopic low performers demonstrated marked
inconsistencies in their performance throughout the training period.
K-means clustering, which is a novel data mining technique within the surgical education
literature, but well established within the field of data science, was used in the present study
instead of previously described approaches that characterize individual learning curves such as
the learning curve – cumulative sum analysis (LC-CUSUM) or curve fitting (Biau, Williams,
Schlup, Nizard, & Porcher, 2008; M. P. Schijven & Jakimowicz, 2004). LC-CUSUM is
applicable to the analysis of dichotomous variables (Biau et al., 2008), and allows researchers to
identify trainees who reach proficiency with performance variability below a pre-defined
threshold. However, little additional information is gained to further characterize the
performance of trainees that do not reach proficiency. Curve fitting endeavors to address this
issue by classifying learners’ individual learning curves by fitting an exponential, linear,
logarithmic, or S curve to individual learners’ change in performance over time, allowing for
stratification based on similar curve types (M. P. Schijven & Jakimowicz, 2004). However,
because many trainees’ learning curves demonstrate considerable residual variability, the
discriminatory value of statistics used to quantify the goodness-of-fit of learning curves to
different curve types is poor (Feldman, Cao, Andalib, Fraser, & Fried, 2009; M. P. Schijven &
Jakimowicz, 2004). In contrast, k-means clustering allows the observations to be objectively
separated based on patterns within the data, rather than relying on force fitting of data to one of a
limited number of pre-defined mathematical curves. Furthermore, the analysis is not restricted to
a dichotomous interpretation (ie. did or did not meet proficiency), and detailed comparisons
between clusters can be performed to further quantify performance differences between the
groups.
Many surgeons acknowledge that some trainees have inherent talent and thus demonstrate quick
acquisition of technical skill (Anthony G. Gallagher et al., 2009). However, conceding that a
subset of trainees will likely never reach proficiency is contentious. Perhaps this controversy
exists because consistent inability to reach proficiency may only be true for a subset of skills (i.e.
laparoscopic and not open). On examining the learning curves of the different tasks, all students
reached proficiency for the advanced open task, albeit at different rates. In contrast, practice did
123
not close the performance gap between clusters for the laparoscopic tasks even over 40
repetitions. The current study purposefully lengthened the training period beyond the 10 to 30
repetitions previously reported in the literature (A. Alvand, S. Auplish, T. Khan, et al., 2011;
Grantcharov & Funch-Jensen, 2009; M. P. Schijven & Jakimowicz, 2004), in an attempt to
determine whether more practice would allow low performers to catch up. Despite doing so, the
performance clusters remained disparate throughout the learning curve for laparoscopic tasks.
Students with low innate ability remained at a continued disadvantage throughout the 40
repetitions. Furthermore, the proportion of low performers did not change substantially when
compared to previously reported studies, despite the added practice. For basic tasks, Grantcharov
et al. reported 8.1% of trainees to be low performers, compared to 7.6% in the current study
(Grantcharov & Funch-Jensen, 2009). For more difficult tasks, Schijven et al. reported a 20%
rate of low performers on virtual reality laparoscopic cholecystectomy,(M. P. Schijven &
Jakimowicz, 2004) compared to 15.4% for intracorporeal suturing in the current study. These
findings provide further evidence that low performers persist across laparoscopic tasks. They
exhibit low innate ability, with a lasting effect as they continue to perform below the proficiency
threshold despite lengthening the practice window.
The possibility of screening for individuals who are low performers in technical skills prior to
entry into surgical training is controversial. The results of the present study suggest that
screening may be of limited benefit for surgical specialties with predominately open operations,
given that all participants were able to reach proficiency on at least one of these tasks. However,
in specialties where an increasing number of laparoscopic procedures are becoming the standard
of care (Soper, Stockmann, Dunnegan, & Ashley, 1992; Taguchi et al., 2016), screening for
performance on these technical skills may be beneficial. In the present study, the large majority
of low performers continued to struggle throughout the curriculum across a large number of
repetitions and multiple laparoscopic tasks. Furthermore, the inconsistency in performance
between repetitions in this subgroup was substantial, despite completing each repetition in a
controlled simulated environment where the setup, equipment and simulated task were constant.
It is reasonable to believe that this poor and inconsistent performance would be exacerbated in
the real operating room, where patient factors (e.g. body habitus, tissue integrity, patient
anatomy), technical setup (e.g. trocar placement, positioning and camera view), and trainee
feedback may vary substantially. Thus, although controversial, it may be reasonable to consider
124
screening candidates entering surgical specialties with considerable laparoscopic procedure
volumes to identify those individuals who cluster into the lowest performance category. These
individuals are at high risk of ongoing variable performance below an acceptable level of
proficiency, which could have negative implications for both the training program and the
trainee.
Using surrogate markers instead of simulated tasks to screen for technical skill has also been
extensively investigated, but has proven to be largely unreliable (Louridas et al., 206). Currently
all students in North America, applying to surgical training, are required to submit a curriculum
vitae outlining their academic and non-academic achievements (Canadian Resident Matching
Service, 2014; NRMP, 2016). In the current study, very few of these background characteristics
demonstrated significant associations with either technical skill acquisition or the ability to reach
proficiency. Interestingly, students who reported that surgery was their first choice of future
career specialty had no added performance advantage as compared to their peers who expressed
a preference for non-surgical disciplines. Similarly, students who reported “working well with
their hands” had no advantage over their peers. This data adds to the already existing evidence
that self-perceived technical ability does not correlate with objective technical assessments
(Tangchitnob et al., 2011). Furthermore, it has been demonstrated that medical students
intending to pursue a surgical career have no greater innate motor dexterity when compared to
those with a preference for a non-surgical field (J.Y. Lee et al., 2012; Van Hove et al., 2008).
Consequently, surgical programs should not rely on student self-selection as a screening tool
(Panait et al., 2011). Of the other background data, playing video games was significantly
associated with technical skill (Schueneman et al., 1985; Van Hove et al., 2008). However, the
technical advantage associated with playing video games was lost after the basic laparoscopic PT
task, which is consistent with previous reports by Paschold et al.and Dimitriou et al. that
demonstrated that any advantage associated with video game playing was lost after a period of
practice (Dimitriou et al., 2009; Paschold et al., 2011). Therefore, although surrogate markers are
an attractive screening tool due to their feasibility in the current selection process, such markers
are not predictive of technical skill acquisition and thus should not be used in this manner.
We acknowledge several limitations to the present work. First, the study population consisted of
pre-clerkship medical students, not only medical students applying for surgical residency. This
population was chosen because within the Canadian medical system, only medical students in the
pre-c
requi
could
as the
of un
been
result
previ
Cons
four c
the sk
surpa
Given
assist
profic
cut of
partic
motiv
believ
judgm
than
profic
as pre
5.6
Train
skills
to rea
subst
linical years
irement give
d be trained s
ey progresse
ndocumented
possible if s
ts are both u
iously descri
equently, su
clusters iden
kill acquired
ass the skill (
n that these
t in career ad
ciency cutof
ff scores fro
cipants to ea
vation encou
ved that the
ment is deter
stage of train
ciency as de
e-clinical me
Conclu
nees can be s
s. Laparoscop
ach proficien
tantially larg
s of study are
n the purpos
solely in the
ed along thei
d practice in
studying fina
useful and rel
ibed, candida
urgical progr
ntified in the
d and reflecte
(and subsequ
selection cur
dvise prior to
ff scores deri
m Ritter et a
asily determi
uraging conti
ability to ac
rmined prim
ning. This ap
termined by
edical studen
usion
separated int
pic skills are
ncy in these,
ger role in lap
e reliably at
ses of the pre
simulation l
ir learning cu
the operatin
al year medic
levant to me
ates for surg
rams are like
present stud
ed in the pro
uent plateaus
rricula have
o applying fo
ived from ex
al. were used
ne their prog
inued partici
quire techni
marily by the
pproach is su
y scores deriv
nts.
to performan
e more diffic
despite con
paroscopy as
the start of t
esent work. I
laboratory, c
urves withou
ng room. Suc
cal students.
edical studen
gical training
ely interview
dy. Furtherm
oficiency pla
s) of medica
the ability to
or residency
xpert perform
d because the
gress toward
ipation. The
cal skills in
opportunity
upported by
ved from exp
nce grouping
cult to learn
ntinued pract
s compared t
their technic
In addition,
capturing ea
ut the risk of
ch controlled
. Neverthele
nts applying
g do not self-
wing and acce
more, since 4
ateaus by the
al students en
o stratify tra
y positions. S
mance, rathe
ey are report
d proficiency
e proficiency
a controlled
to practice w
the finding
pert perform
gs based on t
than open sk
tice. Innate t
to open skill
cal skill learn
pre-clerkshi
ach repetition
f confoundin
d circumstan
ess, the autho
for surgical
-select based
epting traine
40 repetitions
e study partic
ntering the tr
ainees, early
Second, the
er than more
ted as time s
y, which pro
y scores wer
d environmen
with adequa
that most pa
mance levels,
their ability t
kills, with so
technical ab
ls, and this a
ning curves,
ip medical st
n of the study
ng as a conse
nces would n
ors believe th
training. As
d on technica
ees represent
s per task we
cipants woul
raining prog
implementa
study relied
junior traine
scores, allow
ved to be a s
re not lowere
nt devoid of
ate feedback,
articipants re
, despite thei
to learn tech
ome trainees
ility plays a
advantage pe
125
a
tudents
y tasks
equence
not have
hese
s
al skill.
ting all
ere used,
ld likely
gram.
ation may
d on
ees. The
wing
source of
ed, as we
surgical
, rather
eached
ir status
hnical
s unable
ersists
126
for the duration of the learning curve. Low performing novice trainees generally remain in this
tier across technical tasks. Furthermore, some of these lower tier performers never reach the
proficiency threshold, and demonstrate marked variability in their learning curves with no trend
towards a plateau phase even after forty repetitions in a training environment with structured,
standardized feedback. Given the increasing use of minimally invasive and endo-luminal
techniques in surgical practice, screening potential candidates for surgical training using
simulated tasks to identify the lowest tier performers may benefit both the students and training
programs. Although the findings of the present study take the recent literature further, a
longitudinal follow up of the lowest tier performers within the clinical environment would
further validate these findings.
Cha6
in e
This
Grant
opera
6.1
Back
know
has b
script
crisis
Meth
ment
Partic
part o
Asses
and s
Techn
Resu
the m
impro
conve
apter 6: R
enhancin
chapter has
tcharov TP.
ating room: a
Abstrac
kground: Me
wn to enhanc
been limited
ts, and to ass
s scenario.
hods: Twent
al practice g
cipants’ skill
of a crisis sc
ssment of Te
subjective str
nical Skills f
ults: An impr
mental practic
oved their te
entionally tr
Random
ng advan
been publish
Mental prac
a randomize
ct
ental practic
e performan
to basic ope
sess their eff
y senior surg
groups, the la
ls were asse
enario in a s
echnical Ski
ress paramet
for Surgeons
rovement in
ce group com
echnical perf
rained partici
mized clin
nced lapa
hed as Louri
ctice to enhan
d controlled
ce, the cognit
nce in sports
rations. The
fect on advan
gical trainee
atter being tr
ssed while p
simulated op
ll (OSATS)
ters were me
s rating tool.
OSATS (P =
mpared with
formance dur
ipants deteri
nical tria
aroscopic
idas M, Bonr
nce advance
d trial. Br J S
tive rehearsa
and music. I
purpose of t
nced laparos
es were rando
rained by an
performing a
erating room
and bariatric
easured, as w
.
=0�003) and
h the convent
ring the cris
iorated. Men
al to eva
c surgica
rath EM, Sin
ed laparoscop
Surg. 2015 Ja
al of a task w
Investigation
this study w
scopic skills
omized to ei
expert perfo
a porcine lap
m, using the
c OSATS (B
well as non-t
d BOSATS (
tional trainin
is scenario,
ntal imagery
aluate me
al perfor
nclair DA, D
pic surgical
Jan;102(1):3
without phys
n of this tech
was to develo
and surgeon
ither conven
ormance psy
paroscopic je
Objective S
BOSATS) in
technical ski
(P =0�003) s
ng group. Se
whereas fou
ability impr
ental pra
rmance
Dedy NJ,
performanc
7-44.
sical movem
hnique in su
op mental pra
n stress leve
ntional trainin
ychologist.
ejunojejunos
tructured
nstruments. O
ills using the
scores was s
even of ten tr
ur of the ten
roved signifi
127
actice
e in the
ment, is
urgery
actice
ls in a
ng or
tomy as
Objective
e Non-
seen in
rainees
icantly
follow
differ
Conc
the si
despi
6.2
There
count
restri
care a
balan
to acc
trials
level
with
Dunn
skills
skill d
2012
speci
enhan
requi
pract
impro
and m
been
Mora
pract
wing mental
rences in obj
clusion: Men
imulated ope
ite added stre
Introdu
e has been a
tries in recen
ictions have
and decision
nce. This has
celerate tech
have demon
of technical
those with n
nington, 200
s warm-up be
during surge
; J. Y. Lee e
imens, with a
ncement stra
irements and
ice, defined
ove perform
mental prepa
demonstrate
an A., 1994;
ice may con
l practice tra
jective or su
ntal practice
erating room
ess.
uction
decrease in
nt years (Acc
been driven
n-making, as
s contributed
hnical skills t
nstrated that
l proficiency
no such train
8; Zevin B e
efore enterin
ery (Calatayu
t al., 2012).
associated re
ategy that im
d without add
as the cogni
ance in man
aredness on s
ed repeatedly
Paivio, 1985
nfer benefits
aining (P =0�
ubjective stre
improves te
m, and allows
the total num
creditation C
by societal
well as effo
d increasing
training and
technical sk
y and a short
ning (Palter &
et al., 2013; B
ng the operat
ud et al., 201
Both interve
esource requ
mproves oper
ditional dedi
itive rehears
ny fields. In s
stress levels
y(Burhans, R
5; Weinberg
for the surge
�011), but no
ess levels or
echnical perf
s trainees to
mber of hou
Council for G
concerns tha
orts to limit w
interest in tr
achievemen
kills pre-train
ened learnin
& Grantcharo
B. Zevin, Ag
ting room ha
10; Chen et a
entions requ
uirements, to
rative skill, w
icated infrast
al of a task w
sports psych
and perform
Richman, &
g RS., Seabo
eon in terms
ot in the con
non-technic
formance for
maintain or
urs available
Graduate Me
at fatigue ma
work hours t
raining strate
nt of compet
ning curricul
ng curve in th
ov, 2012; Sc
ggarwal, & G
as also been
al., 2013; Kr
uire simulatio
o run effectiv
with minima
tructure, wo
without phys
hology, posit
mance during
Bergey, 198
ume T., & J
s of increased
nventional gr
cal skills wer
r advanced l
improve the
for surgical
edical Educa
ay result in c
to acquire a
egies outside
tent perform
la result in tr
he operating
cott et al., 20
Grantcharov
shown obje
roft, Ordon,
on technolog
vely. An inex
al incrementa
ould therefor
sical movem
tive effects o
g high-level
88; Drisckell
Jackson A., 1
d technical p
roup (P =0�0
re evident.
laparoscopic
eir performa
training in m
ation, 2013).
compromise
healthier wo
e the operatin
ance. Rando
rainees with
g room comp
000; Scott &
v, 2012). Tec
ctively to im
Arthur, & P
gy and/or cad
xpensive sur
al resource
re be useful.
ment, is used
of mental pra
competition
l J., Cooper
1992). Menta
performance
128
083). No
c tasks in
ance
many
. These
d patient
ork–life
ng room
omized
h a higher
pared
&
chnical
mprove
Pittini,
daveric
rgical
Mental
to
actice
n have
C., &
al
e,
decre
studie
this a
Agga
pract
effect
determ
techn
6.3
The j
lapar
of the
with
repro
reliab
6.3.1
Interv
pract
achie
cues
techn
shoul
& T.,
opera
transc
eased stress l
es, limited to
approach as a
arwal, Sirima
ice script for
tiveness of m
mine whethe
nical skills in
Method
ejunojejuno
oscopic proc
e Roux-en-Y
the camera a
oducibly sim
ble and valid
1 Ment
views were c
ice script. In
eved. Intervie
(what you se
nique was ad
ld not be mis
, 2007; Holm
ation to ident
cribed, and i
levels, and im
o basic lapar
an adjunct to
anna, et al., 2
r the perform
mental practi
er it was ass
n a simulated
ds
stomy (JJ) p
cedure for tw
Y bypass is re
and retractio
ulated in a b
d tools.
tal practice
conducted w
nterviews con
ews were rec
ee) and kine
dopted from t
staken for so
mes & Collin
tify common
iterative con
mproved dec
roscopic ope
o technical s
2011). The a
mance of an
ice on advan
ociated with
d crisis scena
portion of a R
wo reasons. I
ecognized as
on is conside
box trainer u
e script dev
with subspeci
ntinued unti
corded and s
sthetic cues
the performa
olely listing t
ns, 2001). Su
n pitfalls exp
ntent analysis
cision-makin
erations perfo
urgical skill
aims of the p
advanced lap
nced laparos
h differences
ario.
Roux-en-Y g
Independent
s a senior-le
ered a junior-
sing porcine
velopment
ialty-trained
l theme satu
surgeons ask
(what you f
ance psycho
the operative
urgeons were
perienced by
s was used to
ng skills in t
formed by no
ls training (S
present study
aparoscopic p
copic techni
s in stress lev
gastric bypas
t completion
vel procedur
-level task. J
e bowel, and
d bariatric sur
uration of me
ked to descri
feel) at each
ology literatu
e steps (Call
e asked to re
y trainees. Th
o identify em
the operating
ovice surgeo
S. Arora et al
y were to dev
procedure, to
ical skill per
vels and imp
ss was select
n of the lapar
re for trainee
JJ can be rea
d can be asse
rgeons to de
ental practice
ibe, in the fir
step of a lap
ure on script
low & Hardy
eflect back to
he voice rec
merging them
g room. Only
ons, have eva
l., 2010; S. A
velop a men
o assess the
rformance, a
provement in
ted as the ad
roscopic JJ p
es, whereas
alistically an
essed using e
evelop the m
e cues had b
rst person, th
paroscopic JJ
developmen
y, 2004; Gre
o each step o
ordings wer
mes and crea
129
y a few
aluated
Arora,
ntal
and to
n non-
dvanced
portion
assisting
d
existing
mental
been
he visual
J. This
nt and
egg MJ.
of the
e
ate the
130
mental practice scripts. Pitfalls were added to the scripts if identified by more than one-half of
the expert surgeons.
6.3.2 Randomized trial
This study was a randomized single-blinded, two-armed trial conducted at a single large
academic institution in Canada. Before the start of the study the trial was registered through the
International Standard Randomized Controlled Trials Number (ISRCTN). Ethics approval was
granted by St Michael’s Hospital and the University of Toronto. Informed voluntary consent was
obtained from each study participant before randomization into either the mental practice or the
conventionally trained study arm using List Randomizer software (Randomness and Integrity
Services, Dublin, Ireland). No changes were made to the trial protocol during the course of the
study.
Postgraduate year 3 and 4 general surgery residents were eligible for recruitment to this study.
Participants who had completed fewer than five laparoscopic JJs in either the simulation
environment using a porcine bowel model in a laparoscopic box trainer, or in the operating room,
were excluded. Other exclusion criteria were: systemic illness affecting BP or heart rate (for
example, hypertension, diabetes mellitus or mood disorders), and use of prescription drugs that
modify cardiovascular response.
Previous work on the effects of mental practice on improving technical proficiency has shown an
average difference between trained and untrained groups of 5 points on the 35-point Objective
Structured Assessment of Technical Skill (OSATS) global rating scale(S. Arora, Aggarwal,
Sirimanna, et al., 2011). Analyses of these published data suggested a standard deviation for the
study sample of no more than 3�7. Using these findings, sample size calculation revealed the
need for at least nine individuals per group to detect a significant difference with an α of 0.05
and a power of 0.80.
All participants completed a demographic questionnaire at the start. Two objective validated
assessment tools were used to measure baseline mental imagery ability: the Mental Imagery
Questionnaire (MIQ) and the Movement Imagery Questionnaire Revised Second version (MIQ-
131
RS), which itself is broken down into visual imagery and kinesthetic imagery scores (S. Arora et
al., 2010; Gregg, Hall, & Butler, 2010). To assess baseline technical skill, each trainee performed
one laparoscopic JJ on a box trainer, using a porcine bowel model. The simulated laparoscopic JJ
was video recorded for subsequent blinded rating using the OSATS (Martin et al., 1997) and the
bariatric OSATS (BOSATS) scales (Zevin B et al., 2013). Finally, baseline stress levels were
measured objectively by heart rate and BP (using a non-invasive automatic BP machine), and
subjectively by the validated six-item State–Trait Anxiety Inventory (STAI) questionnaire for
adults (Marteau & Bekker, 1992).
All subjects participated in a didactic lecture on the creation of a laparoscopic JJ. The lecture
outlined the technical steps of the operation, with accompanying videos demonstrating the
correct operative technique for each step. Following the lecture, a multiple-choice test was
administered to ensure that all participants understood the steps of the operation. After the
session, the instructional videos used in the didactic session were available to all participants for
review at any time during the study. Residents in the intervention group additionally underwent
mental practice training, consisting of in-person instruction from an expert performance
psychologist and independent practice. Participants were taught first to perform a relaxation
exercise with abdominal breathing, and then to begin mental practice guided by the mental
practice scripts focusing on the kinaesthetic and visual cues required to ‘feel’ and ‘see’ each step
of the laparoscopic JJ. Subsequently, all participants in the intervention group were provided
with the written script, as well as a version of the technical skills videos from the didactic session
that included a voice-over of the mental practice scripts. Each participant had 7 days to perform
mental practice independently at home with the scripts and videos. To promote the use of mental
practice, each trainee participated in three voice-recorded follow-up telephone calls in which
they walked through the scripts verbally and received structured feedback from one investigator.
One week after baseline testing, both groups participated in a crisis scenario of a laparoscopic JJ
procedure in the simulated operating room. The same porcine bowel model in a laparoscopic box
trainer, as baseline, was used to allow blinding and direct comparative assessment before and
after the intervention.
To create a realistic, standard operative environment, the roles of the anaesthetist, scrub nurse,
circulating nurse and assistant surgeon were scripted and played by healthcare professionals who
132
were trained to react appropriately within the script guidelines. As the trainee performed the
procedure, at a set operative step, the simulated patient unexpectedly had an anaphylactic
reaction to the routinely administered preoperative antibiotics. The crisis was introduced
purposefully when the gastrointestinal tract was interrupted, driving the resident to manage the
crisis then finish the operation rather than abort the procedure. Immediately following
participation in the crisis scenario, participants were asked individually to keep all information
concerning the session confidential, and all agreed. Three different crisis scenarios were prepared
in case of a breach of confidentiality. Only a single scenario was used, however, as participants
continued to be surprised and stressed as measured by objective and subjective parameters, with
no indication of any breach of confidentiality.
To ascertain objective performance, the video recordings of the laparoscopic JJs were assessed
using both OSATS and BOSATS scales. All laparoscopic-view videos (baseline and simulation
crisis JJ) were edited to start when the laparoscopic graspers entered the box trainer and to end
after closing the common enterotomy. The pause reflecting the crisis period was removed from
the laparoscopic video before review in random order to ensure that the raters were blinded to
both the participant and study stage (baseline versus after intervention).
Stress levels were assessed using both objective and subjective instruments. During the
simulation, BP was taken at five fixed points in the operation using an automatic BP cuff, and
heart rate was recorded at 1-s intervals, using a non-invasive chest strap monitor (Polar Electro,
Kempele, Finland). Participants wore monitors for the whole of the scenario, allowing parameter
measurement with minimal interruption to the surgical task being performed. Subjective stress
was scored by each participant using a validated six-item version of the Spielberger
STAI(Marteau & Bekker, 1992), which was administered both before and after the simulation.
The full simulated surgical scenario was video recorded for each participant to assess non-
technical skills. Two trained raters used the Non-Technical Skills for Surgeons (NOTSS) scoring
system to evaluate the trainees(Yule et al., 2008). Raters were non-surgeons who had received
dedicated training from a surgeon researcher experienced in the assessment of surgeons’ non-
technical skills within the operating room. This training included instruction and training on
roles, responsibilities and behaviours within the operating room, multiple case-based training
sessions, practice ratings using intraoperative videos and rater calibration by comparing scores.
To en
rando
progr
assoc
To as
comp
surgi
6.3.3
Descr
to com
(Wilc
score
tests
6.4
6.4.1
Satur
opera
tactil
bowe
nsure objecti
omization of
ramme, no p
ciated with th
ssess particip
pleted a shor
cal training.
3 Statis
riptive statis
mpare techn
coxon rank s
es, change sc
(independen
Results
1 Ment
ration of them
ative substep
e cues. For e
el 50 cm from
Visual: ‘I
sight. I do
grasped. T
incorrectl
ive assessme
f the particip
personal or p
he developm
pant satisfac
rt questionna
stical analy
stics were ca
nical, non-tec
sum test and
cores (baselin
nt and paired
s
tal practice
mes was rea
ps were ident
example, a p
m the ligame
grasp the bo
on’t want to
These are vis
ly’.
ent, raters we
pants, had no
rofessional r
ment or imple
tion with the
aire to gauge
ysis
alculated for
chnical skills
Mann–Whi
ne score – fi
d t tests) wer
e scripts
ched after ei
tified for the
portion of the
ent of Treitz
owel whilst
see any blot
sual signs of
ere blinded t
o association
relationship
ementation o
e use of men
e whether it w
all variables
s and STAI s
tney U test).
inal score) w
e used to ev
ight bariatric
e laparoscop
e script of st
’, read:
I see it; I nev
tchy haemato
f trauma: wh
to the purpo
n with the gen
with the stu
of this resear
ntal practice,
was consider
s. Non-param
scores withi
. To account
were used to
aluate differ
c surgeons h
ic JJ, with e
tep 1 ‘runnin
ver grab the
omas, or exc
hich means, I
se of the stu
neral surgery
udy participa
rch.
, all interven
red a valuab
metric statist
in and betwe
t for the diff
compare the
rences in hea
had been inte
ach broken d
ng the proxim
bowel with
cessive blanc
I’ve grasped
udy and grou
y training
ants, and wer
ntion group r
ble adjunct to
tical tests we
een groups
ference in ba
e cohorts. Pa
art rate and B
erviewed. Six
down into vi
mal limb of s
the graspers
ching where
d too hard or
133
up
re not
residents
o
ere used
aseline
arametric
BP.
x
isual and
small
s out of
e I’ve
r
134
Kinaesthetic: ‘I try to feel where the bowel wants to go. I don’t push the bowel or direct it
to an area where I can feel tension. I run the bowel with zero tension in a clockwise
fashion. If the bowel doesn’t want to come, something’s wrong. It’s probably stuck
somewhere or I haven’t got a good grab’.
A minimum of two to a maximum of five pitfalls was identified for each step of the operation.
An example of a pitfall repeatedly identified by experts during step 1 was ‘working off screen’:
Working off screen: ‘Trainees may grab the bowel when it’s out of sight and therefore
they cannot see if they are applying too much force or have caused injury. This can result
in unrecognized bowel injury’.
Figur
6.4.2
The p
chara
the in
re 14: CONS
2 Rand
progress of p
acteristics of
ntervention g
SORT diagra
domized tri
participants t
f the groups w
group had pe
am illustratin
ial
through the p
were similar
erformed mo
ng progress t
phases of the
r, but laparos
ore laparosco
through the
e study is sh
scopic exper
opic procedu
phases of th
hown in Figu
rience was n
ures as the pr
he study
ure 14. Demo
not the same
rimary surge
135
ographic
because
eon and
136
more laparoscopic JJs in the operating room and simulation centre (Table 19). The OSATS and
BOSATS scores at baseline were higher in the intervention group than those in the control group.
Change scores (final score – baseline score) were therefore used to compare the cohorts. At
baseline, all remaining measurements were equal between groups including mental imagery
ability, and objective and subjective stress parameters (Table 20).
Table 19: Demographics of study participants
Conventional training
(n = 10) Mental practice
(n = 10) Postgraduate year 3 4
5 5
6 4
Sex ratio (M : F) 9 : 1 7 : 3 Handedness Right Left
10 0
10 0
Completed FLS training Yes No
4 6
3 7
No. of junior-level laparoscopic procedures
< 10 > 10
2 8
2 8
No. of procedures as primary surgeon < 10 7 5 > 10 3 5 No. of laparoscopic JJs in OR and simulation centre
< 10 8 5 > 10 2 5
FLS, Fundamentals of Laparoscopic Surgery; JJ, jejunojenustomy; OR, operating room.
Table 20: Result of baseline assessments of technical skill and mental rotation ability
Conventional training Mental practice P*
Technical skill score (points)
OSATS 18 (17, 21) 22 (21, 27) 0.043
BOSATS 21 (20, 32) 28 (26, 35) 0.123
Mental rotation ability score (points)
MIQ 32 (27, 39) 33 (26, 39) 0.912
137
RVMIQ 43 (41, 45) 46 (35, 49) 0.684
RKMIQ
40 (36, 44) 42 (33, 48) 0.971
Values are median (i.q.r.). OSATS, Objective Structured Assessment of Technical Skill; BOSATS, bariatric
Objective Structured Assessment of Technical Skill; MIQ, Mental Imagery Questionnaire; RVMIQ, Revised
Vividness Mental Imagery Questionnaire; RKMIQ: Revised Kinaesthetic Mental Imagery Questionnaire. *Mann–
Whitney U test.
Greater improvements in technical skills were seen among patients who underwent mental
practice training, with significantly higher OSATS and BOSATS change scores than among
those who received conventional training. Seven of ten participants in the intervention group
improved their OSATS score by at least 5 points on the final crisis scenario, and the remaining
three improved by 0–4 points. None of these participants deteriorated in skill when placed in the
crisis environment. In contrast, only two of the ten conventionally trained residents improved
their OSATS score by 5 points or more, four improved by 0–4 points and four deteriorated in
skill when tested in the crisis environment (Figure 15 and Figure 16). Those in the intervention
group had significantly greater median absolute OSATS and BOSATS scores on final
assessment than at baseline. These changes were not seen in the conventional group (Table 21).
138
Figure 15: Comparison of Objective Structured Assessment of Technical Skill (OSATS) change
scores between groups. Median (line within box), interquartile range (box), and range (error
bars) excluding outliers (circles) are shown. Dotted line indicates baseline performance. P
=0�003 (Mann–Whitney U test)
139
Figure 16: Comparison of bariatric Objective Structured Assessment of Technical Skill
(BOSATS) change scores between groups. Median (line within box), interquartile range (box),
and range (error bars) excluding outliers (circles) are shown. Dotted line indicates baseline
performance. P =0�003 (Mann–Whitney U test)
mental practicegroup
conventionally trainedgroup
BO
SA
TS
ch
an
ge
sco
re15
10
5
0
-5
-10
baselineperformance
p = 0.003
140
Table 21: Technical skill results at baseline and following training
Baseline Crisis scenario P
Conventional training
OSATS score (points) 18 (17, 21) 19 (16, 25) 0.734
BOSATS score (points) 21 (20, 32) 24 (20, 30) 0.622
Mental practice
OSATS score (points) 22 (21, 27) 30 (25, 36) 0.005
BOSATS score (points) 28 (26, 35) 37 (35, 40) 0.005
Values are median (i.q.r.). OSATS, Objective Structured Assessment of Technical Skill; BOSATS, bariatric
Objective Structured Assessment of Technical Skill. *Wilcoxon signed-rank test.
Residents in the mental practice group demonstrated a significant improvement in median MIQ
scores following training. No significant difference in median scores was identified in the
conventional training group. No significant differences in mental imagery abilities were
identified for either study arm, using the MIQ-RS (Table 22). Systolic BP, diastolic BP and heart
rate increased significantly during the crisis scenario in all participants compared with baseline:
mean (s.d.) increase 18(25) mmHg (P =0�004), 20(13) mmHg (P <0�001) and 14(14) b.p.m. (P
<0�001) respectively. During the crisis moment the intervention and control groups experienced
equally high systolic BP (139(19) and 137(17) mmHg; P =0�830) and diastolic BP (94(17)
versus 94(10) mmHg; P =1�000). The groups reported feeling equally stressed during the crisis
scenario as measured using the STAI score: median 11 (i.q.r. 10–14) versus 12 (11–13)
respectively (P =0�853).
There was no significant difference in non-technical performance when mean values of the four
categories of NOTSS were compared between groups (P =0�853). Inter-rater agreement was
excellent, with an intraclass correlation coefficient of 0�80.
Eight
the tr
opera
Table
Conv
M
R
R
Ment
M
R
R
ValuImagsigne
6.5
Ment
impro
Some
skills
Ment
perfo
Decre
surge
2009
t of ten train
raining curric
ations in the
e 22: Mental
ventional train
MIQ score (poin
RVMIQ score (p
RKMIQ score (p
tal practice
MIQ score (poin
RVMIQ score (p
RKMIQ score (p
es are median gery Questionned-rank test
Discuss
tal practice w
oved mental
ewhat unexp
s.
tal practice n
ormance also
eased perfor
e of cortisol,
; Kirschbaum
ees in the in
culum and s
future.
l imagery ab
ning
nts)
points)
points)
nts)
points)
points)
(i.q.r.). MIQ, Maire; RKMIQ,
sion
with scripts a
imagery abi
pectedly, men
not only imp
o improved, d
rmance follo
with an incr
m, Wolf, Ma
ntervention g
even of ten b
ility at basel
Mental ImageryRevised Kinae
and accompa
ility as well
ntal practice
roved advan
despite the a
wing stressf
rease in hear
ay, Wippich,
group felt tha
believed the
line and follo
Baseline
32 (27, 39)
43 (41, 45)
40 (36, 44)
33 (26, 39)
46 (35, 49)
42 (33, 48)
y Questionnairesthetic Mental
anying voice
as advanced
e had no effe
nced laparosc
added stresse
ful situations
rt rate and B
, & Hellham
at mental pra
y would use
owing traini
Crisis s
42 (2
43 (3
39 (3
41 (3
43 (4
44 (3
re; RVMIQ, Rel Imagery Que
e-over instru
d laparoscop
ect on measu
copic perfor
es of the une
s secondary t
P, is well re
mmer, 1996).
actice should
e this techniq
ing
scenario
27, 45)
9, 49)
6, 47)
1, 43)
40, 47)
5, 47)
evised Vividneestionnaire. **W
uctional vide
pic technical
ured stress le
rmance but t
expected intr
to an involu
cognized (S
None of the
d be incorpo
que to prepar
P
0.083
0.270
0.670
0.011
0.757
0.234
ess Mental Wilcoxon
eos significan
skill.
evel or non-t
echnical
raoperative c
untary physio
onal Arora e
e trainees wh
141
rated in
re for
ntly
echnical
crisis.
ological
et al.,
ho
142
received mental practice exhibited deteriorating performance scores from baseline during the
crisis, whereas four of the ten conventionally trained had worse performance. This finding is
important for surgical trainees who work routinely under pressure, and undoubtedly will be
required to perform under stress during their surgical careers. Previous studies (S. Arora,
Aggarwal, Sirimanna, et al., 2011; Geoffrion et al., 2012) have tested participants directly after
in-person mental practice coaching. The present study demonstrated the effectiveness of this
technique with delayed testing after mental practice at home, which may be a more realistic
approach to implementation of mental practice into residency training.
The effectiveness of mental practice in improving performance is dependent on achieving
improvements in mental imagery ability(Sevdalis, Moran, & Arora, 2013). Although it is
difficult to quantify and measure participants’ use of mental practice techniques directly, tools to
assess mental imagery ability, such as the MIQ, may be able to identify improvements associated
with their use. In the present study, participants demonstrated a significant improvement in MIQ
scores following mental practice training but not after conventional training. Surprisingly, stress
levels and non-technical skills were similar in the two groups. This was in contrast to one other
study (S. Arora, Aggarwal, Moran, et al., 2011) in which mental practice significantly decreased
stress scores measured by heart rate, STAI and urinary cortisol in novice surgeons performing a
simulated laparoscopic cholecystectomy. Reasons for similar responses between groups in the
present study may be related to the mental practice scripts being highly focused on kinaesthetic
and visual imagery for technical skill, rather than motivational enhancement imagery that
focuses on optimizing performance during times of increased pressure or stress. Although the
intervention group was able to outperform the conventional trainees in the presence of significant
stress, similarities between the groups with respect to non-technical skills suggest that altering
behaviours such as communication and team management require dedicated non-technical skills
training30, rather than an indirect intervention such as mental practice or progression of
increased experience during surgical training (Alvand et al., 2012; Flin et al., 2007).
This study has a number of limitations. Baseline OSATS scores were significantly different
between the groups. This difference was probably explained by the study participants’ variable
exposure to minimally invasive surgery rotations before the start of the study. The use of change
scores in the analysis should, however, have overcome most of this effect. Non-technical skills
raters were not surgeons, which may have impeded their ability to discern differences in non-
143
technical skills associated with crisis management. These individuals had, however, received
extensive instruction and training in non-technical skills assessment before the start of the study,
minimizing the impact of their lack of direct surgical experience. It is possible that there might
have been differences in the relative contribution of the different components of the mental
practice intervention (in-person training, independent practice with written script, review of
performance videos with audio or written script) to the observed differences in technical
performance, but the study design did not allow assessment of these potential differences.
Cortisol levels were not used to quantify stress because the majority of trainees had clinical
duties to attend to, and it was not feasible for them to participate in the study while avoiding the
many confounders of cortisol levels (Kelly, Young, Sweeting, Fischer, & West, 2008).
Mental practice as an adjunct for training of advanced laparoscopic skills appears to be an
effective method of improving performance in surgical trainees, with the added stressor of a
crisis scenario in the simulated operating environment. These findings suggest that it may be a
promising adjunct to surgical training programmes.
Cha7
7.1
Work
invas
reach
There
into t
thesis
select
mark
have
comp
perfo
There
determ
reside
incor
lapar
techn
rates
any o
rando
impro
a sim
no di
stress
apter 7: G
Thesis
k hour restric
sive surgical
h technical co
efore, optim
training may
s was to inve
tion of incom
kers that may
the ability to
pleted an orig
ormance of in
efore, my se
mine which
ents. Both op
rporated thes
oscopic task
nical skills on
and a small
of the laparo
omized contr
ove technica
mulated crisis
fference was
s levels.
General
summary
ctions, string
techniques i
ompetence(L
izing the sel
y help ensure
estigate diffe
ming surgica
y predict tech
o predict lap
ginal study t
ncoming sur
cond origina
simulated te
pen and lapa
se recommen
ks over a one
n low fidelit
subset (8-15
scopic tasks
rolled trial w
al performan
s scenario. T
s demonstrat
discussi
y
gent patient s
into training
Levine & Sp
lection of sur
e technical co
erent method
al trainees by
hnical skill.
paroscopic pe
to assess wh
rgical trainee
al study was
echnical skil
aroscopic sk
ndations and
e month train
ty simulation
5%) were low
. Lastly, to e
was complete
nce of senior
The MP grou
ted between
ion
safety precau
g have made
pang, 2014; W
rgical reside
ompetence i
ds of screeni
y first compl
This review
erformance i
ether visual
es, however
a national D
lls would be
ills met expe
d assessed m
ning curricul
n models. Stu
w performer
explore in-tra
ed to assess w
surgical res
up outperform
the groups f
autions, and t
it increasing
Watson, Ma
ents and inco
is consistentl
ing for techn
leting a syst
demonstrate
in the novice
spatial tests
no reliable a
Delphi quest
most approp
ert consensu
edical studen
lum starting
udents acqui
rs who were
aining adjun
whether a co
idents durin
med the cont
for non-tech
the incorpor
gly difficult
athew, & Wil
orporating pe
ly met. Thus
nical ability a
ematic revie
ed that visua
e trainee. T
s predict lapa
association w
tionnaire of C
priate to scre
us. My third
nts’ learning
with basic t
ired technica
unable to re
ncts for techn
ognitive MP
ng advanced
trol group fo
hnical perform
ation of min
for residents
lliams, 1995
erformance a
s the focus o
at the time o
ew on surrog
al spatial test
o this end, I
aroscopic
was found.
Canadian GS
een incomin
original stud
g curves of o
then advance
al skills at va
each proficie
nical perform
P could be us
laparoscopy
or technical s
mance or me
144
nimally
s to
5).
adjuncts
of this
of
gate
ts may
then
S PDs to
ng
dy
open and
ed
ariable
ency in
mance, a
sed to
y, during
skill but
easured
Over
demo
acqui
Furth
perfo
surgi
7.2
At th
7.2.1
In the
predi
inves
proce
stand
backg
existi
or co
minim
A sys
surro
exper
found
perfo
2005
Tang
probl
all the work
onstrated tha
isition may b
hermore, MP
orming advan
cal training p
Finding
e time of sel
1 To ev
exper
skills
e context of
ict technical
stigated first
ess. As outlin
dardized curr
ground chara
ing written a
mputer-base
mal addition
stematic revi
gate makers
rience, time
d to have a h
ormance (Dim
; Nomura et
gchitnob et al
lematic scree
in this thesi
at a simulatio
be an effectiv
P is an affect
nced laparos
programs.
gs with re
lection
valuate wh
riences and
s in the nov
finding a scr
performance
because bot
ned in sectio
riculum vitae
acteristics w
application. I
ed and could
nal resources
iew was perf
s and their ab
spent in the
higher numbe
mitriou et al
al., 2008; P
l., 2011; Van
ening questio
is, including
on surgical s
ve method f
tive adjunct t
scopy and th
espect to t
hether prev
d 2D-3D v
vice surgic
reening test
e, surrogate
th these com
on 2.1, the se
e, followed b
were able to p
In addition, t
d also be add
s.
formed to as
bility to pred
real operatin
er of studies
., 2009; Hisl
aschold et al
n Hove et al
on because t
one systema
kills curricu
for screening
to improve t
erefore shou
the aims o
vious surgic
visual spati
cal trainee
for incoming
markers and
mponents cou
election proc
by participat
predict techn
the visual sp
ded to the exi
ssess the exis
dict technica
ng room and
s demonstrat
lop et al., 20
l., 2011; D.
., 2008). Ho
the reported
atic review a
ulum assessin
g technical ab
technical per
uld be consid
of the the
cal experie
al tests cor
g surgical tr
d background
uld easily be
cess already
tion in an in-
nical skill, th
patial tests st
isting in-per
sting eviden
al skill (sectio
d visual spati
ting a positiv
006; Kolozsv
Stefanidis, K
wever, video
gaming adv
and four orig
ng progressi
bility of inco
rformance in
dered for inc
esis
ences, non-
rrelate with
rainees that h
d characteris
added to the
requires tha
-person inter
hey could be
tudied were
rson intervie
nce supportin
on 2.3.2). Vi
ial tests wer
ve associatio
vari et al., 20
Korndorffer
o game expe
vantage is lik
ginal manusc
ion of skill
oming traine
n senior resid
corporation i
-surgical
h laparosco
has the abilit
stics were
e existing se
at applicants
rview. There
added to the
either paper
w process w
ng the use of
ideo game
e the only m
on with techn
011; Madan
Jr, et al., 20
erience is a
kely due to th
145
cripts,
ees.
dents
into
opic
ty to
election
submit a
efore, if
e
r-based
with
f
markers
nical
et al.,
06;
he learnt
146
process of responding to visual cues from a 2D screen which can then be transferred to
laparoscopy (Dimitriou et al., 2009; Paschold et al., 2011). Thus video game experience likely
does not evaluate technical aptitude but rather a learnt skill. Similarly, time spent in the real
operating room is not a fair selection question, because it does not address technical aptitude but
again, learnt skill. Furthermore, time spent in the operating room will differ between applicants
due to their medical school program structure, the number of weeks of surgery they are exposed
to and the number of learners in the operating room. Although these external factors directly
influence this screening question it will have no bearing on true technical ability. Therefore,
these two background factors are not suitable for screening incoming trainees.
However, finding a background characteristic or experience that may be associated with
technical performance would be advantageous to the existing selection process due to ease of
implementation. Therefore, background experiences were explored in both original studies of
incoming surgical trainees and medical students in this thesis (chapter 3 and 5). The background
experiences were expanded to include involvement in sports, music, gaming and characteristics
such as gender, handedness etc. but unfortunately none of these demonstrated a consistent
positive association with technical performance. Despite my persistent effort to search for a
background characteristic or experience to incorporate into selection to screen for technical
aptitude, none reliably demonstrated a positive association with technical ability. Instead,
background experiences may be helpful to determine other desirable attributes for incoming
trainees (e.g. ability to work in a team, work ethic, interest in research) and further study in this
area may prove fruitful.
Twenty-five visual spatial tests were identified in the literature search but only three were
thought to have some evidence of predicating laparoscopic performance: PicSOR, cube
comparison tests and card rotation test. Of the three, PicSOR demonstrated the most consistently
positive association with simulated laparoscopic skills in a box trainer and virtual reality
simulator (Buckley et al., 2013; Buckley et al., 2014; A. G. Gallagher et al., 2003; Kolozsvari et
al., 2011; McClusky et al., 2005). However, often these associations were of subcomponents of
the scoring metrics, rather than overall performance. Scores on the cube comparison and card
rotation tests had equal numbers of studies that demonstrated positive and negative associations
with laparoscopic performance, therefore it was still unknown whether these tests would be
useful for the purpose of selection(Buckley et al., 2013; Buckley et al., 2014; L. Enochsson et al.,
147
2006; Groenier et al., 2014; Kolozsvari et al., 2011; McClusky et al., 2005; Nugent et al., 2012;
Dimitrios Stefanidis et al., 2007). To further understand this relationship, these three visual
spatial tests were selected and compared to laparoscopic camera navigation on the VR simulator
and laparoscopic circle cut on the box trainer.
Of the three visual spatial tests listed above, only the cube comparison test demonstrated a
positive association with overall technical performance on the VR LCN task. No other positive
associations were noted. Although disappointing that these tests were not able to screen for
technical ability of novice trainees, it speaks to the complexity of predicting technical aptitude.
Visual spatial tests may play a role in laparoscopy, however in isolation it does not explain the
reason for variable performance between trainees. Technical performance is likely a multifaceted
process involving many interacting abilities, which to date are not fully understood.
7.2.2 To solicit program directors’ opinions on the proportion of trainees who
do not achieve the minimum technical standards expected at the time of
graduation
To assess whether there is a potential for improving the selection process by screening for future
technical ability, current PDs of Canadian GS training programs were surveyed. Specifically,
they were asked whether they felt some contemporary GS trainees are unable to reach technical
competence despite completing a full post-graduate GS training program. The majority of PDs
reported that some trainees do not achieve the minimum technical standard at the time of
graduation, with the stated proportion ranging from 5-15%. Although no historical Canadian data
are available for comparison, this number is consistent with previously published findings from a
number of European centers which reported that 5-20% of their participants had difficulty
reaching technical competence (A. Alvand, S. Auplish, T. Khan, et al., 2011; Cushchieri, 2003;
Grantcharov & Funch-Jensen, 2009; M. P. Schijven & Jakimowicz, 2004).
In 2015, 87 trainees entered GS training across Canada. Over the past 5 years, the number of
available national training positions in GS has ranged from 86-95. Consequently, if the 2015
numbers are extrapolated to reflect 10-15% of trainees, nationally 4-14 incoming residents per
year are at risk of not reaching technical competence. These numbers represent a proportion of
trainees that have successfully gone through the full selection process. They have worked with a
148
number of GS staff, obtained reference letters from surgeons in the field, received multiple
evaluations and have interviewed to be matched into competitive positions. This information
further supports that technical aptitude screening is an area within the selection process that has
room for improvement.
7.2.3 To establish a national consensus on the desired attributes of GS
candidates, and the technical skills that would be most indicative of
future performance
Clear consensus was met amongst the PDs on a number of non-technical attributes that are
desirable for a GS trainee and were prioritized above technical skill. The attributes that received
the highest level of agreement included: work ethic, passion for GS, professionalism, ability to
work in a team, sound judgment, ability to make decisions and the ability to assimilate
information to formulate an opinion. Therefore, it was clear from the PDs that although technical
ability is required for surgical training its importance was second to the non-technical skills listed
above. GS is a team based interdisciplinary surgical field where nurses, allied health members
(e.g. occupational therapist, physiotherapist, wound care specialists and dieticians), patients and
their families are imperative to each patient’s peri-operative care. Therefore, it is understandable
that non-technical skills were emphasized by the PDs as most important. Selecting residents who
will excel in these qualities is of utmost importance and future research is needed to identify
screening tools that predict these attributes.
However, even though technical skill was not considered the most important attribute, it also
reached positive expert consensus. Furthermore, consensus was reached on two basic open tasks
(one-handed tie and subcuticular suture) and two laparoscopic tasks (coordination, grasping and
cutting). The tasks that reached consensus were all basic skills that are required across all GS
disciplines, whether applying to a rural, community or urban training environment or future
surgical practice. Furthermore, the simulation equipment required to implement these tasks is
readily available in all basic simulation centres with no specialized equipment required.
The reason expert consensus was used to select technical tasks is due to the paucity of
information available in the literature to select evidence based technical tasks for selection.
Furthermore, many questions remain unanswered, including what the appropriate standards are
149
for each task, how well these tests discriminate between technical performance and at what time
point(s) in the learning of these skills reflects future clinical performance. Therefore, although
expert consensus was used as a starting point to choose technical tasks, original research is
needed to delineate these details.
7.2.4 To quantify different learning patterns among trainees for both basic and
more advanced laparoscopic and open skills
The existing surgical literature had examined the LCs of single simulated laparoscopic tasks or
procedures,(Grantcharov & Funch-Jensen, 2009; M. P. Schijven & Jakimowicz, 2004) but had
not determined the learning patterns of tasks of varying difficulty. Furthermore LC data for open
tasks to discriminate different learners had not been reported in the literature.
In today’s surgical training environment, at the time of graduation residents are expected to be
component in both open and laparoscopic procedures. With that said, subspecialty training
within GS allows each individual to tailor their surgical practice to encompass technical elements
that complement their strengths and interests (e.g. breast surgery requires predominately open
skills vs bariatric and foregut surgery which require advanced laparoscopic skills). Thus,
understanding learning curve patterns across different tasks within open and laparoscopic skill
may be helpful in understanding where low performers are most disadvantaged.
Overall, learning patterns ranged from very efficient learning, demonstrated by a steady
progression towards competence, to, in contrast, widely variable learning, where improvement
was incrementally small due to highly fluctuant performance. Furthermore, as the task increased
in difficulty, variability in performance increased as well. Trainees took longer to reach
competence and found it to be more difficult to maintain consistent performance at this level.
Therefore, trainees and faculty should be aware that fluctuating performance is expected as task
difficulty increases and increased practice will be required to maintain competence performance.
Simulation training is likely most beneficial for the fluctuating portion of the learning curve of
these tasks.
More specifically, four distinct learning clusters were identified within these general learning
patterns. Clusters 1-4 grouped individuals over the range from high to low performers,
150
respectively. Low performers had persistent difficulty with laparoscopic tasks as compared to
open tasks, and also demonstrated the most significant amount of fluctuation between repetitions.
7.2.5 To determine whether trainees stay within their learning patterns across
simulated tasks of varying difficulty (basic and advanced) and type
(minimally invasive and open)
The study described in chapter 5 demonstrated that were was a significant association between
individuals to stay within their assigned cluster with lateral movement to a neighboring cluster.
Thus, despite the tasks increasing in difficulty, low performers demonstrated poor acquisition of
technical skill within laparoscopy as compared to their peers, even when learning the easier
tasks.
This finding is interesting because anecdotally it may be hypothesized that all trainees will be
able to learn easy tasks but not all trainees will be able to learn more difficult tasks. However,
this was not the case. Instead, trainees who demonstrated difficulty with basic tasks had even
more difficulty with advanced tasks. Furthermore, this provides evidence that screening with
basic laparoscopic skills may a sufficient discriminatory test to differentiate technical aptitude
for incoming trainees.
7.2.6 To identify a subset of trainees who consistently fail to reach proficiency
on simulated tasks and determine the features of their learning curves that
separate them from their peers
The distinct learning features of low performing trainees were unique to laparoscopic tasks and
included high variability between repetitions, inability to reach the proficient cut off score and no
true plateau phase in the learning curve. These learning curve features were demonstrated in the
simulation center, which is a very controlled environment when compared to the operating room.
The equipment, instrument set-up and simulation models were constant throughout the learning
period and despite this, variable performance was demonstrated from the lowest performers. The
real operating environment is far more complex. Patient factors include tissue integrity, set-up,
body habitus and anatomy different from case to case. Therefore, it may be reasonable to infer
that these factors would further exacerbate already variable technical performance.
151
Furthermore, the question whether continued practice would eliminate these concerning learning
curve features is unresolved. So far in the surgical education literature, 10-30 repetitions have
been used to quantify learning curves of simulated tasks. The study in chapter 5 increased the
practice window to 40 repetitions, and when comparing the results of each task a 5-20% low
performing group was consistently present. Furthermore, current surgical trainees are expected to
reach technical competence within an environment with work hour restrictions, increasing
patient safety precautions and where surgical techniques are becoming more difficult. Acquiring
technical skill at a reasonable rate is necessary to progress through training in a timely manner.
In addition, technical performance is only one of the many required competencies set by the
Royal College of Physicians and Surgeons. Therefore, dedicated time is required to also grow the
non-technical elements of becoming a safe surgeon. Although the accepted time to acquire a
technical skill to competence has not been formally defined within the literature, forty repetitions
is the longest training window that has presently been examined in the literature. The associated
learning curve features demonstrated by low performing trainees provide some evidence that
proficiency will not be met in a timely manner, as progress toward the proficiency threshold was
not convincing.
During training
7.2.7 To develop a MP script for the performance of an advanced laparoscopic
procedure
A mental practice script was created primarily focusing on CS mental practice as compared to
MG or MGA. CS mental practice was chosen because it is recommended when trainees are
beginning to learning new skills, pertinent to the study population of interest, in comparison to
expert surgeons who would likely benefit from a performance enhancement script that focuses
on MG or MGA.
152
7.2.8 To assess the effectiveness of MP on advanced laparoscopic technical
skill performance
The effectiveness of mental practice was explored as a technique to be incorporated into surgical
residency specifically as an adjunct to improved technical performance in senior level trainees.
In the RCT described in chapter 6, the trainees who participated in MP technically outperformed
the control group. As described in section 2.6.2, MP focuses on the cognitive process of learning
an operation as compared to physical practice. To date, simulation curricula have largely focused
on physical practice, i.e. performing a task and learning through repetitive practice. However,
although simulation has advanced tremendously over the years, creating a realistic simulation of
full laparoscopic operations or even a portion of an operation, is challenging. Trainees can
practice operations on a VR simulator, however senior level residents who have experience
operating with real tissue and real surgical equipment do not respond well to VR simulation due
to the limitations of the technology to create a realistic operative feel (Shetty, Zevin,
Grantcharov, Roberts, & Duffy, 2014). Furthermore, to create a realistic operation in a box
trainer requires a high fidelity model (ie cadaveric porcine bowel), which is expensive and often
limited to one time use. Therefore, instead of relying on physical practice alone in senior
trainees, MP can be used as an adjunct to enhance technical performance in the real operating
room.
7.2.9 To determine whether MP is associated with differences in stress levels
and improvement in non-technical skills in a simulated crisis scenario.
Mental practice training was not associated with differences in stress levels or improvement in
non-technical skills in a simulated crisis scenario. The reason MP was not associated with
decreased stress level may be due to the way stress was measured. The non-invasive physiologic
stress levels used included blood pressure and heart rate, with a higher heart rate and blood
pressure indicative of increased stress. However, all surgical trainees entering the simulated
operating room scenario had heart rate and blood pressure levels above their baseline and the
incremental difference did not differ between the groups. More sensitive and specific measures
of stress such as heart rate variability or salivary cortisol may have detected a difference,
however were not available(Crewther et al., 2016; Kirschbaum et al., 1996). In addition, the
script
pract
and c
techn
7.3
Each
study
influe
and in
opera
7.3.1
The m
the u
limite
optim
tools
set th
differ
A com
is the
of ha
off sc
of the
There
of a l
ts were focu
ice as compa
coping durin
niques may h
Limitat
chapter’s fu
y. However,
ence surgica
n-training: 1
ative technic
1 Defin
methodology
se of a defin
ed by the fac
mal cut off sc
. Some reaso
hese cut off s
rent threshol
mmon metho
e use of expe
ving an expe
core as one o
eir performa
e are many p
limited numb
used on the te
ared to MG
g stressful m
have shown a
tions
ull text manu
in this overa
al education r
1. defining cu
cal skills usin
ning cut of
y portion of t
ned threshold
ct that it rem
cores should
on for this la
scores, the in
lds of perfor
odology for
ert surgeons’
ert surgeon c
or two interq
ance across a
problems wit
ber of expert
echnical com
or MGA. M
moments. Th
a difference
uscript includ
all limitation
research in t
ut off scores
ng GRS and
ff scores
this thesis th
d for compet
mains unclear
d be for diffe
ack of clarity
nconsistent t
rmance, and
setting cut o
’ performanc
complete fiv
quartile rang
all repetition
th this appro
ts who may n
mponent of th
MG and MGA
herefore, train
in stress lev
des a limitat
ns section I w
the area of te
s for compete
3. adequate
hat investiga
tence in tech
r within the s
rent technica
y may be the
erminology
the inability
off scores de
ce as a bench
ve to ten repe
es or standar
s (E. M. Ritt
oach. First, th
not represen
he operation
A MP have a
ning the part
vel and perha
tions section
will focus on
echnical skil
ence, 2. asse
sampling.
ated participa
hnical perform
surgical edu
al task perfo
e simplified m
used within
y to transfer c
escribed in te
hmark. For e
etitions of a
rd deviation
ter & Scott,
he cut off sc
nt the popula
n, which is k
a stronger foc
rticipants wit
aps also in n
n specifically
n three broad
ll assessmen
essment of si
ants’ learnin
rmance. How
ucation litera
ormance leve
methodology
n the literatur
cut off score
echnical skil
example, thi
given task, a
s worse than
2007; Zevin
core depends
ation norm fo
known as CS
cus on relax
th these MP
non-technical
y pertaining t
der concepts
nt in both sele
imulated and
ng curves req
wever, this w
ature what th
els and asses
y used by stu
re to describ
es across stu
lls simulation
s approach c
and defining
n the median
n B et al., 20
s on the perfo
or the accept
153
S mental
ation
l skill.
to the
that
ection
d real
quired
work is
he
ssment
udies to
e
dies.
n studies
consists
g the cut
n or mean
13).
formance
ted
154
safety threshold. Second, the selected experts are often subspecialty trained surgeons (i.e.
experienced laparoscopic surgeons for laparoscopic skill or experienced open surgeons for open
skills). Therefore, their performance scores may be consistently far above the competency
threshold, closer to a proficient or expert level of performance.
Within the surgical education literature, the words competent and proficient are often used
interchangeably, despite these terms describing two distinct performance levels. When these
terms are linked back to their origin in education theory, they are meant to describe different
phases of skills acquisition. Dreyfus and Dreyfus (1980) outline a five-stage model of the mental
activities involved in directed skill acquisition (Dreyfus & Dreyfus, 1980). A student will begin
as a ‘novice,’ and at this stage of learning the trainee is dependent on the rules and only feels
responsibility to follow the rules. With practice and experience the student becomes an
‘advanced beginner’ where he/she identifies conditional rules but continues to only feel a sense
of responsibility within the rules. The next phase is termed ‘competent,’ and is when the student
feels responsible for making decisions and begins to sort information by importance. The forth
state is ‘proficient’ whereby the student feels even more responsibility for his/her actions and is
able to use pattern recognition to assess what to do. Lastly ‘expert’ level is reached as the skills
become intuitive or automatic (Figure 17). Therefore, using the terms competent and proficient
interchangeably is incorrect and confuses different stages of learning. Discriminating these levels
may be helpful in setting cut off scores for trainees of different levels and may shield against
setting scores that are too high or too low.
155
Figure 17: A five-stage model of the mental activities involved in directed skill acquisition
(adapted from (Dreyfus & Dreyfus, 1980)).
Therefore, using methodology that is able to discriminate between competent and proficient may
help with this long-standing limitation of setting cut off scores.
Recently, standard setting methodology has been suggested as an alternative approach to setting
cut off scores. Within medical education, standard setting has been used to set pass/fail scores for
high-stakes assessments for written, oral and OSCE examinations (McKinley & Norcini, 2014;
Norcini, 2003), however this methodology is new to technical skills assessments.
Three standard setting methodologies have been demonstrated to appropriately set pass/fail
scores for technical skills of surgical trainees. These include: contrasting groups, borderline
group and borderline regression methods (Figure 18 and Figure 19) (de Montbrun,
Statterthwaithe, & Grantcharov, 2015). Contrasting groups is centered on the idea that within a
given population there are trainees that will undoubtedly pass or fail a task (Livingston & Ziesky,
1982). For example when performing a technical task an expert examiner will score the trainee’s
overall performance as competent or not competent. Plotting histograms of the scores (either
GRS or checklist scores) in these two categories will result in an intersection between the groups
156
which is determined as the passing score (Norcini, 2003)(Figure 18 a). Borderline group
methodology is centered on the idea that the pass/fail score should be set at the level of the
borderline student (Sturmberg & Hinchy, 2010). The borderline student is defined as the
individuals sitting on the edge of the passing score and this score becomes the pass/fail score
(Figure 18b). For example when the student is performing a technical task the expert examiner
will score their performance as neither competent or not competent but borderline.
a. Contrasting Group b. Borderline Group
Figure 18: Standard setting using: a. contrasting groups and b. borderline group methodology
(images adapted from (de Montbrun, Statterthwaithe, & Grantcharov, 2015))
Borderline regression is also centered on the idea of the borderline candidate (Sturmberg &
Hinchy, 2010). However, instead of using a subset of scores for the calculation of the pass fail
score, a linear regression analysis is used (Figure 19).
157
Figure 19: Standard setting using borderline regression methodology (image adapted from (de
Montbrun et al., 2015))
However, using standard setting methods to set cut off scores does not imply that these scores
will be transferable to other studies. Fraser et al. set passing scores using the contrasting groups
methodology (Figure 18a). The contrasting groups were defined as senior residents and junior
trainees and the interception of their scores was termed the competence threshold (Fraser et al.,
2003). However, despite using one of the described standard setting methodologies, the reported
cut off ‘competency’ scores in this study seem to be very low compared to the proficiency
threshold set by Ritter et al. Reasons for these low scores could be that the residents who
participated in the study may have had minimal experience with the FLS tasks resulting in low
performance scores, which then shifted the competency threshold downward, affecting their
transferability to other studies.
When seeking cut off scores for chapter 5 of the present work, the cut off scores reported by
Fraser et al. were found to be too low for the participants. The medical students in cluster 1 and 2
(top and high performers) surpassed the cut off scores set by Fraser et al. within their first 2-5
repetitions of the FLS tasks, yet it was obvious to the instructors that they were not comfortable
performing the task and certainly far from competent in performing them.
Therefore, there is a need within the field of surgical education to develop and define cut off
scores that are reliable and credible, to use consistent terminology when referring to the
158
performance thresholds that inform the setting of cut off scores, and to strive to set scores for
skills commonly assessed in the literature that are transferable across studies.
7.3.2 The implementation of global rating scales as a routine assessment
method during selection and surgical training
Global rating scales are considered the gold standard in assessing technical performance during
procedures or operations, and are believed to be superior to check lists due to their ability to
discriminate between performance quality (Regehr et al., 1998). However, the inclusion of these
assessment tools in surgical curricula and ongoing surgical training is limited due substantial
time and resource requirements associated with their regular and objective use.
Watching each operation to determine a GRS performance score is extremely time consuming.
To use the GRS assessment tool correctly, the assessor is required to watch the procedure in
normal playback speed or in real time. This method was used in chapter 5 for the two open tasks,
which were graded using the mOSATS GRS. The present work required the assessment of 5200
videos, which resulted in 260 hours of rating time. In the context of a single study, this approach
was feasible. However, if implemented as a routine assessment tool within an ongoing
educational curriculum, the time requirements may make assessment by GRS unsustainable.
Similarly, an attempt at ongoing assessment of surgical residents’ intraoperative performance
with a GRS would result in even more substantial time requirements, given that full surgical
procedures commonly range from 2 to 8 hours in duration, substantially more than the
approximately 5 minute duration of the individual technical tasks performed by the participants
in the present work.
In addition to their time consuming nature, to reduce the bias and subjective nature of GRS, these
assessment tools can be completed in a blinded fashion using video recordings. However to
accomplish this, dedicated raters are needed to assess the videos, which is costly and requires
significant infrastructure. Even if such raters were available, rater fatigue is problematic and can
lead to decreased assessment quality. Rating videos daily for long periods of time is a
monotonous and passive process. Intermittent assessment of inter-rater reliability to ensure
calibration is maintained is important to maintain quality. However, again although doable in a
159
study setting for a relatively short and finite assessment period, using GRS as a routine measure
of trainee competence in a surgical training program may not be a feasible solution. Therefore,
seeking reliable objective assessments that are less time consuming, yet meaningful, is required
to improve assessment of open technical skills and intraoperative procedural skills.
7.3.3 Adequate sampling
Studies within the field of surgical education are often limited by the sample size. Participants
for these studies are recruited from within the medical system, specifically medical students,
residents, fellows or faculty. This limits the potential number of participants available for
recruitment. Furthermore, within the group of potential participants at a given training level, sub-
populations exist that cannot always be considered similar for the purpose of many education
research questions. For example first year and fifth years residents are characterized by very
different levels of technical skill, surgical judgment, decision making and patient management
experience. Therefore residents often cannot be studied as a single cohort.
Powering studies adequately has not been a problem for surgical education studies that compare
two teaching paradigms or when constructing two technical skills curricula (Louridas, Bonrath,
Sinclair, Dedy, & Grantcharov, 2015; Palter & Grantcharov, 2012). In chapter 6, a randomized
controlled study was designed to assess the effect of MP on the interventional group compared to
the conventional residency-training group. In this study, a sample size calculation of 10
participants per group was adequate to detect a difference in the primary outcome. However, as
the surgical education field expands and studies attempt to adopt analytical techniques from the
field of clinical epidemiology to explore the effect of technical skill on patient outcomes
(Birkmeyer et al., 2013) or predictive modeling statistics to predict outcomes, larger sample sizes
will be required. Sixty-five medical students were recruited for the technical skill-training
curriculum described in chapter 5. This study is the largest of its kind in the surgical education
literature and was able to identify performance clusters while exploring the relationship between
clusters using non-parametric statistical tests. However, predictive modeling statistics or
analytics would be very useful for this research. Creating learning curve models with adequate
sensitivity and specificity to predict incoming trainees technical performance would require
160
hundreds of participants and therefore was not feasible for this study. Therefore, to overcome the
inherent limitation of small sample sizes seen in surgical education, multicentre studies should be
encouraged. Furthermore, national surgical education databases are not available to researchers
in the field, further limiting the ability to aggregate national data over many years. Multicentre
studies may also help with this limitation.
C8
Surgi
surge
is to m
surge
are re
demo
the re
aptitu
techn
The f
that s
3D vi
Howe
lapar
spatia
Thus
perfo
abilit
There
hypot
asses
quest
the se
have
progr
aptitu
Chapter 8
ical program
ery. Once the
mentor these
eons. Of the
equired to re
onstrates that
esearch inclu
ude and asse
nical perform
first study ex
surrogate ma
isual spatial
ever, of the s
oscopic expe
al tests did n
, although th
ormance may
ties, rather th
efore, instea
thesized that
sing novice
tionnaire wa
election of te
difficulty re
ram. This sta
ude was a wo
8: Conclu
ms endeavor t
ese students
e students ov
many essent
each technica
t not all train
uded in this t
ssed whethe
mance in surg
xplored surro
arkers, name
tests correla
surgical exp
erience pred
not demonstr
he results we
y have been
han visual sp
d of relying
t simulated t
trainees’ lea
s distributed
echnical apti
eaching techn
atement reco
orthwhile en
usion
to select med
have been s
ver a five-ye
tial compete
al competenc
nees reach th
thesis focuse
er the cogniti
gical trainee
ogate tests th
ly previous
ate with base
erience and
dicted baselin
rate a consist
ere underwhe
an over simp
patial ability
on surrogate
technical tas
arning curve
d to program
itude. PDs re
nical compet
onfirmed tha
ndeavor. The
dical studen
elected into
ear training p
encies acquir
ce prior to g
his performa
ed on seekin
ive training
s.
hat may pred
surgical exp
eline laparos
non-surgica
ne surgical s
tent associat
elming, expe
plification o
alone.
e markers as
sks are better
s over multip
m directors ac
eported that
tence at the
at continuing
e Delphi con
nts who are b
the surgical
period to gra
red by trainin
graduation. H
ance threshol
ng a reliable
adjunct, MP
dict technica
periences, no
scopic skills
al experience
skill for inco
tion with ove
ecting surrog
f a complex
s predictors o
r suited to st
iple repetitio
cross Canada
approximate
end of a stan
g to seek a sc
ncluded that
best suited to
l training pro
aduate as saf
ng doctors, a
However, em
ld (Cushchie
screening pr
P, could be u
al aptitude. It
on-surgical e
in the novic
es studied, on
oming trainee
erall technic
gate makers
process of m
of technical
tratify techni
ons. A nation
a to gain the
ely 5-15% fo
ndard five-y
creening tool
PDs reached
o pursue a ca
ogram the ob
fe independe
all surgical r
merging evide
eri, 2003). T
rocess for te
used to impro
t was hypoth
experiences a
ce surgical tr
nly previous
es. Similarly
cal performan
to predict fu
many interac
skill, it was
ical aptitude
nal Delphi co
ir expert opi
or surgical tr
year training
l for technic
d positive co
161
areer in
bjective
ent
residents
ence
Therefore,
chnical
ove
hesized
and 2D-
rainee.
s
y, visual
nce.
uture
cting
by
onsensus
inion on
rainees
al
onsensus
162
that incorporating both basic open and laparoscopic tasks would be appropriate for incoming
trainees. Therefore, open and laparoscopic technical skills were incorporated into the next study.
The next study was designed to determine whether examining learning curves of simulated
technical tasks over a one-month practice period would differentiate technical aptitude between
students. It was hypothesized that medical students would display similar learning curves for
disparate basic laparoscopic and open surgical skills, and that these will be correlated with their
potential to reach proficiency in subsequent, more complex technical tasks. The results proved
the hypothesis to be partly true. Four distinct learning curve clusters were identified. Cluster 1
selected students who demonstrated strong innate ability and were able to effortlessly learn all
the technical skills, reaching competence quickly. Cluster 2 students were called high performers
because they all eventually reached competence in all tasks but took slightly more time then their
cluster 1 peers. Cluster 3 students were moderate performers and demonstrated difficulty,
reaching proficiency in the more complex laparoscopic tasks. Cluster 4 individuals were unable
to reach technical competence in 4 of 5 tasks including all laparoscopic tasks.
On examining the learning curves, it became apparent that cluster 1 and 2 had strong innate
ability when compared to cluster 3 and 4 students. Furthermore, cluster 3 and 4 individuals did
not catch up to their peers in cluster 1 and 2 over the training period, and demonstrated a
continued disadvantage. However, when comparing the groups, cluster 3 individuals
demonstrated somewhat stable performance below the competency threshold whereas cluster 4
individuals demonstrated large variability throughout their learning curves with no progression
towards a plateau phase. Therefore, it was concluded from this study that learning curve clusters
could be used as a screening tool to identify trainees with different levels of innate ability.
Furthermore, screening for cluster 4 individuals should be considered due to their poor innate
ability, inability to reach the proficiency threshold and variable unstable performance over the
full durations of their learning curve.
In the final study, it was hypothesized that MP aimed at teaching the visual and kinesthetic cues
for the crucial operative steps in a laparoscopic jejunojenunostomy (JJ), as well as for the
management of adverse situations, improves surgical technical and non-technical performance
and decreases stress levels experienced by the surgeon. Mental practice significantly improved
advanced laparoscopic technical skill in the interventional group. Counter to the hypothesis,
163
however, MP was not associated with differences in stress levels or improvement in non-
technical skills in the simulated crisis scenario.
Overall this thesis has proposed a selection curriculum from incoming trainees to assess
technical skill ability by assessing learning curves and demonstrated the effectiveness of MP as
an adjunct to technical performance in senior surgical trainees.
C9
This
ident
and te
asses
valid
select
CanM
resea
meas
quest
moda
9.1.1
To es
asses
ident
Ideal
asses
feasib
altern
real t
room
opera
comp
case e
the re
is dif
utiliz
may b
Chapter 9
thesis focuse
ifying a num
echnical skil
sments, for a
ity. Second,
tion process
MEDs compe
arch is neede
uring wheth
tionnaires, th
alities.
1 Explo
stablish pred
s trainees ob
ify assessme
ly, blinded G
sment tools.
ble due to th
nate approac
ime. The dif
m team comp
ational requi
pleting patien
etc. In additi
esidents and
fficult. Howe
zing both sur
be worth inv
9: Future
ed on selecti
mber of impo
lls, there is a
assessment t
in additiona
es with the c
etencies outl
d to assess t
her students p
herefore mor
oring intra
dictive validi
bjectively in
ent methods
GRS would b
. However, a
he amount of
ch may be to
fficulty for li
lete the asse
rements invo
nt charting r
ion, bias is a
therefore se
ever, prelimi
rgical and no
vestigating (C
e directio
ion and in-tr
ortant directi
a need to exp
to be feasibl
al to technica
creation of p
lined by the
the effect of
performed M
re rigorous e
aoperative t
ity for select
the real ope
that can be e
be utilized w
as mentioned
f manpower
train the fac
ive assessme
essment direc
olved in kee
records, patie
an inherent c
eparating the
inary eviden
on-surgical m
Crossley, M
ons
raining asses
ions for futur
plore alterna
e in longitud
al skill, futur
predictive ob
Royal Colle
MP practice
MP has only
evaluations m
technical s
tion clusterin
erating room
efficiently an
with objectiv
d in the limit
and the hour
culty and nur
ents is obtain
ctly after the
eping the ope
ent transfer,
confound, be
e personal dy
nce from the
members of t
Marriott, Purd
ssment of tec
re research.
ative strategi
dinal studies
re work shou
bjective asses
ege of Physic
e in the real o
been accom
may be bene
skills asses
ng described
m. However, f
nd routinely
ve raters who
tations sectio
rs required t
rses to use th
ning timely
e case is ofte
erating room
cleaning the
ecause facult
ynamics to a
non-technic
the operative
die, & Beard
chnical skills
Firstly, in th
ies to the use
s to be able t
uld focus on
ssment tools
cians and su
operating ro
mplished usin
eficial using
ssment
d in Chapter
further work
y used in this
o are trained
on, this meth
to rate traine
he GRS and
scores. Havi
en challengin
m running eff
e room, prep
ty and nurse
assess technic
cal skills liter
e team to ass
d, 2011).
s, with the fi
he area of se
e of GRS
to measure p
n improving
s for the othe
urgeons. Thir
om. Fourth,
ng rating scal
functional im
5, it is essen
k is needed t
s environmen
to use these
hod is often
ees’ performa
d rate the resi
ing the opera
ng given the
ficiently e.g.
paration for t
s work close
cal skill in is
rature sugge
sess surgical
164
indings
election
predictive
current
er
rd, future
le
maging
ntial to
to
nt.
e
not
ance. An
idents in
ating
other
.
the next
ely with
solation
ests that
l trainees
165
Another novel area of future research may be in video machine learning. This is a field gaining
momentum in computer programing and data science. The concept that a computer can be
programed to interpret video and differentiate good and bad behaviors is fascinating and yet to
be investigated in surgical education. Therefore, collaboration between video data scientists and
surgeons may serve fruitful in working towards computer video assessments. If computers were
able to reliably assess performance this would eliminate bias and human fatigue and promote
sustainable intraoperative assessment.
Using these methods will allow students who have been assigned a performance cluster to be
followed over time to assess whether their learning curve established during medical school is
indicative of intraoperative performance during training.
9.1.2 Establish assessment measures of non-technical competencies
The work in this thesis focuses on technical performance. However, the technical performance of
a surgeon is only one component of being a surgeon. Moreover, as demonstrated by the Delphi
consensus questionnaire, various non-technical skills such as work ethic and passion for surgery
were perceived as highly important by PDs. Furthermore, the Royal College of Physicians and
Surgeons has clearly outlined seven roles in which each physician should strive to achieve
competency during their training. These include the medical expert, communicator, collaborator,
leader, health advocate, scholar and professional roles (("CanMEDS 2015: The next evolution of
the CanMEDS Framework," 2013)). Investigating methods that assess attributes that are directly
linked to these competencies is an essential step in all phases of training. Unfortunately, many of
these attributes are difficult to measure using quantitative methodologies. Therefore qualitative
or mixed method techniques may be more appropriate in understanding how to best assess these
non-technical skills and how to incorporate them into our current selection process.
9.1.3 Quantifying mental practice objectively and transferring this techniques
into the real operating room
Mental practice proved to be a successful adjunct to technical skills training within the simulated
operating room even with added stress. Two areas of future research include translating this
166
technique into the real operating room, and quantifying MP using electroencephalograms (EEG)
or functional magnetic resonance imaging (fMRI) for laparoscopic surgery.
The difficulty with studying the effect of MP on technical skill in the real operating room is
isolating this intervention from the numerous confounders that may also affect technical
performance. Firstly, each patient has anatomical differences that can make the same operation
technically more or less challenging. Secondly, the operating room environment and personal
that work within the OR may also change, which theoretically could also alter technical
performance. Finally, fortunately adverse events are rare, and therefore measuring performance
with respect to a direct stress is not feasible in the real operating room. However, there are
validated metrics to adjust technical skill performance scores to account for case difficulty that
could be incorporated during the assessment to control for some of these factors. Furthermore,
selecting an advanced surgical technique that is very standardized, such as Roux-en-y by pass
may also improve the feasibility of translating this technique into the operating room.
Another area of future research may be to measuring MP directly using functional neuroimaging
modalities that record brain activity noninvasively, such as EEG or fMRI. Within laparoscopic
surgery the use of MP has been quantified using validated questionnaires (e.g. MIQ), which are
subjective in nature and therefore are prone to bias. However, authors have described using
modalities such as EEG or fMRI to assess the use of mental imagery in real-time. The use of
these modalities in the context of assessing surgeon performance during real or simulated
surgical scenarios is in its infancy and has only been assessed in microsurgery but never in
laparoscopy (Ros et al., 2009; Wanzel et al., 2007). Therefore, future studies may consider using
these direct measures to assess whether mental imagery has improved between groups.
167
Copyright Acknowledgements
Chapter 2 (section 2.3.2)
Wolters Kluwer Health Lippincott Williams & Wilkins grants the authors permission to reuse
their full-text article free of charge (non-commercial/ non-for-profit purposes) for the purpose of
reuse in a dissertation.
Louridas M, Szasz P, de Montbrun S, Harris KA, Grantcharov T. Can we predict technical
aptitude? A systematic review. Ann Surg. 2015 June 15
Copyright (C) 2016 Wolters Kluwer Health, Inc. All rights reserved.
The final publication is available at:
http://journals.lww.com/annalsofsurgery/pages/articleviewer.aspx?year=9000&issue=00000&art
icle=97346&type=abstract
Chapter 3
Copyright permission from Springer Science+Business Media
Louridas M, Quinn E, Grantcharov TP. Predictive value of background experiences and visual
spatial ability testing on laparoscopic baseline performance among residents entering
postgraduate surgical training. Surg Endo.2015 June 20.
Published online 20 June 2015 © springer Science+Business Media New York 2013.
The final publication is available at:
http://link.springer.com.myaccess.library.utoronto.ca/article/10.1007%2Fs00464-015-4313-8
Chapter 6
Copyright permission form the British Journal of Surgery through John Wiley and Sons states:
A. The following rules apply to all Users:
168
You may download, view, copy and save to hard disk or diskette and store or print out single
copies of individual articles or items for your own personal and non-commercial use, scholarly,
educational or scientific research or study. You may transmit to a third party, in hard copy or
electronically, a single article or item from the Website for that party's own personal and non-
commercial use or scholarly, educational, or scientific research or study. In addition, you have
the right to use, with appropriate credit, brief excerpts from the Website content in your own
work for the above purposes.
Louridas M, Bonrath EM, Sinclair DA, Dedy NJ, Grantcharov TP. Mental practice to enhance
advanced laparoscopic surgical performance in the operating room: a randomized controlled
trial. Br J Surg. 2015 Jan;102(1):37-44.
© 2014 BJS Society Ltd. Published by John Wiley & Sons Ltd
Original full- text article available at: http://www.bjs.co.uk/details/article/7095621/Randomized-
clinical-trial-to-evaluate-mental-practice-in-enhancing-advanced-lapa.html
10
Accre
Accre
Adam
Adm
Adm
Alpay
Alter
Alvan
Alvan
Refere
editation Co
December
http://ww
editation Co
Graduate
Education
http://ww
_surgery_
ms, B. J., Ma
simulator
69(6), 714
ission requir
22, 2015,
http://ww
ission requir
from http
ydin, E. (201
rman, D. M.,
Predictive
J Surg Ed
nd, A., Aupl
and variat
nd, A., Aupl
and variat
ences
ounci for Gra
r 4, 2013, fro
ww.acgme.org
ouncil for Gr
Medical Edu
n in General
ww.acgme.org
_01012008_0
argaron, F., &
s in the deve
4-717.
rements for M
from
ww.md.utoron
rements Univ
://mdprogram
14). Introduc
, Jones, T. M
e Value of G
duc, 68(6), 5
lish, S., Gill,
tion in learni
lish, S., Gill,
tion in learni
aduate Medic
om
g/acgmeweb
raduate Medi
ucation, AC
Surgery. R
g/acgmeweb
07012012.pd
& Kaplan, B
elopment of
Medical Sch
nto.ca/admis
versity of Br
m.med.ubc.c
ction to Mac
M., Heidel, R
General Surg
13-518.
, H., & Rees
ing curves. J
, H., & Rees
ing curves. J
cal Educatio
b/tabid/271/G
ical Educatio
CGME Progr
Retrieved Ma
b/Portals/0/P
df
B. J. (2012). C
laparoscopic
hool at the U
ssions/inform
ritish Colum
ca/admission
chine Learni
R. E., Daley,
ery Applicat
s, J. (2011). I
J Bone Joint
s, J. (2011). I
J Bone Joint
on. (2013). W
GraduateMe
on. (2013). A
ram Requirem
ay 26, 2014,
PFAssets/Pro
Comparing v
c skills in su
Univeristy of
mation/requi
mbia (2015).
ns/admission
ing. Cambrid
B. J., & Gol
tion Data for
Innate arthro
t Surg Am, 9
Innate arthro
t Surg Am, 9
Work hour re
edicalEducat
Accreditatio
ments for Gr
, from
ogramRequir
video games
urgical reside
f Toronto. (2
irements/Un
Retrieved
n-requiremen
dge, MA: Th
ldman, M. H
r Future Res
oscopic skill
93(19), e115(
oscopic skill
93(19), 1-9.
estrictions.
tion/DutyHo
n Council fo
raduate Med
rements/440
s and laparos
ents. J Surg
015). Retri
dergraduate
January 22,
nts/
he MIT Pres
H. (2011). Th
sident Perfor
ls in medical
(111-119).
ls in medical
169
urs.aspx
or
dical
0_general
scopic
Educ,
ieved Jan
.htm
2015,
s.
he
rmance.
l students
l students
170
Alvand, A., Auplish, S., Khan, T., Gill, H. S., & Rees, J. L. (2011). Identifying orthopaedic
surgeons of the future - The inability of some medical students to achieve competence in
basic arthroscopic tasks despite training: A randomised study. 93 (12), 1586-1591.
Alvand, A., Gill, H. S., Price, A. J., Dodd, C. A., Murray, D. W., & Rees, J. L. (2012). Does a
mixed training course on the Oxford unicompartmental knee arthroplasty improve non-
technical skills of orthopaedic surgeons? J Orthop Surg (Hong Kong), 20(3), 356-360.
Antiel, R. M., Thompson, S. M., Camp, C. L., Thompson, G. B., & Farley, D. R. (2012).
Attracting students to surgical careers: preclinical surgical experience. JAMA Surg, 69(3),
301-305.
Arora, H., Uribe, J., Ralph, W., Zeltsan, M., Cuellar, H., Gallagher, A., & Fried, M. P. (2005).
Assessment of construct validity of the endoscopic sinus surgery simulator. Arch
Otolaryngol Head Neck Surg, 131(3), 217-221.
Arora, M., Diwan, A. D., & Harris, I. A. (2013). Burnout in orthopaedic surgeons: a review. ANZ
J Surg, 83(7-8), 512-515.
Arora, S., Aggarwal, R., Moran, A., Sirimanna, P., Crochet, P., Darzi, A., Kneebone, R., &
Sevdalis, N. (2011). Mental practice: effective stress management training for novice
surgeons. J Am Coll Surg, 212(2), 225-233.
Arora, S., Aggarwal, R., Sevdalis, N., Moran, A., Sirimanna, P., Kneebone, R., & Darzi, A.
(2010). Development and validation of mental practice as a training strategy for
laparoscopic surgery. Surg Endosc, 24(1), 179-187.
Arora, S., Aggarwal, R., Sirimanna, P., Moran, A., Grantcharov, T., Kneebone, R., Sevdalis, N.,
& Darzi, A. (2011). Mental practice enhances surgical technical skills: a randomized
controlled study. Ann Surg, 253(2), 265-270.
Arora, S., Sevdalis, N., Nestel, D., Tierney, T., Woloshynowych, M., & Kneebone, R. (2009).
Managing intraoperative stress: what do surgeons want from a crisis training program?
Am J Surgery, 197(4), 537-543.
171
Babineau, T. J., Becker, J., Gibbons, G., Sentovich, S., Hess, D., Robertson, S., & Stone, M.
(2004). The "cost" of operative training for surgical residents. Arch Surg, 139(4), 366-
369; discussion 369-370.
Banerjee, C., Cosentino, M., Hatzmann, W., & Noe, K. G. (2010a). Endoscopic intracorporal
knotting and suture techniques: Talent or experience? Gynecological Surgery, 7, S210-
S211.
Banerjee, C., Cosentino, M., Hatzmann, W., & Noe, K. G. (2010b). Endoscopic intracorporal
knotting and suture techniques: Talent or experience? Gynecol Surg, 7(Suppl 1), S164-
S170.
Beard, J. D. (2008). Assessment of surgical skills of trainees in the UK. Ann R Coll Surg Engl,
90(4), 282-285.
Bell, J. A. (1988). Royal Air Force Selection Procedures. Annals of the Royal College of
Surgeons of England, 70, 270-275.
Bell, R. M., Fann, S. A., Morrison, J. E., & Lisk, J. R. (2011). Determining personal talents and
behavioral styles of applicants to surgical training: a new look at an old problem, part I. J
Surg Educ, 68(6), 534-541.
Bell, R. M., Fann, S. A., Morrison, J. E., & Lisk, J. R. (2012). Determining personal talents and
behavioral styles of applicants to surgical training: a new look at an old problem, part II.
J Surg Educ, 69(1), 23-29.
Biau, D. J., Williams, S. M., Schlup, M. M., Nizard, R. S., & Porcher, R. (2008). Quantitative
and individualized assessment of the learning curve using LC-CUSUM. Br J Surg, 95(7),
925-929.
Biondi, A., Grosso, G., Mistretta, A., Marventano, S., Toscano, C., Drago, F., Gangi, S., &
Basile, F. (2013). Laparoscopic vs. open approach for colorectal cancer: evolution over
time of minimal invasive surgery. BMC Surg, 13 Suppl 2, S12.
172
Birkmeyer, J. D., Finks, J. F., O'Reilly, A., Oerline, M., Carlin, A. M., Nunn, A. R., Dimick, J.,
Banerjee, M., & Birkmeyer, N. J. (2013). Surgical skill and complication rates after
bariatric surgery. N Engl J Med, 369(15), 1434-1442.
Brandt, M. G., & Davies, E. T. (2006). Visual-spatial ability, learning modality and surgical knot
tying. Can J Surg, 49(6), 412-416.
Brandt, M. G., & Wright, E. D. (2005). Medical student career choice and mental rotations
ability. Clin Invest Med, 28(3), 112-117.
Bridges, M., & Diamond, D. L. (1999). The financial impact of teaching surgical residents in the
operating room. Am J Surg, 177(1), 28-32.
Buckley, C. E., Kavanagh, D. O., Gallagher, T. K., Conroy, R. M., Traynor, O. J., & Neary, P. C.
(2013). Does aptitude influence the rate at which proficiency is achieved for laparoscopic
appendectomy? J Am Coll Surg, 217(6), 1020-1027.
Buckley, C. E., Kavanagh, D. O., Nugent, E., Ryan, D., Traynor, O. J., & Neary, P. C. (2014).
The impact of aptitude on the learning curve for laparoscopic suturing. Am J Surg,
207(2), 263-270.
Burhans, R. S., Richman, C. L., &, & Bergey, D. B. (1988). Mental imagery training: effects on
running speed performance. International Journal of Sport Psychology, 19, 26-37.
Cadeddu, J. A., & Kondraske, G. V. (2007). Human performance testing and simulators. J
Endourol, 21(3), 300-304.
Calatayud, D., Arora, S., Aggarwal, R., Kruglikova, I., Schulze, S., Funch-Jensen, P., &
Grantcharov, T. (2010). Warm-up in a virtual reality environment improves performance
in the operating room. Ann Surg, 251(6), 1181-1185.
Callow, N., & Hardy, L. (2004). The relationship between the use of kinaesthetic imagery and
different visual imagery perspectives. J Sports Sci, 22(2), 167-177.
173
Canadian Resident Matching Service, C. (2014). CaRMS Online technical guide for applicants:
Application period. Retrieved April 9, 2015, from http://www.carms.ca/wp-
content/uploads/2014/09/2015-technical-guide-applicants-application-period.pdf
CanMEDS 2015: The next evolution of the CanMEDS Framework. (2013). Retrieved
December 27, 2013, 2013, from
http://www.royalcollege.ca/portal/page/portal/rc/canmeds/canmeds2015
Carlson, M. L., Archibald, D. J., Sorom, A. J., & Moore, E. J. (2010). Under the microscope:
assessing surgical aptitude of otolaryngology residency applicants. Laryngoscope,
120(6), 1109-1113.
CaRMS. (2016). Canadian residency matching services Retrieved November 23, 2014, from
http://www.carms.ca/
CaRMS: The Match Algorithm. (2015). Retrieved October 19, 2015, from
http://www.carms.ca/en/residency/match-algorithm/
Chamberlain, R. S., Patil, S., Minja, E. J., & Kordears, K. t. (2012). Does residents' involvement
in mastectomy cases increase operative cost? If so, who should bear the cost? J Surg Res,
178(1), 18-27.
Chen, C. C., Green, I. C., Colbert-Getz, J. M., Steele, K., Chou, B., Lawson, S. M., Andersen, D.
K., & Satin, A. J. (2013). Warm-up on a simulator improves residents' performance in
laparoscopic surgery: a randomized trial. Int Urogynecol J, 24(10), 1615-1622.
Cofer, J. B., Lee, K., Pritts, T., & Harvey, A. (2015). ACS/APDS Surgical Skills Curriculum for
Residents. Module 12. Retrieved October 12, 2015, 2015, from
http://elearning.facs.org/course/view.php?id=3
Condon, R., Duvivier, R., Kirition, R., Kafoa, B., McKimm, J., & Roberts, G. (2013). Medical
internship programs in the Pacific: Current situation and future challenges. In H. R. f. H.
K. Hub (Ed.). Sydney, Australia.
174
Cope, D. H., & Fenton-Lee, D. (2008). Assessment of laparoscopic psychomotor skills in interns
using the MIST Virtual Reality Simulator: a prerequisite for those considering surgical
training? ANZ J Surg, 78(4), 291-296.
Crewther, B. T., Shetty, K., Jarchi, D., Selvadurai, S., Cook, C. J., Leff, D. R., Darzi, A., &
Yang, G. Z. (2016). Skill acquisition and stress adaptations following laparoscopic
surgery training and detraining in novice surgeons. Surg Endosc, 30(7), 2961-2968.
Crothers, I. R., Gallagher, A. G., McClure, N., James, D. T., & McGuigan, J. (1999).
Experienced laparoscopic surgeons are automated to the "fulcrum effect": an ergonomic
demonstration. Endoscopy, 31(5), 365-369.
Cuhls, K. (2005). Delphi Method. Retrieved Jan 19, 2015, from
http://www.unido.org/fileadmin/import/16959_DelphiMethod.pdf
Cushchieri, A. (2003). Lest we forget the surgeon. Semin Laparosc Surg, 10(3), 141-148.
Dashfield, A. K., Lambert, A. W., Campbell, J. K., & Wilkins, D. C. (2001). Correlation
between psychometric test scores and learning tying of surgical reef knots. Ann R Coll
Surg Engl, 83(2), 139-143.
Dashfield, A. K., & Smith, J. E. (1998). Correlating fibreoptic nasotracheal endoscopy
performance and psychomotor aptitude. Br J Anaesth, 81(5), 687-691.
Day, J., & Bobeva, M. (2005). A Generic Toolkit for the Successful Management of Delphi
Studies. The Electronic Journal of Business Research Methodology, 3(2), 103-116.
de Montbrun, S., Statterthwaithe, L., & Grantcharov, T. (2015). Setting pass scores for
assessment of technical performance by surgical trainees. British Journal of Surgery,
103(3), 300-306.
Deary, I. J., Graham, K. S., & Maran, A. G. (1992). Relationships between surgical ability
ratings and spatial abilities and personality. J R Coll Surg Edinb, 37(2), 74-79.
175
Dedy, N. J., Szasz, P., Louridas, M., Bonrath, E. M., Husslein, H., & Grantcharov, T. P. (2015).
Objective structured assessment of nontechnical skills: Reliability of a global rating scale
for the in-training assessment in the operating room. Surgery, 157(6), 1002-1013.
Deziel, D. J., Millikan, K. W., Economou, S. G., Doolas, A., Ko, S. T., & Airan, M. C. (1993).
Complications of laparoscopic cholecystectomy: a national survey of 4,292 hospitals and
an analysis of 77,604 cases. Am J Surg, 165(1), 9-14.
Dimitriou, E., Nightingale, A., Khazali, S., Hatzigeorgiades, A., & Prendiville, W. (2009).
Predicting individuals with an aptitude in laparoscopy: Is it possible? Gynecol Surg,
6(Suppl 1), S156-S157.
Downing, S. M. (2004). Reliability: on the reproducibility of assessment data. Med Educ, 38(9),
1006-1012.
Dreyfus, S. E., & Dreyfus, H. L. (1980). A Five-Stage Model of the Mental Activities Involved in
Directed Skill Acquisition.
Drisckell J., Cooper C., & Moran A. (1994). Does mental practice enhance performance? J Appl
Psychol, 79, 481-492.
Dubovsky, S. L., Gendel, M. L., Dubovsky, A. N., Levin, R., Rosse, J., & House, R. (2008). Can
admissions interviews predict performance in residency? Acad Psychiatry, 32(6), 498-
503.
Dubrowski, A., & MacRae, H. (2006). Randomised, controlled study investigating the optimal
instructor: student ratios for teaching suturing skills. Med Educ, 40(1), 59-63.
Ebbinghaus, H. (1913 (Reprinted Bristol: Thoemmes Press, 1999)). Memory. A Contribution to
Experimental Psychology. New York: Teachers College, Columbia University
Eddleman, C. S., Aoun, S. G., & Batjer, H. H. (2013). How to identify the edge of a cliff in the
dark: burnout and neurosurgery. World Neurosurg, 80(5), e111-113.
176
Ekstrom, R. B., French, J. W., Harman, H. H., & Dermen, D. (1976). Kit of Factor-Referenced
Cognitive Tests. In N. J. Education Testing Service Princeton (Ed.), (pp. p1-314). New
Jersey: Office of Naval Research Contract.
Elissen, A. M., Struijs, J. N., Baan, C. A., & Ruwaard, D. (2015). Estimating community health
needs against a Triple Aim background: What can we learn from current predictive risk
models? . Health Policy, 119(9), 672-679.
Enochsson, L., Isaksson, B., Tour, R., Kjellin, A., Hedman, L., Wredmark, T., & Tsai-Fellander,
L. (2004). Visuospatial skills and computer game experience influence the performance
of virtual endoscopy. J Gastrointest Surg, 8(7), 876-882.
Enochsson, L., Westman, B., Ritter, E. M., Hedman, L., Kjellin, A., Wredmark, T., & Fellander-
Tsai, L. (2006). Objective assessment of visuospatial and psychomotor ability and flow
of residents and senior endoscopists in simulated gastroscopy. Surg Endosc, 20(6), 895-
899.
Erasmus, N. (2012). Slaves of the state - medical internship and community service in South
Africa. S Afr Med J, 102(8), 655-658.
Ericsson, K. A. (2007). An expert-performance perspective of research on medical expertise: the
study of clinical performance. Med Educ, 41(12), 1124-1130.
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the
acquisition of expert performance. . Psychol Rev., 100(3), 363-406.
Evgeniou, E., Peter, L., Tsironi, M., & Iyer, S. (2013). Assessment methods in surgical training
in the United Kingdom. J Educ Eval Health Prof, 10(2).
Feldman, L. S., Cao, J., Andalib, A., Fraser, S., & Fried, G. M. (2009). A method to characterize
the learning curve for performance of a fundamental laparoscopic simulator task:
defining "learning plateau" and "learning rate". Surgery, 146(2), 381-386.
Flin, R., Yule, S., Paterson-Brown, S., Maran, N., Rowley, D., & Youngson, G. (2007). Teaching
surgeons about non-technical skills. Surgeon, 5(2), 86-89.
177
FLS Manual Skills Written Instructions and Performance Guidelines. (2014). September 1, 2015,
from http://www.flsprogram.org/wp-content/uploads/2014/03/Revised-Manual-Skills-
Guidelines-February-2014.pdf
Franke, A. G., Bagusat, C., Dietz, P., Hoffmann, I., Simon, P., Ulrich, R., & Lieb, K. (2013). Use
of illicit and prescription drugs for cognitive or mood enhancement among surgeons.
BMC Med, 11, 102.
G., C. (2008). Talent is overrated: what really separates world-class performers from everybody
else. . London: Penguin Books, Ltd.
Gallagher, A. G., Cowie, R., Crothers, I., Jordan-Black, J. A., & Satava, R. M. (2003). PicSOr:
an objective test of perceptual skill that predicts laparoscopic technical skill in three
initial studies of laparoscopic performance. Surg Endosc, 17(9), 1468-1471.
Gallagher, A. G., Leonard, G., & Traynor, O. J. (2009). Role and feasibility of psychomotor and
dexterity testing in selection for surgical training. ANZ J Surg, 79(3), 108-113.
Gallagher, A. G., Neary, P., Gillen, P., Lane, B., Whelan, A., Tanner, W. A., & Traynor, O.
(2008). Novel method for assessment and selection of trainees for higher surgical training
in general surgery. ANZ J Surg, 78(4), 282-290.
Gallagher, A. G., Neary, P., Gillen, P., Lane, B., Whelan, A., Tanner, W. A., & Traynor, O.
(2008). Novel method for assessment and selection og trainees for higher surgical
training in general surgery. ANZ J Surg, 78(4), 282-290.
Gallagher, A. G., Richie, K., McClure, N., & McGuigan, J. (2001). Objective psychomotor skills
assessment of experienced, junior, and novice laparoscopists with virtual reality. World J
Surg, 25(11), 1478-1483.
Gallagher, A. G., & Satava, R. M. (2002). Virtual reality as a metric for the assessment of
laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc,
16(12), 1746-1752.
178
Geoffrion, R., Gebhart, J., Dooley, Y., Bent, A., Dandolu, V., Meeks, R., Baker, K., Tang, S.,
Ross, S., & Robert, M. (2012). The mind's scalpel in surgical education: a randomised
controlled trial of mental imagery. Bjog, 119(9), 1040-1048.
Gettman, M. T., Kondraske, G. V., Traxer, O., Ogan, K., Napper, C., Jones, D. B., Pearle, M. S.,
& Cadeddu, J. A. (2003). Assessment of basic human performance resources predicts
operative performance of laparoscopic surgery. J Am Coll Surg, 197(3), 489-496.
Gibbons, R. D., Baker, R. J., & Skinner, D. B. (1986). Field articulation testing: a predictor of
technical skills in surgical residents. J Surg Res, 41(1), 53-57.
Gladwell, M. (2008 ). Outliers: the story of success. . New York: Little, Brown, and Company.
Graham, B., Regehr, G., & Wright, J. G. (2003). Delphi as a method to establish consensus for
diagnostic criteria. Journal of Clinical Epidemiology, 56(12), 1150-1156.
Grantcharov, T. P., & Funch-Jensen, P. (2009). Can everyone achieve proficiency with the
laparoscopic technique? Learning curve patterns in technical skills acquisition. Am J
Surg, 197(4), 447-449.
Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention.
Nature, 423(6939), 534-537.
Greenberg, C. C., Regenbogen, S. E., Studdert, D. M., Lipsitz, S. R., Rogers, S. O., Zinner, M.
J., & Gawande, A. A. (2007). Patterns of communication breakdowns resulting in injury
to surgical patients. J Am Coll Surg, 204(4), 533-540.
Gregg, M., Hall, C., & Butler, A. (2010). The MIQ-RS: A Suitable Option for Examining
Movement Imagery Ability. Evid Based Complement Alternat Med, 7(2), 249-257.
Gregg MJ., & T., C. (2007). Theoretical and practical applications of mental imagery. Paper
presented at the International Symposium on Performance Science, Portugal.
Grober, E. D., Hamstra, S. J., Reznick, R. K., Matsumoto, E. D., Sidhu, R. S., & Jarvi, K. A.
(2003). Validation of novel and objective measures of microsurgical skill: Hand-motion
analysis and stereoscopic visual acuity. Microsurgery, 23(4), 317-322.
179
Groenier, M., Schraagen, J. M. C., Miedema, H. A. T., & Broeders, I. A. J. M. (2014). The role
of cognitive abilities in laparoscopic simulator training. Adv Health SciEdu Theory Pract,
19(2), No-Specified.
Guyatt G., Oxman AD., Akl EA., Kunz R., Vist G., Brozek J., Norris S., Falck-Ytter Y.,
Glasziou P., DeBeer H., Jaeschke R., Rind D., Meerpohl J., Dahm P., & Schunemann HJ.
(2011). GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of
findings tables. J Clin Epidemiol, 64(4), 383-394.
Guyatt, G. H., Oxman, A. D., Kunz, R., Vist, G. E., Falck-Ytter, Y., & Schunemann, H. J.
(2008). What is "quality of evidence" and why is it important to clinicians? Bmj,
336(7651), 995-998.
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., &
Schunemann, H. J. (2008). GRADE: an emerging consensus on rating quality of evidence
and strength of recommendations. Bmj, 336(7650), 924-926.
Hall, C. R., Mack, D., Paivio, A., & Hausenblas, H. A. (1998). Imagery use by athletes:
Development of the Sport Imagery Questionnaire. International Journal of Sport
Psychology, 29(1), 73-89.
Harrington, D. T., Roye, G. D., Ryder, B. A., Miner, T. J., Richardson, P., & Cioffi, W. G.
(2007). A time-cost analysis of teaching a laparoscopic entero-enterostomy. J Surg Educ,
64(6), 342-345.
Hassan, I., Gerdes, B., Koller, M., Dick, B., Hellwig, D., Rothmund, M., & Zielke, A. (2007).
Spatial perception predicts laparoscopic skills on virtual reality laparoscopy simulator.
Childs Nerv Syst, 23(6), 685-689.
Hedman, L., Klingberg, T., Enochsson, L., Kjellin, A., & Fellander-Tsai, L. (2007). Visual
working memory influences the performance in virtual image-guided surgical
intervention. Surg Endosc, 21(8), 2044-2050.
180
Hedman, L., Strom, P., Andersson, P., Kjellin, A., Wredmark, T., & Fellander-Tsai, L. (2006).
High-level visual-spatial ability for novices correlates with performance in a visual-
spatial complex surgical simulator task. Surg Endosc, 20(8), 1275-1280.
Hislop, S. J., Hsu, J. H., Narins, C. R., Gillespie, B. T., Jain, R. A., Schippert, D. W., Almudevar,
A. L., & Illig, K. A. (2006). Simulator assessment of innate endovascular aptitude versus
empirically correct performance. J Vasc Surg, 43(1), 47-55.
Hoffer, M. M., & Hsu, S. C. (1990). Hand function in selection of orthopedics residents. Acad
Med, 65(10), 661.
Holdsworth, R. F. (1988). Objective assessment-the state of the art. Annals of the Royal College
of Surgeons of England, 70, 266-270.
Holmes, P. S., & Collins, D. J. (2001). The PETTLEP Approach to Motor Imagery: A Functional
Equivalence Model for Sport Psychologists. Journal of Applied Sport Psychology, 13(1),
60-83.
Jairath, N., & Weinstein, J. (1994). The Delphi methodology (Part one): A useful administrative
approach. Can J Nurs Adm, 7(3), 29-42.
Johnson, D. B., Kondraske, G. V., Wilhelm, D. M., Jacomides, L., Ogan, K., Pearle, M. S., &
Cadeddu, J. A. (2004). Assessment of basic human performance resources predicts the
performance of virtual ureterorenoscopy. J Urol, 171(1), 80-84.
Ju, R., Chang, P. L., Buckley, A. P., & Wang, K. C. (2012). Comparison of Nintendo Wii and
PlayStation2 for enhancing laparoscopic skills. Jsls, 16(4), 612-618.
Keehner, M., Lippa, Y., Montello, D. R., Tendick, F., & Hegarty, M. (2006). Learning a Spatial
Skill for Surgery: How the Contributions of Abilities Change With Practice. Appl.
Cognit. Psychol., 20(4), 487-503.
Keehner, M. M., Tendick, F., Meng, M. V., Anwar, H. P., Hegarty, M., Stoller, M. L., & Duh, Q.
Y. (2004). Spatial ability, experience, and skill in laparoscopic surgery. Am J Surg,
188(1), 71-75.
181
Kelly, S. J., Young, R., Sweeting, H., Fischer, J. E., & West, P. (2008). Levels and confounders
of morning cortisol collected from adolescents in a naturalistic (school) setting.
Psychoneuroendocrinology, 33(9), 1257-1268.
Kenny, S., McInnes, M., & Singh, V. (2013). Associations between residency selection strategies
and doctor performance: a meta-analysis. Med Educ, 47(8), 790-800.
Khatuja, R., Jain, G., Mehta, S., Arora, N., Juneja, A., & Goel, N. (2014). Changing trends in use
of laparoscopy: a clinical audit. Minim Invasive Surg, 2014, 562785.
Kirschbaum, C., Wolf, O. T., May, M., Wippich, W., & Hellhammer, D. H. (1996). Stress- and
treatment-induced elevations of cortisol levels associated with impaired declarative
memory in healthy adults. Life Sci, 58(17), 1475-1483.
Kolozsvari, N. O., Andalib, A., Kaneva, P., Cao, J., Vassiliou, M. C., Fried, G. M., & Feldman,
L. S. (2011). Sex is not everything: the role of gender in early performance of a
fundamental laparoscopic skill. Surg Endosc, 25(4), 1037-1042.
Kothari, S. N., & Ponce, J. (2014). Issues With "Issues in General Surgery Residency Training-
2012". Ann Surg, 261(4), p e113
.
Kroft, J., Ordon, M., Arthur, R., & Pittini, R. (2012). Does surgical "warming up" improve
laparoscopic simulator performance? Simul Healthc, 7(6), 339-342.
Kwon, Y. H., Kwon, J. W., & Lee, M. H. (2015). Effectiveness of motor sequential learning
according to practice schedules in healthy adults; distributed practice versus massed
practice. J Phys Ther Sci, 27(3), 769-772.
Lafayette Instrument evaluation Dexterity Tests. (2015). Retrieved Jan 22, 2015, from
http://lafayetteevaluation.com/product_list.asp?catid=40
Lee, J. Y., Kerbl, D. C., McDougall, E. M., & Mucksavage, P. (2012). Medical students pursuing
surgical fields have no greater innate motor dexterity than those pursuing nonsurgical
fields. 69(3), 360-363.
182
Lee, J. Y., Mucksavage, P., Kerbl, D. C., Osann, K. E., Winfield, H. N., Kahol, K., &
McDougall, E. M. (2012). Laparoscopic warm-up exercises improve performance of
senior-level trainees during laparoscopic renal surgery. J Endourol, 26(5), 545-550.
Lee, T. D., & Genovese, E. D. (1989). Distribution of practice in motor skill acquisition:
different effects for discrete and continuous tasks. Res Q Exerc Sport, 60(1), 59-65.
Levine, W. N., & Spang, R. C., 3rd. (2014). ACGME Duty Hour Requirements: Perceptions and
Impact on Resident Training and Patient Care. J Am Acad Orthop Surg, 22(9), 535-544.
Levitt, C. A., & Klein, M. C. (1991). The Demise of the Rotating Internship and Family Practice
Program Expansion: View from the trenches. Canadian Family Physician, 37, 604-742.
Livingston, S. A., & Ziesky, M. J. (1982). A manual for setting standards of performance on
educational and occupational tests.: Educational testing service. Retrieved October 19,
2015, from https://http://www.ets.org/Media/Research/pdf/passing_scores.pdf
Loeffen, E. A., Mulder, R. L., Kremer, L. C., Michiels, E. M. C., Abbink, F. C. H., Ball, L. M.,
Segers, H., Mavinkurve-Groothuis, A. M. C., Smit, F. J., Vonk, I. J. M., Vd Wetering, M.
D., & Tissing, W. J. (2015). Development of clinical practice guidelines for supportive
care in childhood cancer-prioritization of topics using a Delphi approach. Support Care
Cancer, 23(7), 1987-1995.
Lokuge, S., Fruci, V., Abraham T., Youash S., Bates S., Leyland, N. (2012). Does Innate Motor
Skill Predict Medical Student's Interest in Pursuing a Surgical Specialty? The Journal of
Minimally Invasive Gynecology, 19(6), S25.
Louridas, M., Szasz, P., de Montbrun, S., Harris, K. A., & Grantcharov, T. (2016). International
Assessment Practices Along The Continuum of Surgical Training. Am J Surg.
Louridas, M., Szasz, P., de Montbrun, S., Harris, K. A., & Grantcharov, T. P. (206). Can We
Predict Technical Aptitude? A Systematic Review. Ann Surg, 263(4), 673-691.
Luursema, J.-M., Buzink, S. N., Verwey, W. B., & Jakimowicz, J. J. (2010). Visuo-spatial ability
in colonoscopy simulator training. Adv Health Sci Educ Theory Pract, 15(5), 685-694.
183
Lyk-Jensen, H. T., Jepsen, R. M., Spanager, L., Dieckmann, P., & Ostergaard, D. (2014).
Assessing Nurse Anaesthetists' Non-Technical Skills in the operating room. Acta
Anaesthesiol Scand, 58(7), 794-801.
Maan, Z. N., Maan, I. N., Darzi, A. W., & Aggarwal, R. (2012). Systematic review of predictors
of surgical performance. Br J Surg, 99(12), 1610-1621.
Macario, A. (2010). What does one minute of operating room time cost? J Clin Anesth, 22(4),
233-236.
Mackay, S., Morgan, P., Datta, V., Chang, A., & Darzi, A. (2002). Practice distribution in
procedural skills training: a randomized controlled trial. Surg Endosc, 16(6), 957-961.
Macmillan, A. I., & Cuschieri, A. (1999). Assessment of innate ability and skills for endoscopic
manipulations by the Advanced Dundee Endoscopic Psychomotor Tester: predictive and
concurrent validity. Am J Surg, 177(3), 274-277.
Madan, A. K., Frantzides, C. T., Park, W. C., Tebbit, C. L., Kumari, N. V. A., & O'Leary, P. J.
(2005). Predicting baseline laparoscopic surgery skills. Surg Endosc, 19(1), 101-104.
Madan, A. K., Harper, J. L., Frantzides, C. T., & Tichansky, D. S. (2008). Nonsurgical skills do
not predict baseline scores in inanimate box or virtual-reality trainers. Surg Endosc,
22(7), 1686-1689.
Marteau, T. M., & Bekker, H. (1992). The development of a six-item short-form of the state
scale of the Spielberger State-Trait Anxiety Inventory (STAI). Br J Clin Psychol, 31(Pt
31), 301-306.
Martin, J. A., Regehr, G., Reznick, R., MacRae, H., Murnaghan, J., Hutchison, C., & Brown, M.
(1997). Objective structured assessment of technical skill (OSATS) for surgical residents.
British Journal of Surgery, 84(2), 273-278.
Masud, D., Undre, S., & Darzi, A. (2012). Using manual dexterity to predict the quality of the
final product in the small bowel anastomosis after a period of training. Am J Surg, 203(6),
776-781.
184
Matsumoto, E. D., Kondraske, G. V., Ogan, K., Jacomides, L., Wilhelm, D. M., Pearle, M. S., &
Cadeddu, J. A. (2006). Assessment of basic human performance resources predicts
performance of ureteroscopy. Am J Surg, 191(6), 817-820.
Mattar, S. G., Alseidi, A. A., Jones, D. B., Jeyarajah, D. R., Swanstrom, L. L., Aye, R. W.,
Wexner, S. D., Martinez, J. M., Ross, S. B., Ross, S. B., Awad, M. M., Franklin, M. E.,
Arregui, M. E., M.E., A., Schirmer, B. D., & Minter, R. M. (2013). General surgery
residency inadequately prepares trainees for fellowship: results of a survey of fellowship
program directors. Ann Surg, 258(3), 440-449.
Mayberry, P., & Carey, N. (1997). The Effect of Aptitude and Experience on Mechanical Job
Performance. Educational and Psychological Measurement, 57(1), 131-149.
McClusky, D. A., 3rd, Ritter, E. M., Lederman, A. B., Gallagher, A. G., & Smith, C. D. (2005).
Correlation between perceptual, visuo-spatial, and psychomotor aptitude to duration of
training required to reach performance goals on the MIST-VR surgical simulator. Am
Surg, 71(1), 13-11.
McGaghie, W. C., Cohen, E. R., & Wayne, D. B. (2011). Are United States Medical Licensing
Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection
decisions? Acad Med, 86(1), 48-52.
Medical School Admission Requirement Dalhousie (2015). Retrieved Jan 22, 2015, from
http://medicine.dal.ca/departments/core-units/admissions/admissions.html
Merril, R. M. (2015). Statistical Methods in Epidemiologic Research. Massachusetts: Jones and
Bartlett Learning.
Mitchell, L., Flin, R., Yule, S., Mitchell, J., Coutts, K., & Youngson, G. (2013). Development of
a behavioural marker system for scrub practitioners' non-technical skills (SPLINTS
system). J Eval Clin Pract, 19(2), 317-323.
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2010). Preferred reporting items for
systematic reviews and meta-analyses: the PRISMA statement. Int J Surg, 8(5), 336-341.
185
Moore, E. J., Price, D. L., Van Abel, K. M., & Carlson, M. L. (2014). Still under the microscope:
Can a surgical aptitude test predict otolaryngology resident performance? The
Laryngoscope, 125(2), E57-61.
Moulton, C. A., Dubrowski, A., Macrae, H., Graham, B., Grober, E., & Reznick, R. (2006).
Teaching surgical skills: what kind of practice makes perfect?: a randomized, controlled
trial. Ann Surg, 244(3), 400-409.
Murdoch, J. R., Bainbridge, L. C., Fisher, S. G., & Webster, M. H. (1994). Can a simple test of
visual-motor skill predict the performance of microsurgeons? J R Coll Surg Edinb, 39(3),
150-152.
Murphy, M. K., Black, N. A., Lamping, D. L., McKee, C. M., Sanderson, C. F. B., Askham, J.,
& Marteau, T. (1998). Consensus development methods, and their use in clinical
guideline development. Health Technol Assessment, 2(1), 1-88.
Neumann, M., Meining, A., Buerschaper, C., Reingruber, B., Rosch, T., Hohenberger, W., &
Schneider, I. (2005). Training in GI endoscopy: can we objectively predict the
performance of trainees? A prospective correlation study. Z Gastroenterol, 43(5), 445-
450.
Nomura, T., Miyashita, M., Shrestha, S., Makino, H., Nakamura, Y., Aso, R., Yoshimura, A.,
Shimura, T., Akira, S., & Tajiri, T. (2008). Can interview prior to laparoscopic simulator
training predict a trainee's skills? Surg Educ, 65(5), 335-339.
Norcini, J. J. (2003). Setting standards on educational tests. Med Educ, 37(5), 464-469.
NRMP. (2016). The American National Resident Matching Program Retrieved January 5, 2016,
from http://www.nrmp.org/residency/
Nugent, E., Hseino, H., Boyle, E., Mehigan, B., Ryan, K., Traynor, O., & Neary, P. (2012).
Assessment of the role of aptitude in the acquisition of advanced laparoscopic surgical
skill sets: results from a virtual reality-based laparoscopic colectomy training programme.
Int J Colorectal Dis, 27(9), 1207-1214.
186
Offner, P. J., Hawkes, A., Madayag, R., Seale, F., & Maines, C. (2003). General surgery
residents improve efficiency but not outcome of trauma care. J Trauma, 55(1), 14-19.
Paivio, A. (1985). Cognitive and motivational functions of imagery in human performance. Can
J Appl Sport Sci, 10(4), 22s-28s.
Palter, V. N., & Grantcharov, T. P. (2012). Development and validation of a comprehensive
curriculum to teach an advanced minimally invasive procedure: a randomized controlled
trial. Ann Surg, 256(1), 25-32.
Palter, V. N., & Grantcharov, T. P. (2014). Individualized deliberate practice on a virtual reality
simulator improves technical performance of surgical novices in the operating room: a
randomized controlled trial. Ann Surg, 259(3), 443-448.
Palter, V. N., Orzech, N., Reznick, R. K., & Grantcharov, T. P. (2013). Validation of a structured
training and assessment curriculum for technical skill acquisition in minimally invasive
surgery: a randomized controlled trial. Ann Surg, 257(2), 224-230.
Panait, L., Larios, J. M., Brenes, R. A., Fancher, T. T., Ajemian, M. S., Dudrick, S. J., &
Sanchez, J. A. (2011). Surgical skills assessment of applicants to general surgery
residency. Journal of Surgical Research, 170(2), 189-194.
Paschold, M., Schroder, M., Kauff, D. W., Gorbauch, T., Herzer, M., Lang, H., & Kneist, W.
(2011). Virtual reality laparoscopy: which potential trainee starts with a higher
proficiency level? Int J Comput Assist Radiol Surg, 6(5), 653-662.
Peters, J. H., Fried, G. M., Swanstrom, L. L., Soper, N. J., Sillin, L. F., Schirmer, B., & Hoffman,
K. (2004). Development and validation of a comprehensive program of education and
assessment of the basic fundamentals of laparoscopic surgery. Surgery, 135(1), 21-27.
Powell, C. (2002). The Delphi technique: myths and realities. Journal of Advanced Nursing,
41(4), 376-382.
RAND, C. (1976). Delphi Method. Retrieved January 21, 2015, from
http://www.rand.org/topics/delphi-method.html
187
Richards, M. K., McAteer, J. P., Drake, F. T., Goldin, A. B., Khandelwal, S., & Gow, K. W.
(2015). A National Review of the Frequency of Minimally Invasive Surgery Among
General Surgery Residents: Assessment of ACGME Case Logs During 2 Decades of
General Surgery Resident Training. JAMA Surg, 150(2), 169-172.
Risucci, D., Geiss, A., Gellman, L., Pinard, B., & Rosser, J. (2001). Surgeon-specific factors in
the acquisition of laparoscopic surgical skills. Am J Surg, 181(4), 289-293.
Risucci, D. A. (2002). Visual spatial perception and surgical competence. Am Surg, 184(3), 291-
295.
Ritter, E. M., McClusky, D. A., 3rd, Gallagher, A. G., Enochsson, L., & Smith, C. D. (2006).
Perceptual, visuospatial, and psychomotor abilities correlate with duration of training
required on a virtual-reality flexible endoscopy simulator. Am J Surg, 192(3), 379-384.
Ritter, E. M., & Scott, D. J. (2007). Design of a proficiency-based skills training curriculum for
the fundamentals of laparoscopic surgery. Surg Innov, 14(2), 107-112.
Ronai, A. K., Golmon, M. E., Shanks, C. A., Schafer, M. F., & Brunner, E. A. (1984).
Relationship between past academic performance and results of specialty in-training
examinations. 59, 4(341-4).
Rosser, J. C., Jr., Lynch, P. J., Cuddihy, L., Gentile, D. A., Klonsky, J., & Merrell, R. (2007).
The impact of video games on training surgeons in the 21st century. Arch Surg, 142(2),
181-186.
Royal College of Surgeons of England Royal College of Surgeons of England. (2013).
Intercollegiate surgical curriculum programme. Retrieved December 29, 2013, from
https://http://www.iscp.ac.uk
Sanfey, H., Hollands, C., & Gantt, N. L. (2013). Strategies for building an effective mentoring
relationship. Am J Surg, 206(5), 714-718.
Sasor, S. E., Flores, R. L., Wooden, W. A., & Tholpady, S. (2013). The cost of intraoperative
plastic surgery education. J Surg Educ, 70(5), 655-659.
188
Schijven, M. P., & Jakimowicz, J. (2004). The learning curve on the Xitact LS 500 laparoscopy
simulator: profiles of performance. Surg Endosc, 18(1), 121-127.
Schijven, M. P., Jakimowicz, J. J., & Carter, F. J. (2004). How to select aspirant laparoscopic
surgical trainees: establishing concurrent validity comparing Xitact LS500 index
performance scores with standardized psychomotor aptitude test battery scores. J Surg
Res, 121(1), 112-119.
Schmitz, C. C., DaRosa, D., Sullivan, M. E., Meyerson, S., Yoshida, K., & Korndorffer, J. R., Jr.
(2014). Development and verification of a taxonomy of assessment metrics for surgical
technical skills. Acad Med, 89(1), 153-161.
Schueneman, A. L., Pickleman, J., & Freeark, R. J. (1985). Age, gender, lateral dominance, and
prediction of operative skill among general surgery residents. Surgery, 98(3), 506-515.
Schueneman, A. L., Pickleman, J., Hesslein, R., & Freeark, R. J. (1984). Neuropsychologic
predictors of operative skill among general surgery residents. Surgery, 96(2), 288-295.
Scott, D. J., Bergen, P. C., Rege, R. V., Laycock, R., Tesfay, S. T., Valentine, R. J., Euhus, D.
M., Jeyarajah, D. R., Thompson, W. M., & Jones, D. B. (2000). Laparoscopic training on
bench models: better and more cost effective than operating room experience? J Am Coll
Surg, 191(3), 272-283.
Scott, D. J., & Dunnington, G. L. (2008). The new ACS/APDS Skills Curriculum: moving the
learning curve out of the operating room. J Gastrointest Surg, 12(2), 213-221.
Sevdalis, N., Moran, A., & Arora, S. (2013). Methodological issues surrounding 'The mind's
scalpel in surgical education: a randomised controlled trial of mental imagery'. Bjog,
120(6), 775-776.
Shah, J., Buckley, D., Frisby, J., & Darzi, A. (2003). Reaction time does not predict surgical
skill. Br J Surg, 90(10), 1285-1286.
Shah, J., Paul, I., Buckley, D., Davis, H., Frisby, J. P., & Darzi, A. (2003). Can tonic
accommodation predict surgical performance? Surg Endosc, 17(5), 787-790.
189
Shetty, S., Zevin, B., Grantcharov, T. P., Roberts, K. E., & Duffy, A. J. (2014). Perceptions,
training experiences, and preferences of surgical residents toward laparoscopic simulation
training: a resident survey. J Surg Educ, 71(5), 727-733.
Short, S. E., Tenute, A., & Feltz, D. L. (2005). Imagery use in sport: mediational effects for
efficacy. J Sports Sci, 23(9), 951-960.
Sir Alfred, C. (2003). Lest we forget the surgeon. Semin Laparosc Surg, 10(3), 141-148.
Soper, N. J., Stockmann, P. T., Dunnegan, D. L., & Ashley, S. W. (1992). Laparoscopic
cholecystectomy. The new 'gold standard'? Arch Surg, 127(8), 917-921; discussion 921-
923.
Steele, R. J., Walder, C., & Herbert, M. (1992). Psychomotor testing and the ability to perform
an anastomosis in junior surgical trainees. Br J Surg, 79(10), 1065-1067.
Stefanidis, D., Haluck, R., Pham, T., Dunne, J. B., Reinke, T., Markley, S., Korndorffer, J. R.,
Jr., Arellano, P., Jones, D. B., & Scott, D. J. (2007). Construct and face validity and task
workload for laparoscopic camera navigation: virtual reality versus videotrainer systems
at the SAGES Learning Center. Surg Endosc, 21(7), 1158-1164.
Stefanidis, D., Korndorffer Jr, J. R., Black, F. W., Dunne, J. B., Sierra, R., Touchard, C. L., Rice,
D. A., Markert, R. J., Kastl, P. R., & Scott, D. J. (2006). Psychomotor testing predicts
rate of skill acquisition for proficiency-based laparoscopic skills training. Surgery,
140(2), 252-262.
Stefanidis, D., Korndorffer, J. R., Jr., Black, F. W., Dunne, J. B., Sierra, R., Touchard, C. L.,
Rice, D. A., Markert, R. J., Kastl, P. R., & Scott, D. J. (2006). Psychomotor testing
predicts rate of skill acquisition for proficiency-based laparoscopic skills training.
Surgery, 140(2), 252-262.
Stefanidis, D., Scott, D. J., & Korndorffer, J. R., Jr. (2009). Do metrics matter? Time versus
motion tracking for performance assessment of proficiency-based laparoscopic skills
training. Simul Healthc, 4(2), 104-108.
190
Straus, S. E., Johnson, M. O., Marquez, C., & Feldman, M. D. (2013). Characteristics of
successful and failed mentoring relationships: a qualitative study across two academic
health centers. Acad Med, 88(1), 82-89.
Sturmberg, J. P., & Hinchy, J. (2010). Borderline competence--from a complexity perspective:
conceptualization and implementation for certifying examinations. J Eval Clin Pract,
16(4), 867-872.
Surgeons, R. A. C. o. (16/01/2014). Selection into Surgical Education and Training (SET).
Retrieved 23/11/2014
Surgeons, R. C. o. P. a. (2015). Program Direcors. Retrieved January 4, 2016, from
http://www.royalcollege.ca/portal/page/portal/rc/credentials/accreditation/program_direct
ors_handbook
Szasz, P., Louridas, M., Harris, K. A., Aggarwal, R., & Grantcharov, T. P. (2015). Assessing
Technical Competence in Surgical Trainees: A Systematic Review. Ann Surg, 261(6),
1045-1055.
Taguchi, Y., Komatsu, S., Sakamoto, E., Norimizu, S., Shingu, Y., & Hasegawa, H. (2016).
Laparoscopic versus open surgery for complicated appendicitis in adults: a randomized
controlled trial. Surg Endosc, 30(5), 1705-1712.
Tan, P.-N., Steinbach, M., & Kumar, V. (2006). Introduction to Data Mining Retrieved from
http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf
Tang, C. G., Hilsinger, R. L., Jr., Cruz, R. M., Schloegel, L. J., Byl, F. M., Jr., & Rasgon, B. M.
(2014). Manual dexterity aptitude testing: a soap carving study. JAMA Otolaryngol Head
Neck Surg, 140(3), 243-249.
Tangchitnob, E. P., Solnik, J. M., Saad, C., Rad, S., & Ogunyemi, D. (2011). Identification of
background factors that predict baseline laparoscopic skill in 3rd year medical students.
Journal of Minimally Invasive Gynecology, 18, S91-S113.
The Air Force. (2013). Air Force Officer Qualifying Test (AFOQT). December 29, 2013, from
http://m.airforce.com/joining-the-air-force/officer/qualifying-test
191
United States Coast Guard. (2013). Aviation Selection Test Battery. December 29, 2013, from
http://www.uscg.mil/hq/capemay/Education/astb.asp
van Dongen, K. W., Tournoij, E., van der Zee, D. C., Schijven, M. P., & Broeders, I. A. (2007).
Construct validity of the LapSim: can the LapSim virtual reality simulator distinguish
between novices and experts? Surg Endosc, 21(8), 1413-1417.
Van Herzeele, I., O'Donoghue, K. G. L., Aggarwal, R., Vermassen, F., Darzi, A., & Cheshire, N.
J. W. (2010). Visuospatial and psychomotor aptitude predicts endovascular performance
of inexperienced individuals on a virtual reality simulator. J Vasc Surg, 51(4), 1035-
1042.
Van Hove, C., Perry, K. A., Spight, D. H., Wheeler-Mcinvaille, K., Diggs, B. S., Sheppard, B.
C., Jobe, B. A., & O'Rourke, R. W. (2008). Predictors of technical skill acquisition
among resident trainees in a laparoscopic skills education program. World J Surg, 32(9),
1917-1921.
Vassiliou, M. C., Ghitulescu, G. A., Feldman, L. S., Stanbridge, D., Leffondre, K., Sigman, H.
H., & Fried, G. M. (2006). The MISTELS program to measure technical skill in
laparoscopic surgery : evidence for reliability. Surg Endosc, 20(5), 744-747.
Wadey, R., & Hanton, S. (2008). Basic psychological skills usage and competitive anxiety
responses: perceived underlying mechanisms. Res Q Exerc Sport, 79(3), 363-373.
Wanzel, K. R., Hamstra, S. J., Anastakis, D. J., Matsumoto, E. D., & Cusimano, M. D. (2002).
Effect of visual-spatial ability on learning of spatially-complex surgical skills. Lancet,
359(9302), 230-231.
Wanzel, K. R., Hamstra, S. J., Caminiti, M. F., Anastakis, D. J., Grober, E. D., & Reznick, R. K.
(2003). Visual-spatial ability correlates with efficiency of hand motion and successful
surgical performance. Surgery, 134(5), 750-757.
Watson, D. I., Mathew, G., & Williams, J. A. (1995). Impact of laparoscopic cholecystectomy in
a major teaching hospital: clinical and hospital outcomes. Med J Aust, 163(10), 527-530.
192
Weinberg RS., Seaboume T., & Jackson A. (1992). The effect of mental practice on diving
performance. Int J Sport Psychol, 23, 60-69.
Westman, B., Ritter, E. M., Kjellin, A., Torkvist, L., Wredmark, T., Fellander-Tsai, L., &
Enochsson, L. (2006). Visuospatial abilities correlate with performance of senior
endoscopy specialist in simulated colonoscopy. J Gastrointest Surg, 10(4), 593-599.
White, M. T., & Welch, K. (2012). Does gender predict performance of novices undergoing
Fundamentals of Laparoscopic Surgery (FLS) training? Am J Surg, 203(3), 397-400.
Wickham, J. A., & Dilworth, R. L. (1987). Alternate flight aptitude selection test (AFAST) In D.
o. t. Army (Ed.), (Vol. 611–256–2, pp. 1-15).
Williams, J. A., & Harris, D. V. (2006). Relaxation and energizing techniques for regulating of
arousal. New York: Mayfield.
Willis, R. E., & Van Sickle, K. R. (2015). Current Status of Simulation-Based Training in
Graduate Medical Education. Surg Clin North Am, 95(4), 767-779.
Yule, S., Flin, R., Maran, N., Rowley, D., Youngson, G., & Paterson-Brown, S. (2008).
Surgeons' non-technical skills in the operating room: reliability testing of the NOTSS
behavior rating system. World J Surg, 32(4), 548-556.
Zevin B, Dedy NJ, Bonrath EM, & TP., G. (2013). Development and validation of a
comprehensive simulation-enhanced training curriculum in bariatric surgery. Paper
presented at the American College of Surgeons Annual Meeting, Washington D.C.
Zevin, B., Aggarwal, R., & Grantcharov, T. P. (2012). Simulation-based training and learning
curves in laparoscopic Roux-en-Y gastric bypass. Br J Surg, 99(7), 887-895.
Zevin, B., Levy Js Fau - Satava, R. M., Satava Rm Fau - Grantcharov, T. P., & Grantcharov, T.
P. A consensus-based framework for design, validation, and implementation of
simulation-based training curricula in surgery. (1879-1190 (Electronic)).
193
Zevin, B., Levy, J. S., Satava, R. M., & Grantcharov, T. P. (2012). A Consensus-Based
Framework for Design, Validation, and Implementation of Simulation-Based Training
Curricula in Surgery. J Am Coll Surg, 215(4), 580-586.e583.
