Development and Development and Implementation of a Log-Implementation of a Log-based Assessment Tool based Assessment Tool for Clinical Clerkshipfor Clinical Clerkship
Susan Chamberlain, MD, FRCSCSusan Chamberlain, MD, FRCSC
Phil Hahn, MScPhil Hahn, MSc
Dept of Obstetrics & GynaecologyDept of Obstetrics & Gynaecology
Health Sciences Education RoundsHealth Sciences Education Rounds
May 21, 2009May 21, 2009
ObjectivesObjectives
Describe log system linked to Describe log system linked to assessment and its rationaleassessment and its rationale
Present results of survey of Present results of survey of Meds2009 regarding their Meds2009 regarding their attitudes towards their attitudes towards their clinical logs (quantitative and clinical logs (quantitative and qualitative data)qualitative data)
BackgroundBackground
Problem # 1Problem # 1: :
ED-2 requires students in clerkship to ED-2 requires students in clerkship to log clinical experiences but student log clinical experiences but student participation is lowparticipation is low Students are not altruisticStudents are not altruistic Logs are incomplete and inaccurateLogs are incomplete and inaccurate
If results will be used to direct If results will be used to direct courses, they must be accuratecourses, they must be accurate
Students will only provide accurate Students will only provide accurate information, if there is clear information, if there is clear individual benefitindividual benefit
BackgroundBackground
Problem # 2Problem # 2: :
Ob/Gyn end of rotation SAQ examOb/Gyn end of rotation SAQ exam Out of dateOut of date Time consuming to mark and Time consuming to mark and maintainmaintain
Inconsistency in markingInconsistency in marking
BackgroundBackground
January 2006 (Meds 2007) – new January 2006 (Meds 2007) – new system of evaluation linked to system of evaluation linked to clinical log startedclinical log started Each logged encounter corresponded to Each logged encounter corresponded to LMCC clinical presentationLMCC clinical presentation
Each covered by clerkship seminarEach covered by clerkship seminar Students logged patient encounters, Students logged patient encounters, seminars attendedseminars attended
Exam question chosen based on clinical Exam question chosen based on clinical log (i.e. students only examined in SOE log (i.e. students only examined in SOE on what they had seen)on what they had seen)
BackgroundBackground
MCQ lengthenedMCQ lengthened SAQ replaced by Structured Oral SAQ replaced by Structured Oral examination (SOE)examination (SOE) 50 mark questions focusing on data gathering 50 mark questions focusing on data gathering (history, physical and choice of (history, physical and choice of investigation, plus basic management choices)investigation, plus basic management choices)
Each exam question written by attending staff Each exam question written by attending staff who gives the seminar (all reviewed/edited by who gives the seminar (all reviewed/edited by course chair for consistency)course chair for consistency)
Student answers 2 questions in 30 minute examStudent answers 2 questions in 30 minute exam
Survey on Student Survey on Student AttitudesAttitudesMarch 2008March 2008 Survey of class of meds 2009, blocks Survey of class of meds 2009, blocks 2-8 on use of logs, attitudes2-8 on use of logs, attitudes
Student asked to complete online Student asked to complete online questionnaire after 2 clerkship questionnaire after 2 clerkship rotations: ob/gyn and surgeryrotations: ob/gyn and surgery
Confidentiality of responses assured Confidentiality of responses assured by code to identify responsesby code to identify responses
74 student surveyed74 student surveyed
Survey ResultsSurvey Results 34 Clerks filled out questionnaire 34 Clerks filled out questionnaire after one of the rotations (OB or after one of the rotations (OB or Surgery)Surgery)
19 Clerks filled out questionnaire 19 Clerks filled out questionnaire after both rotations (paired data)after both rotations (paired data) Responses after first rotation (either Responses after first rotation (either OB or Surgery) were included in the OB or Surgery) were included in the analysisanalysis
Survey ResultsSurvey Results
CodesCodes One duplicate One duplicate (0023f91973) (0023f91973) (Block 2 OB and Block 3 Surgery)(Block 2 OB and Block 3 Surgery)
One with no match One with no match (a0f3d5c663)(a0f3d5c663)
These two responses were not These two responses were not included in analysisincluded in analysis
Survey ResultsSurvey Results 53 Clerks53 Clerks Two independent groupsTwo independent groups
29 OB rotation29 OB rotation 24 Surgery rotation24 Surgery rotation
Q1. I completed my log:Q1. I completed my log:
After each patientAfter each patient DailyDaily 1-2 times/week1-2 times/week Less than once/weekLess than once/week
Q1. I completed my log:Q1. I completed my log:
ResponseResponseRotationRotation
OB * OB * (n=28)(n=28)
SurgerySurgery(n=24)(n=24)
After each patient After each patient encounterencounter
0 (0)0 (0) 0 (0)0 (0)
DailyDaily 2 (7.1)2 (7.1) 0 (0)0 (0)
1-2 times per week1-2 times per week 14 (50.0)14 (50.0) 5 (20.8)5 (20.8)
Less than once per Less than once per weekweek
12 (42.9)12 (42.9) 19 (79.2)19 (79.2)* One response (both 1-2 times per week and less than once per week) not included
Values = n (%)
Q1. I completed my log:Q1. I completed my log:
ResponseResponseRotationRotation
OB OB (n=28)(n=28)
SurgerySurgery(n=24)(n=24)
After each patient After each patient encounterencounter
0 (0)0 (0) 0 (0)0 (0)
DailyDaily 2 (7.1)2 (7.1) 0 (0)0 (0)
1-2 times per week1-2 times per week 14 (50.0)14 (50.0) 5 (20.8)5 (20.8)
Less than once per Less than once per weekweek
12 (42.9)12 (42.9) 19 (79.2)19 (79.2)For analysis, “daily” and “1-2 times per week” categories were pooled.
Values = n (%)
Q1. I completed my log:Q1. I completed my log:
ResponseResponseRotationRotation
OB OB (n=28)(n=28)
SurgerySurgery(n=24)(n=24)
1-2 times per week / 1-2 times per week / dailydaily
16 (57.1)16 (57.1) 5 (20.8)5 (20.8)
Less than once per Less than once per weekweek
12 (42.9)12 (42.9) 19 (79.2)19 (79.2)Values = n (%)
P = 0.011, Fisher’s exact test (statistically significant)
Clerks logged more frequently during the OB rotation.
Q2. I found logging useful to Q2. I found logging useful to my learning.my learning.
YesYes NoNo
Q2. I found logging useful to Q2. I found logging useful to my learning.my learning.
OB: 20.7% (6/29)OB: 20.7% (6/29) Surgery: 12.5% (3/24)Surgery: 12.5% (3/24)
Difference = 8.2%Difference = 8.2% P = 0.487, Fisher’s exact test P = 0.487, Fisher’s exact test
Q3. I logged some clinical Q3. I logged some clinical encounters that I encounters that I did notdid not see.see.
YesYes NoNo
Q3. I logged some clinical Q3. I logged some clinical encounters that I encounters that I did notdid not see.see.
OB: 10.3% (3/29)OB: 10.3% (3/29) Surgery: 4.2% (1/24)Surgery: 4.2% (1/24)
Difference = 6.1%Difference = 6.1% P = 0.617, Fisher’s exact test P = 0.617, Fisher’s exact test
Q4. I chose Q4. I chose not to log not to log some some clinical problems that I clinical problems that I did see.did see.
YesYes NoNo
Q4. I chose Q4. I chose not to log not to log some some clinical problems that I clinical problems that I did see.did see.
OB: 62.1% (18/29)OB: 62.1% (18/29) Surgery: 62.5% (15/24)Surgery: 62.5% (15/24)
Difference = 0.4%Difference = 0.4% P = 1.000, Fisher’s exact test P = 1.000, Fisher’s exact test
Q5. After logging clinical Q5. After logging clinical encounter, I did encounter, I did further reading on the topic further reading on the topic ___% of the time___% of the time
Q5: After logging clinical Q5: After logging clinical encounter, I did further encounter, I did further
reading on the topic:reading on the topic:% ofClerks
Response: % of the time.
OB Rotation (n=28), 1 missing
Median (Middle) Response = 15% of the time
Q5: After logging clinical Q5: After logging clinical encounter, I did further encounter, I did further
reading on the topic:reading on the topic:
% ofClerks
Response: % of the time.
Surgery Rotation (n=23), 1 missing
Median Response = 10% of the time
Q5. After logging clinical Q5. After logging clinical encounter, I did encounter, I did
further reading on the further reading on the topic:topic:
Median (% of the time)Median (% of the time) OB: 15% versus Surgery: 10% OB: 15% versus Surgery: 10%
P = 0.453, Mann-Whitney U testP = 0.453, Mann-Whitney U test(nonparametric version of the t test) (nonparametric version of the t test)
Q6: Average time (minutes) Q6: Average time (minutes) spent reading on an clinical spent reading on an clinical encounter encounter
Q6: Average time (minutes) Q6: Average time (minutes) spent reading on an clinical spent reading on an clinical
encounter encounter
% ofClerks
Time (minutes)
OB Rotation (n=28), 1 missing
Median = 30 minutes
Q6: Average time (minutes) Q6: Average time (minutes) spent reading on an clinical spent reading on an clinical
encounter encounter
% ofClerks
Time (minutes)
Surgery Rotation (n=22), 2 missing
Median = 30 minutes
Q6. Average time (minutes) Q6. Average time (minutes) spent reading on an spent reading on an clinical encounter clinical encounter
Median (minutes)Median (minutes) OB: 30 versus Surgery: 30OB: 30 versus Surgery: 30
P = 0.935, Mann-Whitney U testP = 0.935, Mann-Whitney U test(nonparametric version of the t test) (nonparametric version of the t test)
Q7. Logging prompted me to Q7. Logging prompted me to read more about patients read more about patients that I had seen.that I had seen.
YesYes NoNo
Q7. Logging prompted me to Q7. Logging prompted me to read more about patients read more about patients
that I had seen.that I had seen.
OB: 6.9% (2/29)OB: 6.9% (2/29) Surgery: 12.5% (3/24)Surgery: 12.5% (3/24)
Difference = 5.6%Difference = 5.6% P = 0.649, Fisher’s exact test P = 0.649, Fisher’s exact test
Q8. Logging clinical Q8. Logging clinical encounters helped focus my encounters helped focus my studying.studying.
YesYes NoNo
Q8. Logging clinical Q8. Logging clinical encounters helped focus my encounters helped focus my
studying.studying.
OB: 20.7% (6/29)OB: 20.7% (6/29) Surgery: 4.2% (1/24)Surgery: 4.2% (1/24)
Difference = 16.5%Difference = 16.5% P = 0.112, Fisher’s exact test P = 0.112, Fisher’s exact test
SummarySummary
QuestionQuestionObstetricObstetric
ss(n=29)(n=29)
SurgerySurgery(n=24)(n=24)
DifferenceDifference(OB - (OB -
Surgery)Surgery)P valueP value
Q2.Completed log Q2.Completed log 1-2 1-2 times per wk or times per wk or daily daily
16/28 16/28 57.1%57.1%
5 5 20.8%20.8% 36.3%36.3% 0.0110.011
Q3.Logging usefulQ3.Logging useful
6 6 20.7%20.7%
3 3 12.5%12.5% 8.2%8.2% 0.4870.487
Q4.Logged Q4.Logged encounters not encounters not seenseen
3 3 10.3%10.3%
1 1 4.2%4.2% 6.1%6.1% 0.6170.617
Q5.Didn’t log Q5.Didn’t log encounters seenencounters seen
18 18 62.1%62.1%
15 15 62.5%62.5%
(minus)(minus)- 0.4%- 0.4% 1.0001.000
Q8.Read moreQ8.Read more 226.9%6.9%
3312.5%12.5%
(minus)(minus)-5.6%-5.6% 0.6490.649
Q9.Helped focus Q9.Helped focus studiesstudies
6620.7%20.7%
114.2%4.2% 16.5%16.5% 0.1120.112
Fisher’s exact test
SummarySummary
QuestionQuestionObstetricsObstetrics(N=29)(N=29)
SurgerySurgery(N=24)(N=24) P valueP value
Q6.Futher reading Q6.Futher reading (% of the time)(% of the time)
15%15%(0%, 91%)(0%, 91%)
10%10%(0%, 74%)(0%, 74%)
0.4530.453
Q7.Time spent reading Q7.Time spent reading (minutes) (minutes)
3030(0, 120)(0, 120)
3030(0, 111)(0, 111)
0.6490.649
Mann-Whitney U testnonparametric version of the t testValues are medians (5th, 95th percentiles)
Q6. One missing value in each groupQ7. One missing value in OB group and two missing values in surgery group.
Conclusions Conclusions (Quantitative data)(Quantitative data) There is no difference with regards to There is no difference with regards to study use of the log or students’ study use of the log or students’ perception of usefulness of logs as an perception of usefulness of logs as an educational tool between these 2 educational tool between these 2 rotationsrotations
Students on the ob/gyn rotation logged Students on the ob/gyn rotation logged more frequentlymore frequently
A minority of students find that A minority of students find that logging is useful to their learninglogging is useful to their learning
Conclusions – accuracy Conclusions – accuracy of logsof logs Up to 10% of students logged Up to 10% of students logged clinical problems that they did not clinical problems that they did not seesee
62% of students are not logging the 62% of students are not logging the clinical problems that they are clinical problems that they are seeingseeing
Conclusions – log as Conclusions – log as study toolstudy tool Few students use the log as an Few students use the log as an impetus to read around patient impetus to read around patient encountersencounters
Few students find the log helpful as Few students find the log helpful as a focus for their studya focus for their study
Survey Results – Survey Results – qualitative dataqualitative data
Q9: Please comment on how Q9: Please comment on how clinical encounter logs may clinical encounter logs may have contributed to your have contributed to your education in either a positive education in either a positive or negative way.or negative way.
Results – Results – technical/communication technical/communication issuesissues Students find the interface difficult Students find the interface difficult and are frustrated by the limited and are frustrated by the limited choices, what ‘counts’ as a clinical choices, what ‘counts’ as a clinical encounterencounter ““too many drop-down menus”too many drop-down menus” ““topics were not arranged in a useful way”topics were not arranged in a useful way” ““cumbersome”cumbersome” ““... told by staff that you don’t have to ... told by staff that you don’t have to log everything, then get in trouble when log everything, then get in trouble when not everything is logged.”not everything is logged.”
Results – logging is a Results – logging is a ‘make-work project’‘make-work project’ Students consider the log ‘an Students consider the log ‘an administrative task which has administrative task which has no bearing’ on learningno bearing’ on learning
““felt like a tedious waste of time”felt like a tedious waste of time” ““..waste of time that could have been ..waste of time that could have been used studying..”used studying..”
……
Results – Results – Honesty/accuracy of the Honesty/accuracy of the logslogs Unclear thresholds for logging Unclear thresholds for logging of clinical activityof clinical activity
““sometimes difficult to know what counts sometimes difficult to know what counts as having ‘seen’ a clinical problem”as having ‘seen’ a clinical problem”
“…“…I logged...only those patients that I I logged...only those patients that I actually assessed…provides the greatest actually assessed…provides the greatest learning”learning”
Results – Results – honesty/accuracyhonesty/accuracy
Inaccurate recollection Inaccurate recollection
““I was inconsistent in filling logs I was inconsistent in filling logs out, sometimes forgetting for up to a out, sometimes forgetting for up to a week”week”
““I was fairly reliable about logging I was fairly reliable about logging for the first 5 blocks, then completely for the first 5 blocks, then completely forgot about it for the last 3”forgot about it for the last 3”
Results – Results – honesty/accuracy honesty/accuracy Students logged encounters they did Students logged encounters they did not seenot see ““..often completing logs at the conclusion of ..often completing logs at the conclusion of a rotation by simply making up cases that a rotation by simply making up cases that would allow me to cover rotation objectives”would allow me to cover rotation objectives”
“ “ does not ensure anything as it can easily does not ensure anything as it can easily be faked”be faked”
““..students may be tempted to log encounters ..students may be tempted to log encounters they have not experienced to avoid such they have not experienced to avoid such emails”emails”
Results – Results – honesty/accuracyhonesty/accuracy Students did not log encounters Students did not log encounters they did in fact seethey did in fact see
““I did choose to not log a specific I did choose to not log a specific encounter because I did not want to be encounter because I did not want to be asked the question on my oral exam as I asked the question on my oral exam as I did not have the chance to fully read did not have the chance to fully read around the topic”around the topic”
Results – study habitsResults – study habits
Some students found the log useful Some students found the log useful for studying:for studying:
““It does cause me to reflect on particular It does cause me to reflect on particular cases but fitting this extra task in my cases but fitting this extra task in my daily schedule feels like adding one more daily schedule feels like adding one more juggling ball to a performance that exists juggling ball to a performance that exists on the edge of completely falling apart”on the edge of completely falling apart”
““It probably did not increase my reading of It probably did not increase my reading of topics I did encounter, but would likely topics I did encounter, but would likely make me focus on those topics I hadn’t make me focus on those topics I hadn’t encountered”encountered”
Results – study habitsResults – study habits
Most students did not find the Most students did not find the log useful for studying:log useful for studying:
““I read about things that I did not I read about things that I did not understand as they came up, rather than understand as they came up, rather than after I had logged them.”after I had logged them.”
““Logging was not a motivation for me to Logging was not a motivation for me to read more, seeing my patients was.”read more, seeing my patients was.”
Results – Q9Results – Q9
““.. I haven’t found it helpful for identifying .. I haven’t found it helpful for identifying areas of deficiencies. For example I went through areas of deficiencies. For example I went through 6 weeks of surgery (incl urol) without having a 6 weeks of surgery (incl urol) without having a single opportunity to put in a foley catheter”single opportunity to put in a foley catheter”
“… “… after the mid-way meeting and outside of after the mid-way meeting and outside of ob/gyn, no one else seemed to give a toss about ob/gyn, no one else seemed to give a toss about the log, so I didn’t either”the log, so I didn’t either”
““these .. would be more beneficial if they were these .. would be more beneficial if they were emphasized by the surg/obsgyn teams during clerk emphasized by the surg/obsgyn teams during clerk rotations, with some minor teaching involved…rotations, with some minor teaching involved…
Results – positive Results – positive commentscomments Ensured rotation objectives met:Ensured rotation objectives met:
““they direct you to see out certain they direct you to see out certain clinical opportunities that you haven’t clinical opportunities that you haven’t had yet”had yet”
““Did not directly affect my learning for Did not directly affect my learning for the block, but did ensure that I had the block, but did ensure that I had seen all the objectives laid out for the seen all the objectives laid out for the block”block”
Survey Results – Survey Results – qualitative dataqualitative data
Q10: With respect to the OB/GYN Q10: With respect to the OB/GYN rotation, how did knowing that the rotation, how did knowing that the topics for your final oral exam topics for your final oral exam were selected after reviewing your were selected after reviewing your clinical encounter log:clinical encounter log:
A: Influence your logging habits?A: Influence your logging habits? B: Contribute to your overall B: Contribute to your overall learning experience?learning experience?
Survey Results – Survey Results – qualitative dataqualitative data
Q10: With respect to the OB/GYN Q10: With respect to the OB/GYN rotation, how did knowing that rotation, how did knowing that the topics for your final oral the topics for your final oral exam were selected after exam were selected after reviewing your clinical reviewing your clinical encounter log:encounter log:
A: Influence your logging A: Influence your logging habits?habits?
Survey Results – Q10ASurvey Results – Q10A
Some examiners did not comply Some examiners did not comply with guidelines and asked about with guidelines and asked about topics not loggedtopics not logged
Survey results – Q10ASurvey results – Q10A
Obs/gyn students had a higher Obs/gyn students had a higher threshold for loggingthreshold for logging
““It made me be really sure I adequately It made me be really sure I adequately ‘saw’ something before I logged it”‘saw’ something before I logged it”
““.. Didn’t log things only seen briefly .. Didn’t log things only seen briefly that I didn’t want to be tested on”that I didn’t want to be tested on”
Survey results – Q10ASurvey results – Q10A
Some students more Some students more conscientious on obs/gyn conscientious on obs/gyn rotationrotation
““I was more complete in my logs than on I was more complete in my logs than on previous rotations”previous rotations”
“…“…More of an impetus to log the things More of an impetus to log the things I had seen”I had seen”
Survey Results – Survey Results – qualitative dataqualitative data
Q10: With respect to the OB/GYN Q10: With respect to the OB/GYN rotation, how did knowing that rotation, how did knowing that the topics for your final oral the topics for your final oral exam were selected after exam were selected after reviewing your clinical reviewing your clinical encounter log:encounter log:
B: Contribute to your overall B: Contribute to your overall learning experience?learning experience?
Survey results – Q10BSurvey results – Q10B
Focusing/narrowing exam studyFocusing/narrowing exam study Yes –Yes –
helped focus my studying for the oral examhelped focus my studying for the oral exam It was nice to see that the logs were actually being It was nice to see that the logs were actually being usedused
If anything the log-exam relationship gave the student If anything the log-exam relationship gave the student an excuse to not study certain areasan excuse to not study certain areas
No –No – the objectives on the log are too vaguethe objectives on the log are too vague Was surprised at how intense and wide spread the oral Was surprised at how intense and wide spread the oral exam questions are. I think it’s just safest to study exam questions are. I think it’s just safest to study everything:)everything:)
I had to know it all for the MCQ anywayI had to know it all for the MCQ anyway
Lessons learnedLessons learned
Although the students logged more Although the students logged more regularly on ob/gyn they do not regularly on ob/gyn they do not appear to log with more accuracyappear to log with more accuracy
Although students appreciate that Although students appreciate that the log is demonstrably used in some the log is demonstrably used in some way, the link between log and way, the link between log and evaluation does not add any apparent evaluation does not add any apparent educational benefiteducational benefit
Lessons learnedLessons learned
The current system and interface The current system and interface is still problematic and may be is still problematic and may be preventing the log from attaining preventing the log from attaining its potential as a recording toolits potential as a recording tool
Faculty and residents need to be Faculty and residents need to be more aware of the students’ more aware of the students’ objectivesobjectives
Future research ?Future research ?
Future research ?Future research ?
““I found that focusing on I found that focusing on studying in a PROBLEM-BASED studying in a PROBLEM-BASED manner (i.e. to be able to manner (i.e. to be able to discuss my clinical approach in discuss my clinical approach in the oral exam) was VERY the oral exam) was VERY EFFECTIVE.”EFFECTIVE.”
QUESTIONS? COMMENTS?QUESTIONS? COMMENTS?