Development of viva voce exam to assess clinical reasoning in the paraclinical phase of a BVSc program. HelenOwen,SeniorLecturerinVeterinaryPathologyandJus:neGibson,Senior
LecturerinVeterinaryBacteriologyandMycology
DrTamsinBarnesAssociateProfessorRowlandCobboldDrKatrinaGarreGDrStevenKopp
AssociateProfessorJoanneMeersDrDanSchullAssociateProfessorJennySeddonDrFrancesShapterAssociateProfessorRebeccaTraub
Background on Program • TheBVScatUQisanundergraduate5yearprogram• First3yearsconsistpredominantlyofpreclinicalscienceswiththefourthandfi?hyearshavingmoreofaclinicalfocus.• Majorcoursesinthirdyear-ayear-longcourseonpathophysiology–stereotypicalwayswhichFssuesrespondtoinjury,prototypediseases-clinicalsigns,clinicalexamanddiagnosFcapproachtobodysystem-courseoninfecFousdisease(parasitology,virology,bacteriologyandmycology).
Development of viva voce exam • In2012,weweretaskedwithdevelopingavivavoceoralexamasaprogramlevel“hurdle”assessmentattheendofthirdyear.
ObjecFves,toassess:• applicaFonoftheorytocasescenarios• diagnosFcclinicalreasoning• integraFonofmaterialfromboththepathophysiologyandinfecFousdiseasescourses• professionalismandoralcommunicaFonskills
• Exposuretooralexamformatbefore5thyearvivaexams
Advantages of oral format
• Goodforassessingproblem-solvingabiliFes.• Canprovideinsightintostudents’cogniFveprocesses.• Itallowsprobingofthedepthandextentofstudents’knowledgewithfollow-upquesFons• UnclearorambiguousquesFonscanbere-expressedorimmediatelyclarified• Studentscanbeguidedbackontotherighttrackiftheystray
Advantages of oral format • Allowsjudgementsaboutstudents’interpersonalcompetence-confidence,self-awarenessandaspectsofprofessionalism.• SimulaFonofclinicalpracFce–studentsneedto“thinkontheirfeet”• Itimprovesthequalityofstudentlearning–studentshavebeenreportedtostriveforthoroughnessinunderstandinginresponsetotheunpredictabilityandtoavoidembarrassingthemselvesinfrontoftheexaminer
• Itsuitssomestudents
Disadvantages of oral format • Generatesalotofanxietyinstudentsandexaminers• Timeconsumingandrequiresalotofenergy• LackofanonymityandpotenFalforbias–unconsciousbiasbasedoncharismaFcpersonality,welldressed,beingarFculate• Reliability-Inter-case-differentstudentsgetdifferentcasestopreventsharingofinformaFon.
-Inter-rater-Intra-raterreliability
Features of the initial exam • FormulaicseriesofquesFonsbasedonthediagnosFcprocess-ProvidedwithimageandsomeinformaFon–historyandclinicalexamfindings- bodysystem(s)- possiblepathologicalprocesses(highweighFng)–usingpathophysiologyforclinicalreasoning
- differenFaldiagnoses(lowweighFng!)- diagnosFcplan(highweighFng)- Interpretresults,treatment,control,prognosis
• professionalism
Initial exam • Allstudentsreceivedonecase• StudentswhofailediniFalexamweregivenaresit• 10minutesperusal,20minutesoral• 2examinersperstudent-increasereliability–consensusmarksused
• Rubricandmodelanswers-increasereliability
Changes over the years based on examiner and student feedback
• 2cases(usuallyonelargeanimal,onesmall)–increasesreliability• Moreexaminertraining• RemovedhistoryandclinicalexamsecFons• 2-stagepathophysiology• Adjustedrubric-moreemphasisonsystemaFcapproachforpathogenesisanddiagnosFctesFng,forcedtoprioriFseDDxs• Year-longpreparaFonandprovisionofformaFvefeedback.
Youarepresentedwitha1yearolden3remaleDomes3cShortHaircat.Theownerno3cedthatthecatisoffitsfoodand‘quieter’thanusual,thishasbeenthecaseforaboutaweek.
Exam format – written section (30 minutes)
PartA.Descrip:onofcase,presen:ngsigns(images/videos)
Exam format – written section • PartB.OutlinethepathologicalprocesseswhichmaybecausingthepresentaFon• StudentssFllrotelearningpathogenesisforparFcularclinicalpresentaFonwithoutemphasising/prioriFsingbasedoncaseparFculars• OrstudentsusingpacernrecogniFonandbeingabletodescribeonespecificpathogenesisbutnotbeingabletosuggestalternaFves• Breakdownintotwosteps,inthefirststep,theyneedtodemonstrateasystemaFcknowledgestructurethatincludesallpossiblepathogeneses• Secondstep,providedwithaddiFonalinformaFonaboutthecaseandneedtousethistoprioriFseDDxs
• Encouragestudentstoconsiderallpossiblepathogenesesforabroadclinicalsignatthispoint• Mustdemonstratestructure,systemaFcapproach,encouragedtouseschemes
Scheme-inductive knowledge structures
Harasymetal.2008KaohsiungJMedSci,24:7;341-355.
ResearchsuggeststhatexpertproblemsolversaredisFnguishedbythewayintheyorganiseandunderstandtheirknowledge
Scheme-inductive knowledge structures • Mandinetal,UniversityofCalgaryFacultyofMedicine–“ClinicalPresentaFon”curriculum
• “The“hypotheFco-deducFve”strategytradiFonallyusedforPBLshouldbereplacedbyscheme-drivensearchstrategiessothatstudentsdevelopamoreorganisedandlogicalapproachtoproblem-solving”
SchemeshavemulFplepurposes:• ProvidesystemaFcapproachtodecreasetheriskofomidngDDxs
• wayoforganisingknowledgeformoreefficientunderstanding,serveasascaffoldfornewinformaFon
Thehistory,physicalexaminaFonandlaboratoryinvesFgaFonarealldrivenbythederivedscheme
Paraclinical schemes • Probablydifferentcontenttoclinicalschemes,morerelianceonprinciplesofpathophysiology,lesssignalmentetc?
Nodescanconsistof:• locaFone.g.bodysystems-cardiovascularversusrespiratorytract;withinbodysystems-upperversuslowerrespiratorytract;• aformofgeneralpathologicalprocessintheFssuee.g.myocardiFs,hepaFcnecrosis,pulmonaryoedema• SpecificcausaFveagents(prototypic)• SomepresentaFonsabitdifferente.g.anaemia,jaundice• Lotsofschemesprovidedtothestudentsandtheyareencouragedtoimproveonthem
Mnemonics as a system? • e.g.DAMNITV-DegeneraFve,Anomalous,Metabolicetc• Studentso?enpreferthisapproach• tryingtonotbetooprescripFve,flexibilityallowingfordifferentthoughtprocesses,clinicalpresentaFon
Don’tworktoowellinourhands….• Doesn’trequireexplicitidenFficaFonofbodysystems• Studentso?endon’tseemtounderstandpathogenesis• Doesn’tprovidestructureforsystemaFcdiagnosFcapproach• Moreappropriateasamemoryjoggerinclinicalyearsa?erstudentshavedemonstratedpathophysiologyknowledge?• UsefulaspartofamulFplestrategyapproach?
YouwillbeprovidedwithaddiFonalhistoryorclinicalfeatures• YouperformaPCVandthereismarkedanaemia.
• IniFallyweassumethatgoodstudentswouldprioriFseDDxsbasedonallthefeaturesofthecasebutthisrarelyhappened
PartC.Giveaproblemdefini:on• Subacuteinappetence,depression,jaundiceandanaemia.
Exam format – written section
Exam format – oral section PartD.Bodysystem(s)?• haematopoieFc
• PartE.UsetheaddiFonalinformaFontoidenFfymostlikelypathologicalprocess(es),describepathogenesisandprovideaeFologicalexamplesifappropriate
PartF.List5differenFaldiagnoses
PartG.DiagnosFcplan• IniFallyfoundthatstudentswouldjusttestandFckoffeachoftheirdifferenFals• EncouragesystemaFcapproach,useofschemetoguide,mostinformaFonfromfewesttests• EncourageuseofdiagnosFcstoguidetreatment• StudentsdorelaFvelypoorlyinthissecFon
Exam format – oral section
Exam format – oral section
Providediagnosis,prognosis,treatmentplan,controlThissecFonalsopoorlydoneingeneral
Exam format – oral section
• PartI–professionalism–useofprofessionallanguage,professionalbehaviouranddress.
Outcome of viva exams • Thisisamust-passhurdle,alargernumberofstudentsfailthiscomparedtofailingonbasisoftheoryexams(~5-9/120versus~2-3)• O?enabout10studentsonpass/failline,reluctanttoletthemgointo4thyear–whatshouldourexpectaFonsbeforthesenovicediagnosFcians?• Moststudentswhofailvivaareweakintheoryalsobutnotalways
Preparation for viva exams • 6face-to-facetutorialsdispersedthroughtheyear• 5thyearstudentsactastutorsforsmallgroups• 3rdyearsexaminingeachotherinpairs• Includeallstepsinprocess,evenatthebeginning• 3on-linecase-basedtutorialswheregenericfeedbackgivenbutstudentscanelecttohavepersonalfeedback• “diagnosFcwork-shop”• Viva“hint-sheet”
Lessons we have learnt • WetookalotforegrantediniFally:“goodstudentsshouldautomaFcallydemonstrateasystemaFc/structuredapproach,prioriFseDDxsbasedonparFcularsofacase…”• NeedtobecompletelyexplicitinexpectaFonsandspecificforeachstepinprocesswhichmeanshavingagoodunderstandingoftheminyourownmind• NeedtostructurerubricsothatitisnotpossibleforastudenttogetgoodmarkswithoutasystemaFcapproachandbeingabletoprioriFse
• NeedtostructurepreparaFonmaterialtoencouragegoodhabits• EncouragepracFceofverbalcommunicaFonasthisisaverydifferentskilltowricen
Conclusions Oralformatbenefits:• Beingabletoguidestudentsontotherighttrack,test“thinkingonfeet”,probeunderstanding• FearisapowerfulmoFvatorforembracingnewlearningapproaches
OralformatlimitaFons:• VariaFonincases–lackofstandardisaFon?• Inter-raterreliability• MassiveamountofFmeandenergy,difficulttoengageexaminers• InternaFonalstudentsdisadvantagedbylanguage,culturallyingrainedbehaviour? • ReallygoodformatforassessingdiagnosFcclinicalreasoning
(oralcommunicaFonskills,professionalism)butdoesgoodoutweighthebad?
Future directions • Reliability–variaFoninmarksacrosscases,examiners,secFonsofexam• InvesFgatewhetherpreparaFonandtakingtheseexamshasalasFnginfluenceonstudents’studyhabitsandapproachtocases• DevelopingmoreefficientmeansofprovidingformaFveassessment• BecerintegraFonwiththeclinicalyearsoftheprogram–evolvingnatureofclinicalreasoningstrategies,rollingoutofseriesofverFcallyintegratedon-linetutorials• ObtainbecerunderstandingofapproachesofweakstudentssoastobecerassisttheirpreparaFon
References • Davis,M.,Karunathilake,I.2005.TheplaceoforalexaminaFonsintoday’sassessmentsystems.MedicalTeacher,27;4:294-297.
• Harasymetal.2008CurrentTrendsinDevelopingMedicalStudents'CriFcalThinkingAbiliFes.KaohsiungJMedSci,24:7;341-355.
• Joughin,2010.Ashortguidetooralassessment.LeedsMetropolitanPressinAssociaFonwithUniversityofWollongong.
• Mandin,H.,Harasym,P.,Eagle,C.andWatanabe,M.1995.Developinga“clinicalpresentaFon”curriculumattheUniversityofCalgary.AcademicMedicine.70;3:186-193.
• Mandin,H.,Jones,A.,Woloschuk,W.andHarasym,P.1997.Helpingstudentslearntothinklikeexpertswhensolvingclinicalproblems.AcademicMedicine,72;3:173-179.