The Missing Link? Using Reflection to Advance Qualitative Evaluation and Assessment
Sheila Corrall [email protected]
School of Computing and Information
Information Culture & Data Stewardship
Outline• Defini,onsofkeytermsandconcepts(seealsoCorrall,2017)• Background
– Reflec,veprac,ceinlibraries– Reflec,onandreflexivityin(qualita,ve)research
• Tools,modelsandframeworksfacilita,ngreflec,vity– Reflec,veprocessfordevelopmentalevalua,on[handout]– Triangulatedreflexiveinquiryques,ons[handout]– Readingdatathreeways– Analy,cmemosreflec,ontopics
• ReferencesAcknowledgement• [email protected]
Information Culture & Data Stewardship
WorkingDefini,onsofKeyTermsandConcepts• Developmentalevalua-on–anevalua,onwherethefocusison
changeorinnova,onincomplexdynamicenvironments– alsoknownasreal-,meevalua,on,emergentevalua,on,ac,on
evalua,on,andadap,veevalua,on(PaVon,2016)• Reflec-on–theabilitytothinkaboutthepast,inthepresentfor
thefuture(Carroll,2009)– aformofmentalprocessingthatweusetofulfilapurposeorto
achievesomean,cipatedoutcome(Moon,2006)• Reflexivity–theabilitytolookbothinwardsandoutwardsto
recognizeconnec,onswithsocialandculturalunderstandings(Fook&Gardner,2007)– thinkingcri,callyaboutwhatyouaredoingandwhy,
confron,ngando^enchallengingyourownassump,ons,andrecognizingtheextentthatyourthoughts,ac,onsanddecisionsshapehowyouresearchandwhatyousee(Mason,2002)
Information Culture & Data Stewardship
Reflec,vePrac,ceinLibraries• Widelyrecognizedasessen,alforprofessionaldevelopment• Par,cularlyassociatedwithimprovingthecompetenceof
librariansasinforma,onliteracyinstructors/teachers+suppor,ngstudentsasreflec,velearners(Corrall,2017)
• Alsodescribedasacrucialcomponentofwhatittakestobeanevidence-basedprac,,oner-researcher(Wilson,2016)– andafocusareaforassessmentlibrarians(Oakleaf,2013)
• Butsignificantdeficienciesinthetreatmentofreflec,oninlibraryliteratureonresearch,evalua,onandassessment– coverageispoorwhencomparedwithcognatedisciplinesØ offersliVleornoprac,calguidanceonusingreflec,onor
onbecomingareflec,veandreflexiveresearcher
Information Culture & Data Stewardship
Reflec,onin(QUAL)ResearchandEvalua,on
• Explicitintegralpartofclassicfour-stepac,onresearchcycle– plan,act,observe,reflect
• Assumeddimensionofconceptual/designphaseofresearch– reflec,veinquiry(Hernon,Dugan&Nitecki,2011)
• Alsoimportantinformula,ngques,ons,reviewingliterature,interpre,ngresults,andvalida,ngfindings
• Corecharacteris,cofqualita,veresearch,par,cularly(concurrent)datacollec,on/analysisandreport-wri,ng– especiallyethnography,phenomenologyandcasestudies
Information Culture & Data Stewardship
ReflexivityinQualita,veResearchandEvalua,on• Thequalita,veresearcheristheinstrumentofinquiry
– yourprofessionaland/orpersonalbackground,interests,andmo,va,onsmayhaveimplica,onsforyourstudy
Areyouanoutsideorinsider?Areyouanexpertorlearner?Areyoudoingyourresearchon,for,orwithyourpar,cipants?
Ø Thereportofaqualita,vestudyshouldexplaintheposi,onandperspec,veoftheresearchertostrengthenitsvalidityHowareyourexperiences,thoughts,andfeelingsaffec,ng
howyouconductyourstudy,whatyouseeorhearinthefield,andhowyouanalyze,represent,andinterpretyourdata?
• Qualita,veresearchersshouldreflectonsuchques,ons,andrecordtheirresponsesinfieldnotesand/oranaly,cmemos– memoingfacilitatesthetransi,onfromcodingtorepor5ng
Information Culture & Data Stewardship
Using Reflective Practice to Investigate an Issue or Evaluate an Initiative
AData-Based,Story-Based,Engagement-BasedInterac,veApproachØ Reflec,vePrac,ceProcessfor
DevelopmentalEvalua,on(PaVon,2011)
Information Culture & Data Stewardship
2.Sharestories
3.AnalyzepaVernsand
themes
4.Iden,fyimplica,ons:
ac,ons,lessons
1.Ques,onfor
focusedreflec,on
Reflec,vePrac,ceCycleforOngoingDevelopmentalEvalua,on
REFLECTIVEPRACTICE
(PaVon,2011,p.269)
Information Culture & Data Stewardship
Reflec,vePrac,ceforDevelopmentalEvalua,on1. Iden,fyafocusforinquiryandlearningthatprovides
direc,onandvisiontothedesiredchange.2. Turntheconcept,idea,value,orvisionintoanexperien,al
inquiryques,on.3. Sharepersonalexperiencesthatrespondtotheques,on.4. Groupmemberscanaskshortclarifyingques,ons.5. Par,cipantsiden,fypaVernsandthemesinthestories.6. Ifthereismorethanonegroup,eachgroupreportstheir
themesandpaVernstothefullgroup.7. Thegrouppicksoneortwothemesfordiscussionthathave
importantimplica,ons(andlessons)fortheworkathand.8. Generateac,onagreementsandnextstepsforreflec,on.cf.What?SoWhat?NowWhat?(Driscoll/RolfeinCorrall,2017)
Information Culture & Data Stewardship
Reflec,vePrac,ceProcess:ExamplesandGuidelines1. Iden,fyafocus:Transformingacademicliaisonfroma
service-as-supporttoanengagementorpartnershipmodel2. Formulateaques,on:Whatexperienceshaveyouhadof
successfullyengagingorpartneringwithstudentsorfaculty?3. Sharepersonalexperiences:Mustusefirsthandexperiences
andtellthestoryin3-5minutes(don’texplainoranalyze)4. Askclarifyingques,ons:Agreeconfiden,ality,beprepared
foremo,ons/strongfeelings,andenforce,mesensi,vely.5. Iden,fypaVernsandthemes:Commonspecificobserva,ons
andmorecategoricalgeneraltopicalcross-cupngmeanings.6. ReportpaVernsandthemes:Iden,fyperson(s)toreportout.7. Discussandanalyzeimplica,ons:Lookforlessonslearned.8. Agreeac,onsandnextsteps:Formulateanotherques,on…
Information Culture & Data Stewardship
Becoming a Reflective and Reflexive Researcher Frameworks/Models/ToolsYouCanUseØ TriangulatedInquiry:ReflexiveQues,ons
(PaVon,2015)Ø ReadingData:ThreeLevels(Mason,2002)- Literally,Interpre,vely,Reflexively
Ø Analy,cMemos:Reflec,onTopics(Saldaña,2015)
Information Culture & Data Stewardship
ReflexiveScreens:
culture,age,gender,class,socialstatus,educa,on,
family,poli,calpraxis,language,values
Myself,asinquirer:HowdoIknowwhatIknow?
Whatshapesandhasshapedmyperspec,ve?
Peopleinthese5ngbeingstudied:
Howdotheyknowwhattheyknow?Whatshapesandhasshapedtheirworldview?Howdotheyperceiveme?Why?HowdoIknowthat?HowdoIperceivethem?
Audiencesforthestudy:
HowdotheymakesenseofwhatIgivethem?Whatperspec,vesdotheybringtothefindingsIoffer?Howdotheyperceiveme?HowdoIperceivethem?
(PaVon,2015,pp.72,604-605)
ReflexiveQues,ons:TriangulatedInquiry
Information Culture & Data Stewardship
HowDoYouIntendto“Read”YourData?(Mason,2002)• Aliteralreadingwillfocusonelementssuchasform,content,
structure,sequence,style,andlayout– formandstructureofdialogue,wordsandlanguageused
• Aninterpre-vereadingwillgothroughorbeyondthedatatoconstruct/documentwhatyouthinktheymeanorrepresent– notonlywhatiscontainedinadocumentorimage,but
whatorwhoisnotthere;alsothecontextofproduc,on,consump,on,interpreta,onanduse
• Areflexivereadingwilllocateyouaspartofthedataandexploreyourrole,perspec,ve,andinfluenceintheirgenera,onandinterpreta,on(e.g.,viatriangulatedinquiry)
Yourinten,onshaveimplica,onsforhowyoucollect/generateyourdataandfieldnotes,andrepresentthemastextorgraphics
Information Culture & Data Stewardship
Reflec,onTopics:Reac,veorProac,veMemo-Wri,ng• Howyoupersonallyrelatetothepar,cipantsand/orthephenomenon• Yourstudy’sresearchques,ons• Yourcodechoicesandtheiropera,onaldefini,ons• Thepar,cipants’rou,nes,rituals,rules,roles,andrela,onships• EmergentpaVerns,categories,themes,concepts,andasser,ons• Thepossiblenetworksandprocesses(links,connec,ons,overlaps,flows)
amongthecodes,paVerns,categories,themes,concepts,andasser,ons• Anemergentorrelatedexistenttheory• Anyproblemswiththestudy• Anypersonalorethicaldilemmaswiththestudy• Futuredirec,onsforthestudy• Theanaly,cmemosgeneratedthusfar• Thefinalreportforthestudy
CodingDataßàUnderstandingPhenomena:ReciprocalRela,onship
MemosasConversa5onswithYourselforLe<erstoaFriend(Saldaña,2015)
Information Culture & Data Stewardship
ReferencesCarroll,M.(2009).Frommindlesstomindfulprac,ce:Onlearning
reflec,oninsupervision.PsychotherapyinAustralia,15(4),40-51.Corrall,S.(2017).Crossingthethreshold:Reflec,veprac,cein
informa,onliteracydevelopment.JournalofInforma5onLiteracy,11(1),23-53.
Fook,J.,&Gardner,F.(2007).Prac5singcri5calreflec5on:Aresourcehandbook.Maidenhead,UK:OpenUniversityPress.
Hernon,P.,Dugan,R.E.,&Nitecki,D.A.(2011).Engaginginevalua5onandassessmentresearch.SantaBarbara,CA:LibrariesUnlimited.
Mason,J.(2002).Qualita5veresearching(2nded.).London:SAGE.AvailableathVp://www.sxf.uevora.pt/wp-content/uploads/2013/03/Mason_2002.pdf.
Moon,J.A.(2006).Learningjournals:Ahandbookforreflec5veprac5ceandprofessionaldevelopment(2nded.).Abingdon,UK;NewYork:Routledge.
Information Culture & Data Stewardship
ReferencesOakleaf,M.(2013).Buildingtheassessmentlibrarianguildhall:Criteria
andskillsforqualityassessment.JournalofAcademicLibrarianship,39(2),126-128.
PaVon,M.Q.(2011).Developmentalevalua5on:Applyingcomplexityconceptstoenhanceinnova5onanduse.NewYork:GuilfordPress.
PaVon,M.Q.(2015).Qualita5veresearch&evalua5onmethods(4thed.).ThousandOaks,CA:SAGE.
PaVon,M.Q.(2016).Stateoftheartandprac,ceofdevelopmentalevalua,on.InM.Q.PaVon,K.McKegg&N.Wehipeihana(Eds.),Developmentalevalua5onexemplars:Principlesinprac5ce(pp.1-24).NewYork:GuilfordPress.
Saldaña,J.(2015).Thecodingmanualforqualita5veresearchers(3rded.)(pp.43-65).London;ThousandOaks,CA:SAGE.
Wilson,V.(2016).Prac,,oner-researchersandEBLIP.InD.Koufogiannakis&A.BreVle(Eds.),Beingevidencebasedinlibraryandinforma5onprac5ce(pp.81-91).London:Facet.