Large-ScaleAssessments:Developingassessmentpublications
SallyRobertson,ACER,CentreforGlobalEducationMonitoring(GEM)
Processfordevelopingassessmentdocuments
• Planning• Writing• Reviewing• Re-writing• Proof-reading• Addpublicationdetails• Formattinganddesign• Finalreview• Printinganddissemination
Planningconsiderations
• Content(informedbydataanalysisandreportingplan)
• Styleandstructureofdocument• Expectationsoforganisation/client• Teaminvolved• Timeframes• Qualityassuranceprocesses
Writingconsiderations
Ensurethat:1.thedocumentisstructuredinalogicalway2.assessmentinformationisreportedaccuratelyandappropriately3.thekeymessagesstandout4.providingsufficientdetailisbalancedwithwritingconcisely5.theinformationisengaging,providinginformationthatisrelevanttoreaders6.clear,consistentandaccessiblelanguageisutilised
1.Documentstructure
• Presentinformationinalogicalorderthatisappropriatefortheparticulartypeofdisseminationproduct
• Useinformativeheadings• Getearlyfeedbackonthedraftstructure• Forreports,includeanexecutivesummary
http://www.keepeek.com/Digital-Asset-Management/oecd/education/low-performing-students_9789264250246-en#page1
2.Accuratereporting
• Includesufficientbackgroundinformationabouttheassessment(why,who,when,what,how)
• Considerwhatdataisavailableandwhatreportingisappropriatefortheanalysis
• Acknowledgeanylimitations• Providereaderswithinformationonhowtointerpret
theresults• Bewaryofover-interpretingtheresults• Referencepriorresearchorotherassessmentprograms
(whererelevant)
3.Keymessages
Somewaystomakeimportantmessagesstandout:•Reiteratekeymessages(rephrasingtheinformation)•Formatting•Considertheplacementoftheinformation(e.g.,inexecutivesummary,chapterintroductionsandconclusions)•Usegraphsandimages•Useexamplestoillustratekeypoints•Limitthenumberofkeymessagesincludedinsummaries
http://img.asercentre.org/docs/Publications/ASER%20Reports/ASER%20TOT/fullasertrendsovertimereport.pdf
http://wwwstaging.acara.edu.au/docs/default-source/default-document-library/20160719-naplan-online-key-messages.pdf?sfvrsn=2
4.Levelofdetail
• Willdependonthepublicationtype,theaudienceandtheimportanceoftheinformation
• Takeintoaccountthelikelybackgroundknowledgeoftheaudience
• Backupstatementswithevidence• Explaingraphsandtablesinthetext• Providesufficientdetailtosupportunderstandingofthe
information• Providereaderswithdetailsaboutwheretheycanaccess
furtherinformation• Includelengthy/detailedtablesinappendix,whereappropriate
https://www.acer.edu.au/documents/PISA-2012-Report.pdf
5.Engagingreaders
• Firstimpressionscount!• Presenttheinformationinawaythatengagesreaders,sothattheywillwanttofindoutmore.
• Readersneedtofeelconvincedthattheresultsarevalidandreliable,andthattheinformationisimportantandrelevant.
• Addressquestionsandpolicyissuesthatwillbeofinteresttothetargetaudience.
• Usewelldesignedgraphs,tablesandimages.• Useawritingstyleandtonethatisappropriatefortheaudienceandtypeofdocument.
http://parlormultimedia.com/publishing/content/raise-bar
http://img.asercentre.org/docs/Publications/ASER%20Reports/ASER%202014/fullaser2014mainreport_1.pdf
http://epathshala.nic.in/wp-content/uploads/2015/NAS/Why%20NAS%20Matters.pdf
6.Clearcommunication
• Writeclearlyandconcisely• Explaintheresultsandpossibleimplications(makethereaders’jobaseasyaspossible)
• Limitacronymsandabbreviations• Explaintechnicaltermsanduseconsistentterminology
• Usetheappropriatetense• Usetheactivenotpassivevoice• Reviewandproofread!
http://research.acer.edu.au/cgi/viewcontent.cgi?article=1012&context=mteg
Reviewingprocess
• Mayincludereviewbyinternalteamandbyclient/externalreviewers
• Peoplewithdifferentareasofexpertisemayneedtoreviewdifferentsections
• Havesomeonereviewthewholedocumenttocheckforconsistency(aneditor)
• Identifywhatfeedbackisneededatwhatstage(forexample,feedbackonthestructure/bigpicture,detailedfeedbackonphrasing)
• Therearemanystagesofreviewingandre-writing–ensuresufficienttimeisallocated
Proof-readingprocess
• Detailedcheckforerrors(forexample,inspelling,punctuation,grammar,references)
• Willoftenincludecheckingdocumentagainstastyleguide
Publicationdetails
• Informationtoinclude:– Listofauthors– Copyrightdetails– Publisherdetails– ISBNand/orE-ISBN
• Thisinformationhelpswithidentification,marketingandreferencingthedocument
Formattinganddesign
• Utilisetheexpertiseofgraphicdesigners,wherepossible
• Identifywhereitcouldbeusefultouseinfographicsandimages
• Itisimportanttoscheduletimeforcheckingaftertheformattinghasbeendone
Printinganddissemination
Considerationsinclude:•Willtherebeonlineand/orprintversions?•Howmanycopiesshouldbeprinted?•Howshouldthedocumentbepublicised?•Howshouldthedocumentbedistributed?
Activity:Developingasummarypamphlet
Ingroups,youwillbedevelopingasummarypamphletbasedonanexistingthematicreport(seeactivitysheet)Thisactivitywillinclude:•readingthereport•writingthedraftpamphlet•receivingfeedbackfromothers(andprovidingotherswithfeedback)•revisingthepamphlet,inlightofthefeedback•reflectingontheactivity
Wewilltrynottotakethisapproach…
http://freshspectrum.com/in-defense-of-the-200-page-report/
Reflection
• Whatwasyourexperienceofthistask?• Whatarethestrengthsofthepamphletyourgroupdeveloped?
• Whatimprovementscouldbemadetothepamphletyourgroupdeveloped?
• Howwouldyouapproachdesigningasummarypamphletinthefuture,foradifferenttargetaudience?
UsefulresourcesCresswell,J.(2017).Disseminationandreporting.InP.Lietz,J.Cresswell,K.Rust&R.Adams(Eds.),Implementationoflarge-scaleeducationassessments.NewYork:Wiley.Forster,M.(2000).APolicyMakersGuidetoInternationalAchievementStudies.Availableat: http://research.acer.edu.au/policy_makers_guides/1Forster,M.(2001).APolicyMakersGuidetoSystemwideAssessmentPrograms.Availableat:
http://research.acer.edu.au/policy_makers_guides/2Kellaghan,T.,Greaney,V.,&Murray,T.S.(2009).Usingtheresultsofanationalassessmentofeducationalachievement.Nationalassessmentsofeducationalachievement(Vol.5).WorldBank,Washington,DC.Availableat:http://elibrary.worldbank.org/doi/abs/10.1596/978-0-8213-7929-5Masters,G.(2014).Assessment:GettingtotheEssence.Availableat: https://www.acer.edu.au/files/uploads/Assessment_Getting_to_the_essence.pdfMasters,G.(2013).Towardsagrowthmindsetinassessment.Availableat: http://research.acer.edu.au/cgi/viewcontent.cgi?article=1017&context=ar_miscRoss,K&Genevois,J.(Eds.).Cross-nationalstudiesofthequalityofeducation:Planningtheirdesignandmanagingtheirimpact.Paris:UNESCO,InternationalInstituteforEducationalPlanning.Availableat: http://unesdoc.unesco.org/images/0014/001470/147093e.pdf
Note:Fortheexamplesusedinthispresentation,pleaserefertotheURLprovidedontheslides