Dimensions of quality - sparqs Dimensions of Quality.pdf · dimensions of qualiTy by Graham Gibbs 2...

Dimensions of quality byGrahamGibbs

ThehiGhereducaTionacademy

2

The views expressed in this publication are those of the authors and not necessarily those of the Higher Education Academy.

ISBN 978-1-907207-24-2 © The Higher Education AcademySeptember 2010

The Higher Education AcademyInnovation WayYork Science ParkHeslingtonYork YO10 5BR

www.heacademy.ac.uk

Tel: +44 (0)1904 717500Fax: +44 (0)1904 717505

All rights reserved. Apart from any fair dealing for the purposes of research or private study, criticism or review, no part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any other form or by any other means, graphic, electronic, mechanical, photocopying, recording, taping or otherwise, without the prior permission in writing of the publishers.

To request copies of this report in large print or in a different format, please contact the Academy.

Designed by Daniel GrayPrinted by The Charlesworth Group

dimensionsofqualiTy

byGrahamGibbs

2 forewordbycraigmahoney

4 1. executivesummary

8 2. introduction

11 3. Thenatureofdimensionsofquality

14 4. Presagedimensionsofquality

19 5. Processdimensions

38 6. Productdimensionsofeducationalquality

43 7. summaryandconclusions

50 8. abouttheauthor

51 9. acknowledgements

52 10.references


2

foreword

Theperennialdebateaboutwhatconstitutesqualityinundergraduateeducationhasbeenreignitedrecently,notleastbyarangeofpublishedresearch,selectcommitteeactivity,tighteningofresource,andthelarge-scalereviewbylordbrowne.

astheorganisationdedicatedtoenhancingthequalityofstudents’learningexperiences,thehighereducationacademyispleased,throughthispieceofwork,tocontributefurthertothisimportantdebate.

ourstarting-pointistwofold:first,thathighereducationshouldbeatransformativeprocessthatsupportsthedevelopmentofgraduateswhocanmakeameaningfulcontributiontowidersociety,localcommunitiesandtotheeconomy.second,thatanydiscussionaroundqualityneedstobeevidence-informed.asaresult,weidentifiedaneedtosynthesiseandmakesenseofthescatteredresearchinthefieldofhighereducationquality.wewantedtofindoutwhattheresearchevidencetellsusandwhatfurtherworkwecandotoapplytherelevantfindingsinourquesttoimprovethequalityofstudentlearninginuKhighereducation.

GrahamGibbsstatesthatthemostimportantconclusionofthisreportisthatwhatbestpredictseducationalgainismeasuresofeducationalprocess:inotherwords,whatinstitutionsdowiththeirresourcestomakethemostofthestudentstheyhave.examiningtheevidence,hedrawsconclusionsaboutsomekeytopicsthathavebeenthesubjectofmuchdebatearoundquality.forexample,heconcludesthatthenumberofclasscontacthourshasverylittletodowitheducationalquality,independentlyofwhathappensinthosehours,whatthepedagogicalmodelis,andwhattheconsequencesareforthequantityandqualityofindependentstudyhours.

healsoreiteratesresearch(nasret al.,1996)thatshowsthatteacherswhohaveteachingqualifications(normallyaPostgraduatecertificateinhighereducation,orsomethingsimilar)havebeenfoundtoberatedmorehighlybytheirstudentsthanteacherswhohavenosuchqualification.ithinkthisisacrucialpoint.attheacademywebelievethathighqualityteachingshouldbedeliveredbyacademicstaffwhoareappropriatelyqualifiedandcommittedtotheircontinuingprofessionaldevelopment.Tothisendwewillcontinuetoprovideanddevelopanadaptableframeworkforaccreditedteachingqualificationsinhe,incorporatingtheuKProfessionalstandardsframeworkandotherrelevantteachingqualifications.wewillalsocontinuetoworkwithheistodevelopandmanagecPdframeworksforlearningandteaching.

Thereportalsoconcludesthatsomedimensionsofqualityaredifficulttoquantify,anditisthereforedifficulttoseewhateffecttheymighthave.aspectsof

dimensionsofqualiTy

3

departmentalcultureareonesucharea:whetherteachingisvaluedandrewarded,whetherinnovationinteachingissystematicallysupportedandfunded,etc.Theacademyhasalreadyconductedresearchintotherewardandrecognitionofteachingwhichshowedthatover90%ofacademicstaffthoughtthatteachingshouldbeimportantinpromotions.wewillcontinuetofocusonthiswork.

someofthefindingsofthisreportmayconfirmaspectsofinstitutionalpolicyonenhancingquality,someofthemwillpromptnewanddifferentapproachestofocusedinvestmentoffundingandexpertiseinordertomaximiseeducationalgain,particularlyatatimeofdiminishingresource.someofthemwillcallintoquestiontheefficacyandappropriatenessofpracticesandpolicies,andcauseustolooknotathowmuchisspentpercapita,butonhowitisspent;lessonhowmanycontacthoursareprovidedbutwithwhomandwithwhatconsequencesforindependentlearning;ontheextenttowhichwetrulysupportandadoptthekindsofpedagogicpracticesthatengenderstudents’intrinsicengagementintheirlearning.

Grahamarguesforabetterfocusonevidenceinordertounderstandqualityproperly,toensurethatourqualityprocessareinformedtoagreaterextentbywhatweknowaboutwhatconstituteseffectivepracticeandabouttheextenttowhichthesepracticesareemployed,tomakebetterandmorecoordinateduseofthefullrangeofavailabledata,andtounderstandtherelationshipbetweenthem.

Thispaperisprimarilyforanaudienceofseniormanagersofheis–thecolleagueswhodevelopandimplementthekindsofinstitutionalpoliciesthathavethepropensitytoimprovestudentlearningandwhoconceptualisetheframeworkstosupportthatvitalprocess.wehopethatthisreportwillmeaningfullyinformbothpolicyandpracticeandlookforwardtofollowingupthisworkinthecomingmonthsbyengagingwithyouindebatesanddiscussionsaboutthedimensionsofquality.

ProfessorcraigmahoneychiefexecutiveThehighereducationacademy.


4

1. execuTivesummary

“ A … serious problem with national magazine rankings is that from a research point of view, they are largely invalid. That is, they are based on institutional resources and reputational dimensions which have only minimal relevance to what we know about the impact of college on students … Within college experiences tend to count substantially more than between college characteristics.” —Pascarella,2001

1.1 Thefocusofthereport

ThisreporthasbeenwrittentocontributetothecurrentdebatesabouteducationalqualityinundergraduateeducationintheuK,andabouttheneedtojustifyincreasesinresourcesonthebasisofindicatorsofeducationalquality.Thisreportwillidentifyarangeofdimensionsofqualityandexaminetheextenttowhicheachcouldbeconsideredavalidindicator,withreferencetotheavailableresearchevidence.itattemptstoidentifywhichkindsofdataweshouldtakeseriouslyandwhichweshouldbecautiousofplacingweighton.someofthesedimensionswemightbewisetopayattentiontocurrentlylackasolidevidencebase,especiallyinrelationtoresearchcarriedoutintheuKcontext,andsothereportalsoidentifiesprioritiesforresearchandfordatacollectionandanalysis.

1.2 Theapproachtakentoconsideringdimensionsofquality

Thereportidentifieswhichdimensionsofeducationalqualitycanreasonablybeusedtocompareeducationalsettings.itadaptsbiggs’s‘3P’model(biggs,1993)of‘presage’,‘process’and‘product’tocategorisethevariablesunderconsideration(seesection3.2).Thereportexaminesawiderangeofpotentialindicators.Presagevariablesdefinethecontextbeforestudentsstartlearning,processvariablesdescribewhatgoesonasstudentslearn,andproductvariablesrelatetotheoutcomesofthatlearning.forpresageandprocessvariablestheavailableevidenceisexaminedconcerningthevalidityofthevariable:theextenttowhichitpredictsstudentlearningoutcomesandeducationalgains.eachproductvariableisexaminedforitsabilitytoindicatecomparativequality.

dimensionsofqualiTy

5

1.3 Thelimitationsofpresageandproductvariables

Presagevariablessuchasfunding,researchperformanceand,thereputationthatenablesaninstitutiontohavehighlyselectivestudententry,donotexplainmuchofthevariationbetweeninstitutionsinrelationtoeducationalgains.measuresofeducationalproductsuchasgradesdoreflectthesepresagevariables,butlargelybecausethebeststudentscompetetoenterthebest-fundedandmostprestigiousinstitutionsandthequalityofstudentsisagoodpredictorofproducts.measuresofproductsuchasretentionandemployabilityarestronglyinfluencedbyaraftofpresagevariablesthatgowellbeyondthoseusedbyhefceinsettingperformancebenchmarks.ThelackofcomparabilityofdegreestandardsprovesanobstacletointerpretationofstudentperformancedataintheuK.Thismakesinterpretingandcomparinginstitutionalperformanceextremelydifficult.

1.4 Theimportanceofprocessvariables

whatbestpredictseducationalgainismeasuresofeducationalprocess:whatinstitutionsdowiththeirresourcestomakethemostofwhateverstudentstheyhave.Theprocessvariablesthatbestpredictgainsarenottodowiththefacilitiesthemselves,ortodowithstudentsatisfactionwiththesefacilities,butconcernasmallrangeoffairlywell-understoodpedagogicalpracticesthatengenderstudentengagement.intheuKwehavefewdataabouttheprevalenceoftheseeducationalpracticesbecausetheyarenotsystematicallydocumentedthroughqualityassurancesystems,norarethey(inthemain)thefocusofthenationalstudentsurvey.

classsize,thelevelofstudenteffortandengagement,whoundertakestheteaching,andthequantityandqualityoffeedbacktostudentsontheirworkareallvalidprocessindicators.ThereissufficientevidencetobeconcernedaboutallfouroftheseindicatorsintheuK.

1.5 Theimportanceofmultivariateanalysis

fewrelationshipsbetweenasingledimensionofqualityandasinglemeasureofeithereducationalperformanceoreducationalgaincanbeinterpretedwithconfidencebecausedimensionsinteractincomplexwayswitheachother.Tounderstandwhatisgoingonanddrawvalidconclusionsitisnecessarytohavemeasuresofarangeofdimensionsofqualityatthesametimeandtoundertakemultivariateanalysis.large-scalemultivariateanalyseshavebeenrepeatedlyundertakenintheus,andhavesuccessfullyidentifiedthoseeducationalprocessesthataffecteducationalgains,andthosethatdonotorthatareconfoundedbyothervariables.incontrasttherehasbeenlittleequivalentanalysisintheuK.ThisispartlybecausedataintheuKthat


6

couldformthebasisofmultivariateanalysisforthatpurposearecurrentlycollectedbydifferentagenciesandhaveneverbeenfullycollated.

1.6 Theimportanceofeducationalgain

becauseeducationalperformanceispredictedbytheentrystandardsofstudents,tocompareinstitutionalperformanceinavalidwayitisnecessarytomeasureeducationalgain:thedifferencebetweenperformanceonaparticularmeasurebeforeandafterthestudent’sexperienceofhighereducation.whilethemostinfluentialusstudiesmeasureeducationalgaininavarietyofways,thereisverylittleevidenceavailableintheuKabouteducationalgain.

1.7 dimensionsofqualityindifferentkindsofinstitutions

institutionshavedifferentmissions,andcomparingthemusingproductdimensionsofqualitythatarethegoalsofonlyasubsetoftheinstitutionsleadstoconclusionsofdoubtfulvalue.Processdimensionsgiveafairercomparativepictureofqualitythandopresageorproductdimensions.however,differentpedagogicphenomena,andhencedifferentprocessvariables,arelikelytobesalientindifferentinstitutions.forexample,onlysomeoftheverydifferentwaysinwhichTheopenuniversityortheuniversityofoxfordachievesuchhighnationalstudentsurveyratingsarerelevanttootherkindsofuniversity.

1.8 dimensionsofqualityindif ferentdepartments

indicatorsofdimensionsofqualityoftenvarywidelybetweendepartmentswithinthesameinstitution,foravarietyofreasons.Prospectivestudentsneedqualityinformationaboutthespecificdegreeprogrammetheywishtostudyataninstitutionratherthanaboutinstitutionalaveragesoraboutclustersofdegreeprogrammesaggregatedinto‘subjects’asatpresent.Providingsuchinformationatasufficientlevelofgranularitymaybeimpractical.

1.9 dimensionsofqualitythataredifficulttoquantify

studiesofthecharacteristicsofbothinstitutionsanddepartmentsthathavebeenfoundtobeoutstandingintermsofvaliddimensionsofeducationalqualityhaveidentifiedprocessvariablesthatwouldbeextremelydifficulttoquantifyormeasureinasafeway,suchastheextenttowhichteachingisvalued,talkedaboutanddeveloped.

dimensionsofqualiTy

7

1.10 evidenceoftheproductsoflearning

oneofthemosttellingindicatorsofthequalityofeducationaloutcomesistheworkstudentssubmitforassessment,suchastheirfinal-yearprojectordissertation.Thesesamplesofstudentworkareoftenarchived,butrarelystudied.Thereisconsiderablepotentialforusingsuchproductsasmoredirectindicatorsofeducationalqualitythanproxiessuchasnssscores.

1.11 Thepotentialforimprovedquality,andtheevaluationofimprovementsinquality

Thereisclearevidencethateducationalperformanceandeducationalgainscanbeenhancedbyadoptingcertaineducationalpractices.intheusthenationalsurveyofstudentengagement(nsse)hasbeenusedsuccessfullybymanyinstitutionstoidentifywherethereareweaknessesincurrenteducationalprocessesandtodemonstratethepositiveimpactoftheintroductionofcertaineducationalpractices.

Poolingdataacrosssuchinnovationsthenprovidesavalidbasistoguideotherinstitutionsintheadoptionofpracticesthatarelikelytobeeffective.ThensscannotbeusedintheuKinthesameway,despiteitsreliability.Thereisavaluableroletobefulfilledbynationalagenciesinsupportingtheuseofvalidmeasuresoftheimpactofchangededucationalpractices,andinpoolingevidenceacrossinstitutions.

1.12 Thepotentialforinformingpotentialstudentsaboutquality

itseemsunlikelythatcomparativeindicatorsofqualitycurrentlyavailableintheuKcouldprovideprospectivestudentswithavalidbasistodistinguishbetweenindividualcourseswithregardtotheireducationalquality.Thecollationofcurrentlyavailabledataintoleaguetablesisinvalidandmisleading.evenintheuswherearangeofmorevalidindicatorsaremorewidelyavailable,thoseresponsibleforcollectingandinterpretingthedatacounselstronglyagainsttheircollationintoasingleleaguetable.


8

2. inTroducTion

Theextenttowhichindicatorsofqualityhaveshapedboththepoliticsofhighereducationandinstitutionalprioritiesisnotanewphenomenon(Patrickandstanley,1998).however,thereiscurrentlyincreasedemphasisontheoverallqualityofundergraduateeducationintheuK.datafromanumberofrecentsurveysandstudieshaveraisedchallengingissuesabout:

— differencesinqualitybetweeninstitutionswithintheuKthatinthepasthave,rightlyorwrongly,beenassumedtobebroadlycomparable;

— differencesinqualitybetweennationalhighereducationsystems,towhominthepasttheuKhasbeenassumed,rightlyorwrongly,tobesuperior,inthecontextofanincreasinglycompetitiveinternationalhighereducationmarketplace;

— theadequacyofnationalqualityregimesthathaveemphasisedscrutinyofaninstitution’squalityassurancetoagreaterextentthanofitseducationalprocessesoroutcomesofthekindemphasisedinsomeoftherecenthighprofilesurveysandstudies.

aParliamentaryselectcommittee(houseofcommons,2009)hastakenevidencefromawiderangeofsourcesandreachedchallengingconclusionsbothaboutthequalityofuKhighereducationandhowthatqualitycanbeassuredinthefuture.

amongallthedebatetherehassometimesbeenuncriticalacceptanceofsomesourcesofevidencethatcannotbeartheweightofinterpretation,andalsorejectionofevidencethatdeservestobetakenmoreseriously.eveninpublicreportsargumenthassometimesmadenouseofavailableevidence.Togiveoneexamplethequalityassuranceagency(2009)hasrespondedtodatathatsuggestboththatuKstudentsmightstudysignificantlylesshardthantheireuropeancounterparts,andthattherearewidedifferencesbetweeninstitutionsandsubjectswithintheuKinrelationtohowmanyhoursarestudied(hePi,2006,2007;brennanet al.,2009).fromtheperspectiveofthecurrentreportthekeyquestionsinthiscaseare:

— doesitmatterthatsomestudentsreceivelessclasscontactthanothers?areclasscontacthoursanindicatorofquality?

— doesitmatterthatsomestudentsputinlesstotaleffortthanothers?aretotalstudentlearninghoursanindicatorofquality?

dimensionsofqualiTy

9

insection5.2below,evidenceisreviewedthatmightinformtheqaa’scurrentpositiononthisissue.

similarlythefindingsofastudyofstudentexperiencebythenationalunionofstudents(nus,2008)mightbeinterpreteddifferentlyiftheywereinformedbytheavailableempiricalevidenceontheissuesitaddresses,suchastheeffectsofpaidworkonstudents’studyhours.

Theliteratureonthevalidityofindicatorsofqualityisvast,widelydispersedandmostlyamerican.ittendstobefocusedonspecificpurposes,suchascritiquingaparticularuniversityleaguetable,critiquingaparticulargovernment-definedperformanceindicator,establishingthecharacteristicsofaparticularstudentfeedbackquestionnaire,orexaminingthecharacteristicsofaparticularindicator(suchasresearchperformance).muchofthisliteratureistechnicalinnatureandwrittenforaspecialistaudienceofeducationalresearchers.Thecurrentreportattemptstobringmuchofthisdiverseliteraturetogetherencompassingmany(thoughnotall)dimensionsofquality.itisnotintendedtobeanexhaustiveaccount,whichwouldbeaveryconsiderableundertaking,anditiswrittenforageneralaudience.itwillnotdelveintostatisticalandmethodologicalminutiae,althoughsometimesanappreciationofstatisticalissuesisimportanttounderstandingthesignificanceoffindings.

Thisreportisintendedtoinformdebatebypolicyformersoffourmainkinds:thoseconcernedabouttheoverallqualityofuKhighereducation;thoseconcernedwithinstitutionalandsubjectcomparisons;thoseconcernedwithfundingonthebasisofeducationalperformanceandthosewithininstitutionsconcernedtointerprettheirownperformancedataappropriately.itmayalsobeusefultothosedirectingresourcesatattemptstoimprovequalityasitidentifiessomeoftheeducationalpracticesthatareknowntohavethegreatestimpactoneducationalgains.

itisimportantheretobeclearwhatthisreportwillnotdo.itwillnotreviewalternativequalityassuranceregimesormakeacaseforanyparticularregime.inidentifyingdimensionsofqualitythatarevaliditwill,byimplication,suggestelementsthatshouldbeincludedinanyqualityassuranceregime,andthosethatshouldnotbeincluded.

ThereportwillnotbemakingoverallcomparisonsbetweentheuKandotherhesystems,betweeninstitutionswithintheuK,betweensubjectsnationallyorbetweensubjectsordepartmentswithininstitutions.ratherthepurposeistoidentifythevariablesthatcouldvalidlybeusedinmakingsuchcomparisons.

Thereportisnotmakingacaseforperformance-basedfunding.reviewsoftheissuesfacingsuchfundingmechanismscanbefoundelsewhere(Jongbloedandvossensteyn,2001).however,validindicatorsofqualitywillbeidentifiedthatanyperformance-basedfundingsystemmightwishtoinclude,andinvalidindicatorswillbeidentifiedthatanyperformance-basedsystemshouldeschew.

finally,thereportisnotmakingacasefortheuseof‘leaguetables’basedoncombinationsofqualityindicators,nordoesitconsidertheissuesinvolvedinthecompilationanduseofexistingorfutureleaguetables.Trenchantandwell-founded


10

critiquesofcurrentleaguetables,andoftheiruseingeneral,alreadyexist(bowden,2000;brown,2006;clarke,2002;eccles,2002;GrahamandThompson,2001;Kehmandstensaker,2009;Thompson,2000;yorke,1997).someofthesecritiquescoversimilargroundtopartsofthisreportinthattheyidentifymeasurescommonlyusedwithinleaguetablesthatarenotvalidindicatorsofeducationalquality.

ThroughoutthereportthereisadeliberateavoidanceofusingindividualinstitutionsintheuKasexemplarsofeducationalpractices,effectiveorineffective,withtheexceptionofanumberofillustrationsbasedonTheopenuniversityandtheuniversityofoxford.despitebeingfarapartinrelationtofunding,theyareclosetogetheratthetopofrankingsbasedonthenss.Theyhaveachievedthisusingcompletelydifferenteducationalpractices,butthesepracticesembodysomeimportanteducationalprinciples.Theyaresodifferentfromotherinstitutionsthattherecanbelittlesenseinwhichtheycanbecompared,orcopied,exceptatthelevelofprinciples.itistheseprinciplesthatthereportseekstohighlight,becausetheyilluminateimportantdimensionsofquality.

dimensionsofqualiTy

11

3. ThenaTureofdimensionsofqualiTy

3.1 conceptionsofquality

‘quality’issuchawidelyusedtermthatitwillbehelpfulfirsttoclarifythefocusofthisreport.Therehavebeenanumberofattemptstodefinequalityinhighereducation,orevenmultiplemodelsofquality(e.g.chengandTam,1997).ThemostcommonlyciteddiscussionofthenatureofqualityinhighereducationintheuKisthatbyharveyandGreen(1993),andtheirhelpfulnomenclaturewillbeemployedhere.first,qualityisseenhereasarelativeconcept–whatmattersiswhetheroneeducationalcontexthasmoreorlessqualitythananother,notwhetheritmeetsanabsolutethresholdstandardsothatitcanbeseentobeofadequatequality,norwhetheritisreachesahighthresholdandcanbeviewedasoutstandingandofexceptionalquality,norwhetheracontextisperfect,withnodefects.whatisdiscussedhereisthedimensionsthatarehelpfulindistinguishingcontextsfromeachotherintermsofeducationalquality.

qualitymayalsobeseentoberelativetopurposes,whethertothepurposesandviewsofcustomersorrelativetoinstitutionalmissions.Thisreportdoesnottakecustomer-definedorinstitutionallydefinedconceptionsofqualityasitsstartingpoint.ratheraneffortwillbemadetofocusonwhatisknownaboutwhatdimensionsofqualityhavebeenfoundtobeassociatedwitheducationaleffectivenessingeneral,independentlyofpossiblevariationsineithermissionsorcustomers’perspectives.Thereportwillthenreturntotheissueofinstitutionaldifferencesandwillcommentinpassingondifferencesbetweenstudentsinthemeaningthatcanbeattachedtoqualityindicatorssuchas‘drop-out’.

afurtherconceptionofqualitymadebyharveyandGreenisthatofqualityastransformation,involvingenhancingthestudentinsomeway.Thisconceptioncomesintoplaywhenexaminingevidenceoftheeducationalgainsofstudents(incontrasttotheireducationalperformance).Thistransformationconceptionofqualityisalsorelevantwhenexaminingthevalidityofstudentjudgementsofthequalityofteaching,wherewhattheymaywantteacherstodomaybeknownfromresearchevidencetobeunlikelytoresultineducationalgains.whatisfocusedonhereisnotnecessarilywhatstudentslikeorwant,butwhatisknowntoworkintermsofeducationaleffectiveness.

itisusualtodistinguishbetweenqualityandstandards.Thisdistinctionismostrelevantinsection6.1onstudentperformance,wheretheproportionof‘good


12

degrees’canbeseentobeinpartaconsequenceofthequalitiesofwhatstudentshavelearntandinpartaconsequenceofthestandardsappliedinmarkingtheproductsofstudentlearning.Thisreportwillnotfocusonstandardsthathavebeenthesubjectofmuchrecentdebate;forexample,concerningtheoperationoftheexternalexaminersystem.

3.2 categorisingdimensionsofquality:presage,processandproduct

educationisacomplexbusinesswithmanyinteractingdimensionsofqualityinmanyvariedcontexts.Tounderstandwhatisgoingonitisnecessarytohaveawayofconceivingofthevariablesinvolvedandoforganisingandinterpretingstudiesoftherelationshipsbetweenthesevariables.Thisreportwilladoptthecommonlyused‘3P’model(biggs,1993),whichapproacheseducationasacomplexsystemwith‘Presage’,‘Process’and‘Product’variablesinteractingwitheachother.The‘3P’modelisessentiallythesameasthatusedbylarge-scalestudiesintheus(e.g.astin,1977,1993):the‘input-environment-output’model.Presagevariablesarethosethatexistwithinauniversitycontextbeforeastudentstartslearningandbeingtaught,andincluderesources,thedegreeofstudentselectivity,thequalityofthestudents,thequalityoftheacademicstaffandthenatureoftheresearchenterprise.noneofthesepresagevariablesdeterminedirectlyhowtheeducationalprocessmaybeconducted,althoughtheyoftenframe,enableorconstraintheformeducationtakes.

Processvariablesarethosethatcharacterisewhatisgoingoninteachingandlearningandincludeclasssize,theamountofclasscontactandtheextentoffeedbacktostudents.Processvariablesalsoincludetheconsequencesofvariablessuchasclasssizeforthewaystudentsgoabouttheirlearning,e.g.howthosevariablesimpactonthequantityandqualityoftheirstudyeffortandtheiroveralllevelofengagement.

Productvariablesconcerntheoutcomesoftheeducationalprocessesandincludestudentperformance,retentionandemployability.Productscanalsoincludepsychometricmeasuresofgenericoutcomesofhighereducation,suchasstudents’abilitytosolveproblems.insomestudiesthekeyproductmeasureisnotstudentperformance,buteducationalgain:thedifferencebetweenperformanceonaparticularmeasurebeforeandafterthestudent’sexperienceofhighereducation.Thedifferencebetweenperformanceandgainwillbecrucialinunderstandingdimensionsofquality,asweshallsee.

Thecategorisationofvariablesaspresage,processorproductisnotalwaysstraightforward.forexample,someprocessvariablessuchasthelevelofstudentengagementmayberelatedtootherprocessvariables,suchclasssize,whichmayinturnberelatedtofundinglevels.whicharethepresagevariablesandwhichtheproducts?classsizeisnotseenasapresagevariableinthe3Pmodelasitisinpartaconsequenceofpolicydecisionsabouthowtouseresourcesandinpart

dimensionsofqualiTy

13

aconsequenceofeducationaldecisionsaboutteachingmethods.Thepresagevariableofresourcesdoesnotnecessarilypredicteither.norisstudentengagementconceivedofinthe3Pmodelasaproduct.bothclasssizeandstudentengagementareconceivedofaspartoftheprocessesthatmayinfluenceeducationoutcomes,whicharecategorisedasproducts.

inexaminingtheusefulnessofpotentialperformanceindicatorsinvolvedin‘leaguetables’,presage,processandproductvariableshavesometimesbeensubdividedintomorecategorieswithinamorecomplexmodel(finnieandusher,2005;usherandsavino,2006),butforthepurposesofthisreportthesimple3Pmodelwillsuffice.

Thisreportexaminesawiderangeofpresage,processandproductvariablesinturn,andindoingsoidentifiesrelationshipsthatareknowntoexistbetweenthem.


14

4. PresaGedimensionsofqualiTy This section considers four presage dimensions of quality: funding, staf f:student ratios, the quality of teaching staf f and the quality of students

4.1 funding

institutionalfundingpredictsstudentperformancetosomeextent.itpredictscohortsize(boundandTurner,2005),andclasssizepredictsstudentperformance(seesection5.1).fundingalsoaffectsthekindofteacherstheinstitutioncanaffordtoundertaketheteachingandthisaffectsstudentperformance(seesection4.3).howmuchfundingperstudentisallocatedtotheprovisionoflearningresourcesalsopredictsstudentstudyeffort,whichinturnpredictsstudentperformance(seesection5.2).however,fundingpredictsperformancelargelybecausethebeststudentsgotothebest-resourcedinstitutionsandthequalityofthestudentspredictstheirperformance(seesection4.4).aseriesoflarge-scaleusstudieshavefoundlittleornorelationshipbetweeninstitutionalfundingandmeasuresofeducationalgain(PascarellaandTerenzini,2005).

eventheabilityofinstitutionallevelsoffundingtopredictstudentperformanceissomewhatlimited.astudyintheushascomparedgroupsofcollegeswithnearidenticalfundingperstudentandfoundgraduatecompletionratesvaryingbetween35%and70%(ewell,2008),sothedifferencesinwhatcollegesdowiththeirfundingmustbeverywide.inaddition,institutionswithsimilarlevelsofperformancedisplaywidelyvaryinglevelsoffundingwithsomereceivingonly60%oftherevenuesperstudentthatothersreceive,butachievingnearidenticalperformanceonawholerangeofoutcomemeasures.Twentyinstitutionsthathadbeenidentifiedasunusuallyeducationallyeffective,inrelationtostudentretentionandlevelsofstudentengagement,havebeenmatched,intermsofawholeraftofotherpotentiallyinfluentialpresagevariablessuchasstudentselectivity,withagroupofmarkedlylesseffectiveinstitutions.Therewasfoundtobenodifferenceintheleveloffundingallocatedtoteachingbetweentheeffectiveinstitutionsandthecarefullymatchedlesseffectivegroup(nchems,2003).

whatdistinguishedtheseeffectiveinstitutionswasthatthefundingwasuseddifferently;forexample,onfacultydevelopment(seesection5.3.1),teachingandlearningcentresandacademicsupportstaffsuchastutorsandcounsellors(seesection5.7.3).Theseusesoffundingwereexactlywhatpreviousandsubsequentstudieshavefoundtobecharacteristicof“acampusethosdevotedtostudentsuccess”(Gansemer-Topfet al.,2004).Thisissueofethosandvaluesisreturnedtointheconclusion.

dimensionsofqualiTy

15

4.2 student:staffratios

whileattheleveloftheinstitutionstudent:staffratios(ssrs)mayseemtobeaninevitableconsequenceoffundinglevels,institutionsinpracticespendfundsonbuildings,onadministration,on‘centralservices’,onmarketing,onteachersundertakingresearch,andsoon,toveryvaryingextents,ratherthanspendingitallonteachingtime.Thedoublingoftuitionfeesintheusinrecentdecadeshasnotbeenaccompaniedbyanyoverallimprovementinssrs,buthaslargelybeenusedforadministrationandmeetingaccreditationrequirements.institutionsspendverydifferentproportionsoftheiravailablefundingonteachers.sossrsmightbeseentobeamoredirectindicatorofeducationalqualitythanfunding.

lowssrsofferthepotentialtoarrangeeducationalpracticesthatareknowntoimproveeducationaloutcomes.first,closecontactwithteachersisagoodpredictorofeducationaloutcomes(PascarellaandTerenzini,2005)andclosecontactismoreeasilypossiblewhentherearenottoomanystudentsforeachteachertomakeclosecontactwith.lowssrsdonotguaranteeclosecontact,asharvard’srecentself-criticismhasdemonstrated,buttheydomakeitpossible.

second,thevolume,qualityandtimelinessofteachers’feedbackonstudents’assignmentsarealsogoodpredictorsofeducationaloutcomes(seesection5.6),andagainthisrequiresthatteachersdonothavesomanyassignmentstomarkthattheycannotprovideenough,high-qualityfeedback,promptly.again,lowssrsdonotguaranteegoodfeedbackorfeedbackfromexperiencedteachers.intheuKturn-roundtimesforfeedbackmaybeamatteroflocalpolicyratherthandrivenbyssrsandturnaroundtimesvaryenormouslybetweeninstitutions(Gibbsanddunbar-Goddet,2009).

Third,whilelowssrsdonotguaranteesmallclasses,theycertainlymakethempossible,andclasssizepredictsstudentperformance(seesection5.1below).

however,oncestudententrycharacteristicsaretakenintoaccount,educationalgainshavebeenfoundtobelargelyunrelatedtossrs(TerenziniandPascarella,1994).Thissuggestseitherthatinstitutionswithlowssrsarenotexploitingtheirpotentialadvantagesthroughtheuseofeffectiveeducationalpracticesorthatssrfigureshideothervariations,orboth.

ssrsreportedatinstitutionalleveldonotnecessarilygiveagoodindicationofthessrsstudentsactuallyexperience.Patternsofworkvary;forexample,academicsdoagreaterproportionofadministration,withfewersupportstaff,insomeinstitutions,effectivelyreducingtheiravailabilitytostudents.Theyundertakemoreresearchinsomeinstitutionswhiletheproportionoftheirresearchtimefundedbyresearchincomevaries.Thedifferencebetweenstudents’yearsofstudycanbemarked,withmuchgreaterfundingperstudentcharacteristicallybeingallocatedtothird-yearcoursesthantofirst-yearcourses,leadingtobetterssrsandsmallerclassesinthethirdyear(andthenationalstudentsurveyisadministeredinthethirdyear).furthermoreinstitutionsdonotallocatefundingtodepartments


16

instrictproportiontostudentnumbers,buttakeadvantageofwell-recruitingandincome-generatingcourses,andsoincreasetheirssrs.Theoverallconsequenceisthateffectivessrs,asstudentsexperiencethem,withinayearoftheirstudy,withinadepartment,maybearlittleresemblancetossrsreportedatinstitutionallevel.

Therearealsocommonlyexpressedconcernsabouttheaccuracyandmeaningofssrdataasreportedtoinstitutions,andasmadepublicbyhesa,duetovariationsinhowtheyarecalculated,complicatedbyvariationsinthenatureofemploymentofteachersandhowtheyaredeployed(seesection4.3below).

4.3 qualityofteachingstaff

baldssrdataareunhelpfulinthattheydisguisetherealitiesofwhothestaffarewithwhomstudentshavecontact.forexample,undergraduatesatyaleoftendonotreceivefeedbackfromtenuredfacultyuntiltheirthirdyear.inusresearchuniversitiestheteachingundertakenbygraduateteachingassistantsisaconstantqualityconcernandisregularlycitedinstudentexitsurveysastheirnumberonecomplaintaboutthequalityofteaching.

anhourofagraduateteachingassistantmaycostafractionofanhourofatenuredacademic,andmostinstitutionsarequicktoexploitthis.recentsurveys(hePi,2006,2007)revealwidevariationsbetweeninstitutionsintheproportionofteachingthatstudentsexperiencethathasbeenundertakenbyresearchstudentsasopposedtotenuredacademics.ThemajorityofsmallgroupteachingwasfoundtobeundertakenbyteachersotherthanacademicsatrussellGroupandpre-1992universities.attheuniversityofoxfordtheextenttowhichstudentstakea‘surfaceapproach’totheirstudy,emphasisingonlymemorisation(seesection5.5.2below),islinkedtotheproportionoftheirtutorialstakenbyteachersotherthancollegefellows(Trigwellandashwin,2004).amuchlowerproportionofteachingisundertakenbyresearchstudentsatoxfordthanatotherrussellGroupuniversities.

I am currently an undergraduate at a Russell Group University... I have three hour-long lectures (taught by world-class professors) and three hour-long group seminars (taught by unpaid postgraduate students) a week. That is it. If we are going to compromise social mobility and academic freedom in order to fund our universities better, we damn well ought to receive a standard of education that’s worth the sacrifice. —lettertoEducation Guardian,25may,2010

inteaching-orientedinstitutionswithfewresearchstudentswhomightdotheteaching,asignificantproportionofteachingmaybeundertakenbywhattheusterms‘adjunctfaculty’whomayhaveportfolioteachingcareersspanninganumberofinstitutions,withanofficeinnoneofthem.invocationalandcreativeartscourses

dimensionsofqualiTy

17

studentsmayexperienceasignificantproportionoftheirteachingundertakenbyprofessionalswhomayspendonlyafewhoursaweekoncampus.adjunctfacultyandprofessionalsmaybringspecialcharacteristicsandtalentstotheirwork,buttheymayneverhavetheopportunitytodevelopacourseoverseveralyearssothatitprovidesarichandeffectivelearningenvironmentforstudents.Theymaynotbepaidtomeetstudentsoutofclassortoprovidedetailedcommentsontheirassignments.Theymayneverattenddepartmentalmeetingsorfullyunderstandhowthedegreeprogrammeworkstowhichtheirteachingissupposedtocontribute.studentsmaynotknowwhothefull-timetenuredacademicsareandwhoarenot–theywillsimplyexperienceteachingofvaryingqualityandcommitment.whilethehePidatauncovervariationintheproportionofteachingundertakenbyresearchstudents–whostudentscanseearemainlyyoung–itisdifficultforstudentsurveystoidentifytheproportionofteachingundertakenbyadjunctfacultyandhePidatadonotilluminatethisissue.

intheustheproportionofpart-timeandnon-tenure-track‘adjunctfaculty’hasincreasedenormouslyinpublicinstitutions.institutionswithhigherproportionsofadjunctfacultyhavelowergraduationrateswhenotherfactorsareheldconstant.anincreaseof10%inpart-timefacultyisassociatedwithareductionof3%ingraduationrates.withininstitutions,first-yearstudentstaughttoagreaterextentbyadjunctfacultyarelesslikelytopersistintothesecondyear(ehrenberg,2006).

4.4 qualityofstudents

intheus,byfarthebestpredictorofstudents’educationaloutcomeswhetherthemeasureisgrades,apsychometrictestofprincipledreasoning,orcareersuccess,istheirschoolsaTscorewhentheyentercollege,withcorrelationsintherange0.85to0.95.inotherwordsupto90%ofallvariationinstudentperformanceatuniversitycansometimesbeexplainedbyhowtheyperformedbeforetheyentereduniversity.intheuKthelinkislessstrong,buttherehasfordecadesbeenclearevidenceoftheextensiveimpactofschoolingonstudentperformanceinhighereducation,bothintermsofschoolleavinggradesandtypeofschool(smithandnaylor,2005).intheuKstudentsfromindependentschoolsperformlesswellthandostudentsfromstateschoolswithequivalententrygrades(hoskinset al.,1997;smithandnaylor,2005).

Thequestionthatthenarisesiswhetheranyofthisenhancedperformanceisduetoqualitiesoftheinstitutionotherthantheirabilitytobehighlyselectiveonentry.large-scalelongitudinalstudiesofadiverserangeofinstitutionshavetestedstudentsabouttheiracademicbehaviourandexperience(includingtheirengagement,seesection5.5.3below)fromatotalofnearly300collegesandinvolvingdatafromnearly80,000students(forasummaryofthisworkseeKuhandPascarella,2004).Thesestudieshavefoundverylittlerelationshipbetweeneducationalselectivity(i.e.qualityofstudentintake)andtheprevalenceofwhatareknowntobeeducationallyeffective


18

practices.selectivitywasfoundtobenegativelyassociatedwithsomepractices,suchastheamountofteacherfeedbacktostudents,andevenwheretherewerefoundtobesmallpositiverelationships(forexamplewithhigherexpectationsonstudents),selectivityonlyaccountedfor2%ofthevarianceineducationalpractices.

itmightbearguedthatselectiveinstitutionsdonotneedspecialeducationalpracticesbecausetheirstudentsareableenoughtoengagethemselves.however,thedegreeofselectivitydoesnotpredictthedegreeofstudentengagement–studentsarejustasengaged(orunengaged)innon-selectiveinstitutions(Pascarellaet al.,2006).sowhileleaguetablesintheuKinvariablyincludea-levelpointscoresasanindicatorofeducationalquality,iftheusevidenceisanythingtogobytheytellusalmostnothingaboutthequalityoftheeducationalprocesswithininstitutionsorthedegreeofstudentengagementwiththeirstudies.

itmightbearguedthatthereareeducationalbenefitstoastudentofbeingsurroundedbyotherablestudents.Thiscouldraisestudents’expectationsofthemselves(onetheof‘sevenPrinciples’)1,anditisknownthatingroupworkitisthepreviouseducationalattainmentofthebeststudentinthegroupthatbestpredictsthegroupgrade,nottheaveragelevelofpriorattainmentortheleveloftheweakeststudent(Gibbs,2010).wewouldthenneedtolookattheextenttowhichtheeducationalprocessmaximiseshowstudentscouldgainfromeachother,forexamplethroughcollaborativelearning.Theextentofcollaborativelearningisagoodpredictorofeducationalgains(the‘sevenPrinciples’again).however,itwillnothelpastudentmuchiftheotherstudentsarehighlyableiftheythenengagelargelyinsolitarycompetitivelearning.Theusdatacitedabovemakeitclearthatstudentsarenotmorelikelytobeinvolvedincollaborativelearning,ortobeengagedbyit,ininstitutionswithmoreselectiveentry,inwhichthestudentsaremoreable.

studentsbringmoretohighereducationthantheira-levelscores.itislikelythattheirculturalcapital,theiraspirations,self-confidenceandmotivationsallinfluencetheirperformanceandinteractwithteachingandcoursedesignvariables.

1 The‘sevenPrinciplesofGoodPracticeinundergraduateeducation’(chickeringandGamson1987a,1987b,1991)arebasedonaverywidereviewofempiricalevidence,andhavebeenusedwidelyintheusandelsewhereasguidestotheimprovementofuniversityteaching.Theprinciplesarethatgoodpractice:encouragesstudent-facultycontact,encouragescooperationamongstudents;encouragesactivelearning;providespromptfeedback;emphasizestimeontask;communicateshighexpectations;andrespectsdiversetalentsandwaysoflearning.

dimensionsofqualiTy

19

5. Processdimensions This section considers the ef fects on educational ef fectiveness of class size, class contact hours, independent study hours and total hours, the quality of teaching, the ef fects of the research environment, the level of intellectual challenge and student engagement, formative assessment and feedback, reputation, peer quality ratings and quality enhancement processes

5.1 classsize

meta-analysisoflargenumbersofstudiesofclass-sizeeffectshasshownthatthemorestudentsthereareinaclass,thelowerthelevelofstudentachievement(Glassandsmith,1978,1979).otherimportantvariablesarealsonegativelyaffectedbyclasssize,suchasthequalityoftheeducationalprocessinclass(whatteachersdo),thequalityofthephysicallearningenvironment,theextenttowhichstudentattitudesarepositiveandtheextentofthemexhibitingbehaviourconducivetolearning(smithandGlass,1979).Thesenegativeclass-sizeeffectsaregreatestforyoungerstudentsandsmallestforstudents18orover(ibid.),buttheeffectsarestillquitesubstantialinhighereducation.lindsayandPaton-saltzberg(1987)foundinanenglishpolytechnicthat“theprobabilityofgainingan‘a’gradeislessthanhalfinamoduleenrolling50-60thanitisinamoduleenrollinglessthan20”(p218).allsubsequentuKstudieshavereportedsizablenegativecorrelationsbetweenclasssize(asmeasuredbythenumberofstudentsregisteredonacourse)andaveragestudentperformance,inmostbutnotallsubjects,andinmostbutnotallcontexts(Gibbset al.,1996;fearnley,1995).largeclasseshavenegativeeffectsnotonlyonperformancebutalsoonthequalityofstudentengagement:studentsaremorelikelytoadoptasurfaceapproachinalargeclass(lucaset al.,1996)andsotoonlytrytomemoriseratherthanattempttounderstand(seesection5.5.2ondepthofapproachtolearning).

atamicro-levelthereisevidencethattheeducationalprocessiscompromisedasclasssizeincreases.inhighereducationdiscussiongroups,forexample,awholerangeofthingsgowrongasclasssizeincreases.Thereisamuchlowerlevelofparticipationbyallbutaminorityofstudentsandthecontributionsthatstudentsdomaketendtoconcernclarificationoffactsratherthanexplorationofideas(baleset al.,1951).


20

usresearchshowsthathighereducationstudentsgiveloweroverallratingstoteachersoflargeclasses(woodet al.,1974;feldman,1984).however,therearereasonstoquestionthemeaningofthisfinding.Thesameteachersaregivenhigherratingswhentheyteachsmallerclasses.assuchratingsofteachersarerelativelyreliableandstablethissuggeststhatstudents’ratingsofteachersinlargeclassesarereflectingsomethingotherthantheteachersthemselves.aqualitativestudyofstudents’experienceoflargeclasses(GibbsandJenkins,1992)hasthrownlightonvariablesotherthantheteaching.Theremaybeintensestudentcompetitionforlimitedlibraryandotherresourcesinlargeclasses,andteachersmayhavetorelyonafewtextbooksifstudentsaretoreadanything.Theamountandpromptnessoffeedbackonassignmentsislikelytodecline,asteachertimeissqueezed.Thenatureofassessmentsmaychangefromengagingopen-endedprojectstoquicktests,asmarkingotherwisetakestoolong.closecontactwithteachersoutsideofclassandaccesstoremedialtutoringandadvicemaybemorelimited.largeclassesmaybeassociatedwithweaksocialcohesion,alienationandabreakdowninsocialbehaviour,leadingtocheating,hidinglibrarybooks,andsoon.allthisismoretodowithwhathappensoutsideofclassoncourseswithlargeenrolments,ratherthanwhathappensinclass,butitisclassroomactivitythatisthefocusofmostschool-basedresearchandushighereducationresearch.whereout-of-classstudyingisthemajorcomponentofstudentlearningthecrucialvariablemaybecourseenrolmentratherthanclasssize.usdatashowthatcohortsizeisstronglynegativelycorrelatedwithstudentperformance(boundandTurner,2005).

anotherdifferencebetweenschoolandhighereducationinrelationtoclass-sizeeffectsisthatinhighereducationtherangeofclasssizesbeingstudiedisverymuchwider:perhaps20to1,000insteadof10to40inschools.differentvariablesinevitablybecomeprominentinsuchverylargeclasses.inschool,studentsmayexperiencealltheirclassesasmuchthesamesize.inhighereducationwhatmaymattermostisnotthesizeofthelargestlecturethatisattendedonanyparticularcoursebutthesizeofthesmallestseminargrouporproblemclassthattheyattendwithinthesamecourse.openuniversitystudentsmayattendacoursewithanenrolmentofover10,000,buttheyusuallyonlyexperienceatutorgroupof24,andeachtutorusuallyhasonlyonetutorgroupsotheycangettoknowstudentsindividually.attheopenuniversityitwouldprobablymakeadifferenceifthistutorgroupwas12or48butnotiftotalenrolmentwas500or20,000.

classroomsusedforspecialistpurposes,suchaslaboratoriesandstudios,usuallylimitthenumberofstudentsitispossibletoteachatonce,regardlessofhowmanystudentshaveenrolled,andalthoughlaboratorieshavebecomemuchlarger,therearelimitstoclass-sizeeffectswhilewithinthelab.however,increasedenrolmentswithfixedspecialistspaceshavetheinevitableconsequenceofreducingtheamountoftimestudentshaveaccesstothesespecialistfacilities.Thishastransformedartanddesigneducation.insteadofstudents‘owning’apermanentspacetheycantaketimetobecomecreativeandcompetentin,whenenrolmentincreasestheyvisitashared

dimensionsofqualiTy

21

spaceoccasionally.Thenumberofstudentsinthestudioatanyonetimemaynothavechangedmuchbutartstudents’experiencehasbeenchangedoutofallrecognition.Gibbset al.(1996)foundthatinart,designandthePerformingarts,eachadditional12studentsenrolledonacoursegaverisetoadeclineof1%inaveragemarks.

negativeclass-sizeeffectsarenotinevitableandacertainamountisknownabouthowtosupportgoodqualitylearningdespitelargeclasses(GibbsandJenkins,1992).TheTeachingmorestudentsinitiativeintheearly1990strained9,500polytechnicandcollegelecturersontheassumptionthatsuchimprovementswerepossibledespitelargerclasses(Gibbs,1995).ThenationalcentreforacademicTransformationintheushashelpedscoresofinstitutionstoredesignlarge-enrolment,first-yearcourses.Theyhaveshownthatitispossibletoimprovestudentoutcomeswhilereducingteachingcontacttimeandreducingfunding.Theopenuniversityhasretainedthenumberandnatureofassignmentspercourse,theamount,qualityandturnaroundtimeoffeedbackfromtutors,andthesmallsizeoftutorgroups,throughstrictcourseapprovalrules,withcourseenrolmentsthatareseldombelow500.

Theconundrum,ofcourse,isthatintheuKoverallstudentperformancehasincreasedatthesametimethatoverallclasssizehasincreased.Thisissueisaddressedinsection6.1.

5.2 classcontacthours,independentstudyhoursandtotalhours

Thenumberofclasscontacthourshasverylittletodowitheducationalquality,independentlyofwhathappensinthosehours,whatthepedagogicalmodelis,andwhattheconsequencesareforthequantityandqualityofindependentstudyhours.independentstudyhours,toalargeextent,reflectclasscontacthours:ifthereislessteachingthenstudentsstudymoreandifthereismoreteachingstudentsstudyless,makinguptotalhourstosimilartotalsregardlessoftheratioofteachingtostudyhours(vos,1991).however,somepedagogicsystemsuseclasscontactinwaysthatareverymuchmoreeffectivethanothersatgeneratingeffectiveindependentstudyhours.areviewofdatafromanumberofstudiesbyGardiner(1997)foundanaverageofonly0.7hoursofout-of-classstudyingforeachhourinclass,inuscolleges.incontrasteachhouroftheuniversityofoxford’stutorialsgenerateonaverage11hoursofindependentstudy(Trigwellandashwin,2004)andoxford’sstudentshavebeenfoundtoputinthegreatestoverallweeklyeffortintheuKdespitehavingcomparativelyfewerclasscontacthours(hePi,2006,2007).whatseemstomatteristhenatureoftheclasscontact.‘closecontact’thatinvolvesatleastsomeinteractionbetweenteachersandstudentsonapersonalbasisisassociatedwithgreatereducationalgains(Pascarella,1980)independentlyofthetotalnumberofclasscontacthours(PascarellaandTerenzini,2005);theprovisionofclosecontactisoneofthe‘sevenprinciplesofgoodpracticeinundergraduateeducation’(chickeringandGamson,1987a,1987b,1991).


22

Seven principles of good practice in undergraduate education

Good Practice Encourages Student-Faculty Contact Good Practice Encourages Cooperation among Students Good Practice Encourages Active Learning Good Practice Gives Prompt Feedback Good Practice Emphasizes Time on Task Good Practice Communicates High Expectations Good Practice Respects Diverse Talents and Ways of Learning

—chickeringandGamson,1987a

asanillustrationofthelackofrelationshipbetweenclasscontacthoursandoutcomes,medicaleducationworldwidehasmigratedfromtraditionaldidacticpedagogies,characterisedbylargenumbersoflargeclasslectures,towardsproblem-basedpedagogies,characterisedbyamuchsmallernumberofsmallinteractiveclasses,usuallywithinthesameresourceenvelope.Thischangehasbeenaccompaniedbyasubstantialincreaseinindependentlearninghoursandevidenceofgreaterpedagogicaleffectivenessmeasuredinavarietyofways(dochyet al.,2003).similarlytheopenuniversityhasmanagedtotopthenationalstudentsurveyleaguetableswhilehavingverymuchthelowestclasscontacthoursintheuK.

Thisisnotthesameasarguingthatyoucancutclasscontacthoursfromanexistingunchangedpedagogywithoutmakinganydifferencetostudentlearning,orthatincreasinghourswillmakenodifference.ifstudentsreadprimarilyinordertodiscusswhattheyhavereadinaseminar,andtheseminaristakenaway,thentheywillprobablyreadagooddeallessandlearnlessasaconsequence.verylittleclasscontactmayresultinalackofclarityaboutwhatstudentsshouldbestudying,alackofaconceptualframeworkwithinwhichsubsequentstudycanbeframed,alackofengagementwiththesubject,alackoforalfeedbackontheirunderstanding,andsoon.itdependswhatroletheclasscontactisperforming.whatmattersisthequantityandqualityofengagementgeneratedbytheparticularusestowhichclasscontactisput.

whatseemstobemoreimportantthanclasscontacthoursisthetotalnumberofhoursthatstudentsputin,bothinandoutofclass(thequalityofthateffortisconsideredinsection5.5below).Thenumberofhoursthatstudentsstudyoutofclassisusuallymeasuredinoneoftwoways:byaskingstudentstoestimate,retrospectively,howmuchtheythinktheyhavebeenstudying,often‘onaverage’,orbyaskingstudentstokeepalog,astheystudy,ofwhattheydowiththeirtime,overarelativelyshortperiodoftime(suchasaweek).whilelogsarelikelytobemoreaccuratethanretrospectiveestimates,theyalsoruntheriskofchangingstudentstudybehaviourasstudentsbecomemoreawareoftheirpatternandlevelofeffort.indeedkeeping

dimensionsofqualiTy

23

alogisacommonlearningactivityonstudyskillscourses.whenaskingstudentstoestimatetheirstudyhoursretrospectively,theformofthequestionusedvariesbetweendifferentsurveysandthetimingofthesurveysvariesinrelationtohowlongagostudentsareattemptingtorememberorhowwideaspreadofcoursestheyarebeingaskedtomakeaverageestimatesacross.studentswhoattendlessandstudylessmaybemissedbysurveyswhileconscientiousstudentswhoattendmoreandstudymoremaybemorelikelytoreturnsurveys.Theimpactofsuchpotentialbiasesisnotwellresearchedandthereliabilityofstudy-hoursdataisnotknown.

Thequestion:‘arehigherstudyhoursassociatedwithbetterstudentlearningandperformance?’,canbeposedintworatherdifferentways.first:‘arethestudentswhostudylongerhourstheonesthatperformbest?’Theanswertothisquestionisnotstraightforward(stinebricknerandstinebrickner,2008),becauseveryablestudentsmaybeabletomeetassessmentrequirementswithouthavingtostudyveryhard,whilelessablestudentsmayputinmanyhoursunproductively(ashbyet al.,2005).Thereisalsoevidencethatstudentswho,inappropriately,takea‘surface’approachtotheirstudies(seesection5.5.2below)findthissounproductivethattheygraduallyreducetheireffortafterinitiallyworkinghardandendupstudyingfewerhoursthanstudentswhotakea‘deep’approach(svensson,1977).

if,however,thequestionisframeddifferentlyas:‘ifastudentweretostudymorehours,wouldtheyperformbetter?’oreven‘ifaveragestudyhoursonadegreeprogrammewerehigher,wouldaverageperformancebehigher?’,theanswerismuchmoreclearly‘yes’.‘Timeontask’isoneoftheevidence-based‘sevenPrinciplesofGoodPracticeinundergraduateeducation(chickeringandGamson,1987).Thereasonableassumptionhereisthatifyoudon’tspendenoughtimeonsomethingthenyouwon’tlearnit,andthatincreasingthenumberofhoursstudentsspendstudyingisoneofthemosteffectivewaysofimprovingtheirperformance.northamericanresearchanddevelopmentworkon‘studentengagement’(seesection5.5.3below)usesstudenteffortasanimportantindicatorofengagement.

Thebolognaprocesshasusedtotalstudenteffort(classcontacthoursplusindependentstudyhours)asitsmetricfordefiningthedemandsofabachelorsdegreeprogramme,setat1,500to1,800hoursayear:4,500to5,200hoursoverthreeyears.aseriesofstudieshavefoundthatuKstudents’totalweeklyeffortinhoursislowerthanincomparisonwitheithertheparticulareuropeancountriesstudiedorincomparisonwithoveralleuropeannorms(brennanet al.,2009;hochschul-informations-system,2005;sastryandbekhradnia,2007;schomburgandTeichler,2006).Thesefindingsdeservetobetakenseriouslybecausetheyarerelativelyconsistentacrossdifferentstudiesandmethodologies,carriedoutindifferentcountries,oracrossrepetitionsofthesamestudyindifferentyears.

itshouldbepossibletoironoutgrossdifferencesbetweeninstitutionsandsubjectareas,asthenumberofstudyhourspercredit,andhencethenumberofhoursrequiredforabachelorsprogramme,areclearlydefinedincoursedocumentation.however,therangeinweeklystudyeffortbetweenenglish


24

institutions,withinsubjects,foundinthehePistudiesiswide,forexamplefrom14hoursaweektonearly40hoursperweekwithinPhilosophy(sastryandbekhradnia,2007).differencesbetweensubjectsarealsowide.broaddifferencesintotalstudyhoursbetweenscienceandtechnologyprogrammes(whichtendtohavebothhighclasscontacthoursandweeklydemandsforworksuchasproblemsheetsandlaboratoryreports)andthehumanities(whichtendtohavebothlowerclasscontacthoursandlessregularassignmentssuchasessays)arewellknownandhavebeenreportedfrequentlyovertheyears(e.g.vos,1991).however,thedifferencesbetweensubjectsidentifiedbythehePisurveysaresubstantial,withsomesubjectshavingnationalaverageweeklystudyeffortsofonlyaround20hoursperweek.TwentyhoursperweekwithinthecomparativelyshortuKsemestersequatestoaround500hoursayear:onethirdoftheminimumspecifiedunderthebolognaagreement.Toachievethebolognaspecificationofaminimumof4,500hoursforabachelorsprogramme,studentsinthesesubjectsintheuKwouldhavetostudyfornineyears.differencesonthisscalecannoteasilybearguedawaybyclaimingthatuKstudentsaresomehowinherentlysuperiororthatuKeducationalpracticesaresomehowinherentlymoreefficient,intheabsenceofanyevidencetobackupsuchclaims.

asurveyofinternationalstudentswhohaveexperiencedbothauKhighereducationinstitutionandanothereuhighereducationinstitution(brennanet al.,2009)foundthatsuchstudentsaremorelikelytorateuKbachelorsprogrammesas‘lessdemanding’andlesslikelytoratethemas‘moredemanding’,afindingthatdoesnotjustifythelowernumberofhoursinvolved.uKstudentshavebeenreportedtohavedonemoreworkthanwasrequiredofthemtoagreaterextentthaninanyothereuropeancountry(ibid.).yetthetotalnumberofhoursstudiedintheuKisstillbeloweuropeannorms,whichsuggeststhattheuKrequirementsmustbelower.

ifitwerethecasethatlessablestudentsneededtostudymore,thenonewouldfindthelargerstudyhoursfiguresininstitutionsthathavestudentswithweakereducationalbackgrounds.insteadthereverseisthecase,withhigherweeklystudyhoursreportedininstitutionswithstudentswiththestrongesteducationalbackgrounds(sastryandbekhradnia,2007).Themostlikelyexplanationthereforeisthatthedemandsmadeonstudentsaredifferentindifferentinstitutions,andthatevenweakerstudentsareabletomeetthesedemandswhilestudying,insomeinstitutions,andinsomesubjects,athirdofthehoursthebolognaagreementspecifies.

Thereareanumberofpossibleexplanationsofwhysuchstudentsmightstudyfewhours:

— somedegreeprogrammesrelyheavilyoncourseworkassessment,withover90%ofmarksonsomedegreeprogrammescomingfromcourseworkratherthanonexaminations.whilethismayhavecomeaboutforsoundeducationalreasons,itmayallowstudentstostudyonlyoneortwotopicsindepthpercourse,becausethatisalltherearetheteachingresourcestomark,reducingstudenteffortinweekswhentherearenoformal

dimensionsofqualiTy

25

assessmentdemands.studentshavebeenfoundtoworkmoreregularlyandcoverthesyllabustoagreaterextentwhenthereisahigherproportionofmarksfromexaminations(Gibbsandlucas,1997).

— highlevelsofdetailincoursespecifications,oflearningoutcomesandassessmentcriteria,inresponseinparttoqaacodesofpractice,allowstudentstoidentifywhattheyoughttopayattentionto,butalsowhattheycansafelyignore.arecentstudyhasfoundthatinsuchcoursesstudentsmaynarrowtheirfocustoattentiontothespecifiedassessedcomponentsattheexpenseofeverythingelse(Gibbsanddunbar-Goddet,2007).studentshavebecomehighlystrategicintheiruseoftimeandadiarystudyhasfoundstudentstoprogressivelyabandonstudyinganythingthatisnotassessedastheyworktheirwaythroughthreeyearsoftheirdegree(innisandshaw,1997).

— studiesoftheeffectsofpaidworkundertakenbyfull-timestudentshavereportedasubstantialreductioninstudyhoursinrelationtotheextentoftheirpaidwork,althougheffectsvarysomewhatbetweencontextsanddegreeprogrammes(carneyet al.,2005;curtisandwilliams,2002;fordet al.,1995;huntet al.,2004;Paton-salzbergandlindsay,1993).atypicalstudyisthatofsvanumandbigatti(2006),whofoundthatpaidworkreducedcourseeffortandreducedgrades.intheus,studentsrespondtotheneedto‘worktheirwaythroughcollege’bytakingfewercreditsatatimeandgainingtheirdegreeovermoreyears(andacrossseveralinstitutions)asaconsequence.intheuKfull-timestudentsseemtoassumethattheycanundertakepaidworkwhilemeetingthedemandsofafull-timecourseandstillgraduateinthreeyears.

— studentswholiveathome,ratherthanonaresidentialcampus,arelikelytoexperiencecompetingdemandsontheirtime,andlesssocialandacademicintegration(Tinto,1975).TheinstitutionsintheuKwiththelowestaveragestudyhoursincludeuniversitiesinurbanconurbationswithasubstantialproportionofstudentslivingathome.

— Theuniversitieswithlowaveragestudyhoursareoftenalsoinstitutionswithlowannualinvestmentperstudentinlibrariesandotherlearningresources.Thiswouldmakeitmoredifficultforstudentstogainaccesstotheresourcestheyneedfortheirstudy:thebookwillbeoutandthestudyspacewithacomputerwillbeoccupied.datafromhesa,hePiandthenationalstudentsurveyhavebeenanalysedforthepurposeofthecurrentreportonthisissue.Theyshowedthatinstitutionalfundsallocatedtolearningresources,perstudent,predicttotalstudentlearninghours(withcorrelationsof+0.45forthesocialsciencesandhumanitiessubjectsanalysed).fundingforlearningresourcesalsopredictsaveragestudents’responsestothenssquestiononthequalityoflearningresources,


26

althoughlesswell.Theinstitutionwiththehighestweeklyaveragestudyhoursalsohasthegreatestannualinvestmentinlearningresourcesandthehighestnationalstudentsurveyratingsfor‘learningresources’:theuniversityofoxford.

whilethefocusofthisreportisonundergraduateprogrammes,therehasrecentlybeenagooddealofattentionpaidtotherelativequalityofuKmasterslevelcourses,giventhattheyareusuallyplannedtobeconsiderablyshorterindurationthantheirmainlandeuropeancounterparts.forexample,thechairoftheuKcouncilforGraduateeducationhasarguedthatthesegrossdifferencesdonotmatterbecauseuKmasterscoursesare‘moreintensive’,andclaimedthattheoverallamountoflearningtimeisroughlyequalbetweentheuKandmainlandeurope(olcott,2010).Thisunsubstantiatedclaimcouldbecheckedbyrepeatingthesastryandbekhradniaundergraduatestudyofstudyhours,citedabove,inmasterscoursesandaddingquestionstomeasuretheextenttowhichstudentstakeadeepapproachtotheirstudies(seesection5.5.2below).

5.3 qualityofteaching

5.3.1. qualityofteaching:experienceandtraining

Teacherswhohaveteachingqualifications(normallyaPostgraduatecertificateinhighereducation,orsomethingsimilar)havebeenfoundtoberatedmorehighlybytheirstudentsthanteacherswhohavenosuchqualification(nasret al.,1996).Thisfindingwasinacontextwhereobtainingsuchaqualificationwaslargelyvoluntary,andthosewhohavethequalificationmightbeconsideredtobedifferentinsomewayfromthosewhohavenot,andthiscouldbearguedtoinvalidatethecomparison.Thedifferencemightconcerntheextentofprofessionalismorcommitmenttoteaching,butneverthelesstherewasnocontrolgroupinthestudy.alongitudinalstudythatovercomesthisobjectionhasexaminedtheimpactovertimeonstudents’ratingsoftheirteachers,andonteachers’thinkingaboutteaching,of(mainly)compulsoryinitialtrainingduringtheirfirstyearofuniversityteaching,ineightcountries.itfoundimprovementsoneveryscaleofthe‘studentevaluationofeducationalquality’,aquestionnairedevelopedintheus(marsh,1982)andtestedforuseintheuK(coffeyandGibbs,2000),andimprovementsinthesophisticationofteachers’thinking(asmeasuredbythe‘approachestoTeachinginventory’,ameasureofteachingthatpredictsthequalityofstudentlearning,Trigwellet al.,2004).Thisimprovementinmeasuresofteachingqualitycouldnotbeattributedtomerematurationorexperienceasteachersinacontrolgroupininstitutionswithoutanyinitialtrainingwerefoundtogetworseovertheirfirstyear,onthesamemeasures(Gibbsandcoffey,2004).ThequalityofschoolteachingintheuKisinpartunderpinnedbyabeliefthatinitialorin-servicetrainingisessentialandthe,admittedlylimited,evidencesuggeststhatthesamebeliefwouldbejustifiedinuKhighereducation.

dimensionsofqualiTy

27

5.3.2. qualityofteaching:researchrecord

Thereislittleornorelationshipbetweenmeasuresofthequalityorquantityofteachers’researchandmeasuresofthequalityoftheirteaching(forareviewof58studiesoftheevidence,seehattieandmarsh,1996).

…the common belief that teaching and research were inextricably intertwined is an enduring myth. At best teaching and research are very loosely coupled. —hattieandmarsh,1996,p529

someexcellentresearchersmakeexcellentteachersandsomedonot.despitecritiquesofthemeasuresofresearchandteachingthatarenormallyused,noneofthecriticshavemanagedtodeveloporusealternativemeasuresthatdemonstratearelationshipbetweenresearchandteaching.aminorityofundergraduatestudentshavebeenreportedtovaluetheirteachersbeingactiveresearchersprovidedthisdoesnotinterferewiththeirstudies(forexample,throughtheirteacherbeingabsentwhileundertakingresearch)(lindsayet al.,2002),butthereisnoevidencethatthisimprovestheirlearning.

5.3.3. qualityofteaching:judgedbystudents

despitethecommondisparagementofstudentratingsofteachers,theycanbehighlyreliable(inthesensethatstudentsagreewitheachotheraboutwhothebestteachersare,agreewithteachers’peers,andmakethesamejudgementsondifferentoccasions)andrelativelyimmunefrombiasesofvariouskinds.studentscanreadilytellthedifferencebetweenteacherstheylikeandteacherswhotheythinkaregoodteachers,andthecommoncriticismthatstudentfeedbackissimplyapopularityparadeislargelyunfounded(seemarsh,1987,forareviewofthevastliteratureonthereliabilityandvalidityofstudentfeedbackquestionnaires).Theseobservationsarebasedontheuseofthoroughlydevelopedandtestedfeedbackquestionnaires.however,intheuKalmostallsuchquestionnairesare‘home-grown’andarelikelytobeofdoubtfulreliabilityandopentoallkindsofbiases.differentquestionnairesareusedindifferentinstitutionsandthereisthereforenobasisforcomparisonofthequalityofteachersbetweeninstitutionsorsubjects.ifstudentratingsofteacherswereconsideredausefulindicatorofcomparativequalitythentherewouldneedtobeanagreementforeveryonetousethesamequestionnaire,suchasthestudentevaluationofeducationalquality(seeq)(marsh,1982;coffeyandGibbs,2000).

severalwell-developed,reliable,studentfeedbackquestionnaireshavereasonablelevelsofvalidityinthatscoresonscaleswithinthequestionnaires(involvingaddingratingsfromclustersofsimilarquestions)predictaspectsofstudentlearningprocess


28

(suchastheextentofstudyingfollowingteaching),learningoutcomes(suchasgrades)andotherworthwhileconsequences(suchasthelikelihoodofstudentschoosingtostudyfurthercourseswiththesameteacher).Theproportionofvarianceinsuchmeasuresoftheproductsofgoodteaching,thatisexplainedbystudentratings,variesacrossdifferentquestionnairescalesanddifferentmeasuresofproducts,butitisusuallyhighenoughtotakemeasuresofteachingbasedonstudentratingsseriously(abramiet al.,1990).

Thereisanimportantdistinctiontobemadeherebetweenstudentratingsoftheextenttowhichteachersengageinactivitiesthatareknowntoimprovelearning(suchasprovidingenoughpromptfeedbackonassignments),whichtendtobereliableandvalid,andglobaljudgementsofwhetherteachingis‘good’,whichareopentoallkindsofsubjectivevariationintheinterpretationofwhat‘good’means.studentsalsochangeovertimeintheirsophisticationaslearners,forexampleintheirconceptionoflearning(säljö,1979)andintheirconceptionofknowledge(Perry,1970).astheychange,sotheirconceptionsofwhat‘goodteaching’consistsofevolve(vanrossumet al.,1985).whatanunsophisticatedstudentmightconsidertobegoodmightconsistoftheteacherprovidingallthecontentinlecturesandthentestingformemoryofthatcontent,whileamoresophisticatedstudentmightseegoodteachingasinvolvingsupportingindependentlearningandthedevelopmentofapersonalstancetowardsknowledge.whatunsophisticatedstudentswanttheirteacherstodoisoftenbadfortheirlearningandrespondingtotheirglobalratingsuncriticallyisnotthewaytoimprovequality.whenasurveyreportsasingleglobalratingoftheextenttowhichstudentthinkalltheteachingoverthreeyearsissimply‘good’,theseverydifferentstudentconceptionsofgoodteachingaremuddledtogetherandtheaverageratingisthenverydifficulttointerpret.incontrastso-called‘lowinference’questionsthatrefertospecificteacherbehaviours,suchasthepromptnessoftheirfeedback,aremucheasiertointerpret.

5.4 researchenvironment

aswehaveseenabove(insection5.3.2)thereisnorelationshipbetweenmeasuresofanindividualacademic’sresearchandmeasuresoftheirteaching.however,itcouldbearguedthatitisnotindividualresearchers’teachingthatmattershere,buttheresearchenvironmentgeneratedbythemajorityofteachersinadepartmentbeingresearchactive.Thismightbeconsideredapresagevariable,butasweshallsee,whatmattersistheeducationalprocess,notpriorresearchperformance.

atthelevelofdepartmentswithinaninstitutionthesituationisthesameasitisatthelevelofindividualteachers.Thebestresearchdepartmentsmayormaynotbethebestteachingdepartments:thereisnocorrelationbetweenmeasuresofadepartment’sresearchandmeasuresofitsteaching(ramsdenandmoses,1992).Therearesuggestionsthattheremayberelationshipsbetweentheextentof

dimensionsofqualiTy

29

researchintensivenessofadepartmentandstudents’senseofbenefit,inawaythatisnotmediatedbytheteaching,butthisisinanycaseonlyforsomestudents(e.g.dunbar-GoddetandTrigwell,2006),andthisrelationshipappearstobeafunctionofthefocusofattentionofthestudentsratherthanaconsequenceofdifferentorbetterteaching:ifstudentsareinterestedinscholarshiptheywillbemorelikelytosaythattheybenefitfromascholarlyenvironment.

atthelevelofwholeinstitutions,thosewherethereisastrongorientationtowardsresearchoftenrevealaweakemphasisonteaching,andviceversa–thereisastrongnegativerelationshipinrelationtoinstitutionalprioritiesandthishasmeasurableeffectsoneducationalgains:

… a college whose faculty is research-orientated increases student dissatisfaction and impacts negatively on most measures of cognitive and affective development. —astin,1993,p363

Thereisevidencethattheresearchenvironmentcanimpactpositivelyonundergraduatestudents,inawaythatcanbemeasured,wheremaximisingthebenefitstoundergraduatesofresearchstrengthshasbeenmadeadeliberatepriority.forexample,miTenablesabout80%ofitsundergraduatestoengageinarealresearchprojectasaninternorjuniorresearchstudentplacedwitharesearchgroup,andtheyhavegoodevidenceconcerninghowstudentsbenefit(bergrenet al.,2007).Thenationalsciencefoundationintheushassimilarevidenceacrossmanyinstitutionsthatsuchinitiatives:

… have a major impact on most participants’ confidence and their understanding of research-related issues, increase their interest in careers in research and science and engineering, and lead them to raise their degree expectations. —ramaley,2004,citedinJenkins,2004

(seealsobauerandbennett,2003;hathawayet al.,2002.)Thekeypointhereisthatsuchbenefitshavetobedeliberatelyengineered–theydonotaccruebymagicsimplybecauseresearchisgoingonaswellasteaching.Theinstitutionalindicatorofqualityinthesestudiesistheexistenceofanundergraduateresearchopportunitiesscheme,notthestrengthoftheinstitution’sresearch.similarlythepositiverelationshipfoundattheuniversityofoxfordbetweenstudents’experienceofresearch-activestaffandtheextenttowhichtheytakeadeepapproachtolearning(Trigwell,2005)isaconsequenceofthecollegialsystemfosteringactiveinclusioninacommunityof(research)practice,notsimplyoftheexistenceofaresearchenterprise.

forthesereasonsdepartmentalraescoresorothermeasuresofresearchactivityorperformanceintheenvironmentstudentsstudywithinarenot,ontheirown,validindictorsofeducationalquality.


30

5.5 levelofintellectualchallenge

itcanbearguedthataninstitution,orevenanentirehighereducationsystem,isofhigherqualitybecauseitoffersagreaterlevelofintellectualchallenge.inthissectionthenotionofintellectualchallengeisoperationalisedinthreeways:challengeasdeterminedbythelevelofthecurriculumstudentsarestudying,challengeasindicatedbythedepthofapproachstudentstaketotheirstudies,andchallengeasreflectedinthelevelofstudentengagementwiththeirstudies.

5.5.1. levelofthecurriculum

insomesubjectareaswithrelativelyhierarchicalknowledge,formalcoursedocumentationcanbeveryinformativeaboutthelevelofthecurriculum.itisthenpossibletocomparethelevelofintellectualchallengebyexaminingcoursedescriptionstoseehowfarthroughthesubjectstudentsareexpectedtoprogress.itisinprinciplepossibletostudycurriculumdocumentationwithinwell-defineddisciplines,fromdifferentperiods,inordertoascertainwhethertherehasbeenawateringdownofintellectualdemandsovertime.Thereareseverallimitationstothiskindofstudy:

— documentationchangesovertimeinwhatisdescribedandinwhatdetail,andalsodiffersbetweeninstitutions;

— fashionschangeovertimewithinsubjectsaboutwhatshouldbeincluded,whichmakeschangesoflevelofdemanddifficulttojudge;

— eveninsupposedlyhierarchicalsubjectsitissometimesdifficulttoagreewhatisnecessarytostudyfirst,orwhatcountsasmoreadvanced.inmedicine,forexample,problem-basedlearningmaynowadaysconfrontfirst-yearstudentswithproblemsofacomplexitythatinpastcurricula,thathadcleardivisionsbetweenpre-clinicalandclinicalstages,onlyfourth-yearstudentsencountered.however,thefirst-yearstudentsnowtacklethesecomplexproblemsatalowerlevel,withalesserknowledgebase.calibratingtheleveloftheproblemandthelevelatwhichtheyaretackledisnotstraightforward.

neverthelessthereisprobablymorescopefordeterminingqualitybyexaminingcurriculumdocumentationthaniscurrentlyexploited,provideditisundertakenbythosewithindisciplinarycommunitiesthatunderstandthecomplexitiesinvolved.

Thereareseveraltypologiesforeducationalobjectivesorlearningoutcomes(forexamplethatofbloomet al.,1956)thatcouldbeusedtocomparethelevelofdifficultyofwhatstudentsaresupposedtodowiththecontent(forexample,rememberit,applyitorcritiqueit).itispossibletodistinguishlevelsofintellectualdemandbetweendifferentlevelsofbTeccoursesinthesamesubjectbycomparing

dimensionsofqualiTy

31

theproportionofeducationalobjectivesspecifiedateachofanumberoflevels(suchas‘knowledge’and‘synthesis’).higherlevelsofcoursesaresupposedtohaveahigherproportionoftheircurriculaobjectivesatahigherlevelinthehierarchy.Proposalsfornewcoursescanbeturneddowniftoomanyobjectivesaresetattoolowalevel.

whereacurriculumauthorityhasthepowertoimposeacommonsetofdefinitionsandacommonformatofdocumentation,asbTecdoes,thisisjustaboutpossibletoimplement.whereeachinstitutionandeachdegreeprogrammehasthefreedomtospecifylearningoutcomesindependentlyofanycommonconceptualframeworkorsetofdefinitions,suchcomparisonsbecomeimpossible,exceptinanimpressionisticway.forlearningoutcomesthatconcerngenericskills,suchas‘communication’and‘groupwork’,consistentjudgementoflevelsagainststandardsisevenmoredifficult,independentlyofexemplarsthatgobeyondcurriculadocumentation.alvernocollegeintheushasdemonstratedthatevenforsuchgenericoutcomesorcapabilities,itispossibletodefinestandardsinunambiguouswaysandtotrainteacherstousethesestandardsinconsistentways,butunlessalverno’sapproach,orsomethinglikeit,weretobeadoptedbyall,comparisonacrosscontextswouldstillnotbepossible.

neverthelessthereisplentyofscopeforsubjectassociationstomakemoreuseofcurriculumdocumentationinjudgingtheintellectualdemandsofprogrammes,atleastinprinciple.Tojudgetheminpracticeitisalsonecessarytoexaminetheproductsofstudentlearning(seesection7.7).

5.5.2. depthofapproachtostudying

inthe1970sferencmartonandhiscolleaguesinGoteborgdistinguishedbetweena‘surfaceapproach’tolearninginwhichstudentsintendtoreproducematerial,anda‘deepapproach’inwhichstudentsintendtomakesenseofmaterial:adistinctionbetweenafocusofattentiononthesignorwhatissignified.Toillustratetheconsequencesforstudentlearningoutcomes,astudentwhotakesasurfaceapproachtoreadinganarticlewithaprinciple-examplestructure(suchasacasestudy)mayremembertheexample,whilethestudentwhotakesadeepapproachismorelikelytounderstandtheprinciple(martonandwenestam,1978).asurfaceapproachhasbeendemonstratedinawidevarietyofstudiestohavedepressinglylimitedandshort-lastingconsequencesevenformemoryoffacts.adeepapproachisessentialforlong-termandmeaningfuloutcomesfromhighereducation(seeGibbset al.(1982)andmartonet al.(1984)foroverviewsofthisliterature).

studentsarenot‘surfacestudents’or‘deepstudents’–approachtolearningisinthemainacontext-dependentresponsebythestudenttoperceiveddemandsofthelearningcontext(ramsden,1979).Therelevancetodimensionsofqualityisthatitispossibletoidentifythosefeaturesofcoursesthatfosterasurfaceoradeepapproach.studentstendtoadoptasurfaceapproachtoagreaterextentwhenthere


32

is,forexample,anassessmentsystemthatrewardsmemorisation,suchassuperficialmultiple-choice-questiontests.incontraststudentstendtoadoptadeepapproach,forexample,whentheyexperiencegoodfeedbackonassignments,andwhentheyhaveaclearsenseofthegoalsofthecourseandthestandardsthatareintendedtobeachieved.Theseinfluentialcharacteristicsofcoursesarethefocusofthecourseexperiencequestionnaire(ceq)(ramsden,1999),originallydevelopedinstudiesatlancasteruniversityinthe1970s,throughwhichstudentsindicatetheextenttowhichthesecoursefeaturesareexperienced.reasonablycloserelationshipshavebeenfoundbetweenscoresonscalesoftheceqandtheextenttowhichstudentstakeadeepandsurfaceapproachtotheirstudies,andsoceqscalescoresthatfocusoncertaincoursefeaturescanactasaroughproxyforeducationaloutcomes,becauseapproachpredictsoutcomestosomeextent.Theceqbecamethebasisofthequestionnaireusedannuallythroughoutaustralianhighereducationtomeasurecomparativequalityofdegreeprogrammes,publishedinannualreportsaimedatstudents.ithasbeenusedforsomeyearswithinsomeinstitutionsasaperformanceindicatorforallocatingaproportionoffundingforteachingtodepartments,asattheuniversityofsydney.ithasnowbeenadoptednationallyinaustraliaasonecomponentofperformanceindicatorsforallocatingovera$100millionofteachingfunding(in2008)touniversitieseachyear.ithasbecomethedrivingforcebehindevidence-basedinstitutionaleffortstoimproveteachingthatfocusoncoursedesignratherthanonindividualteacher’sskills(barrieandGinns,2007).amodifiedversionoftheceq(theosceq)hasbeenusedannuallyattheuniversityofoxford.

itisoftenassumedthatthevalidityofthenationalstudentsurvey(nss)isbasedonthesameresearchandevidence.uptoapointthisistrue.however,thecharacteristicofstudents’intellectualengagementwiththeirstudyingthatbestpredictstheirlearningoutcomes,theextenttowhichtheytakeadeepapproach,isnotincludedasascaleinthenss(andnorisitintheceq).somecharacteristicsofwhathavebeenfoundtobeeffectivecourses,suchasconcerningfeedback,areincludedinthenss.however,mostofthescalesoftheoriginalversionoftheceqthatrelatesomewhattotheextenttowhichstudentstakeadeepapproach,suchas‘clearGoalsandstandards’or‘appropriateworkload’arenotincludedinthenss(andneitheraretheyinthemostrecentversionsoftheceq).infactbothquestionnaireslackmostofthescalesthatwouldstrengthentheirvalidity.Themissingscalesarecurrentlyincludedasoptionsinbothquestionnaires,butthismeansthatcomparabledataarenotpublishedoravailableforcomparisonbetweeninstitutionsorcourses.

evensomeofthesemissingscaleshaveasomewhattenuousclaimtovaliditytoday.forexampleinthe1970sitwasfoundthatifstudentsweregrosslyoverburdenedthentheymightabandonadeepapproachandadoptasurfaceapproachtotheirstudies.however,30yearslaterexcessiveworkloadseemsadistantmemory(seesection5.2),sothe‘appropriateworkload’scalenolongerseemslikelytopredict,toaworthwhileextent,whichstudentswilladoptasurfaceapproach,andhencetheirlearningoutcomes.

dimensionsofqualiTy

33

Therehavebeennorecentstudiestoconfirmtheoriginalfindingsconcerningrelationshipsbetweenfeaturesofcourses,studentresponsesandlearningoutcomesincurrentcontexts.Therehavebeennodirectstudiesofthevalidityofthenssinrelationtoitsabilitytopredicteducationalgains.Therehavebeennostudiesthatdemonstratethatifevidence-basedpracticesareadopted,andnssscoresimprove,thiswillbeassociatedwithimprovededucationalgains.forthatkindofevidencewehavetolooktomeasuresofstudentengagement.

5.5.3. studentengagement

Thereiscurrentlyahighlevelofinterestintheusinmeasuringstudentengagementasacrucialindicatorofeducationalquality,embodiedintheuseofaquestionnaire:thenationalsurveyofstudentengagement(nsse).Theresearchunderlyingthensse,anditswide-scaleuseandperceivedvalue,hasinvolvedthreemainstages.

first,verylarge-scalestudiesexaminedahugerangeofprocessvariablesoncampusandalsoassessedahugerangeofmeasuresofeducationalgain,acrossaverywiderangeofcoursesandinstitutions,toidentifywhichprocessvariablesrelatetoanyofthemeasuresofgain.noresearchencompassingsomanyvariablesatonce,oronasimilarscale,hasbeenconductedoutsideoftheus.Thesevaststudieshavebeenrepeatedoverthreedecadesandverysimilarfindingshaveidentifiedthesamefewprocessvariablesthataremostcloselylinkedtoeducationalgains(PascarellaandTerenzini,2005).essentiallythecrucialvariableis‘studentengagement’andithasprovedpossibletoidentifytheprocessvariablesinvolvedinengagingstudents,suchasthelevelofacademicchallenge,theextentofactiveandcollaborativelearningandtheextentandqualityofstudent-facultyinteraction.Theseprocessvariablesareprominentintheinfluentialevidence-based‘sevenprinciplesofgoodpracticeinundergraduateeducation’elaboratedabove(chickeringandGamson,1987,1991).

Theseprincipleshavebeenusedbymanyinstitutionsintheusasindicatorsofqualityinreviewingandimprovingtheireducationalpractices,andtherehasbeenmuchpoolingofinstitutionalexperienceinusingtheseprinciples.Pascarellaet al.(2008)listaplethoraofstudiesthatdemonstratethatifyouactonthe‘sevenprinciplesofgoodpractice’thenthisimprovesstudentoutcomes.

second,aquestionnairehasbeendevelopedthroughwhichstudentscanindicatetheextenttowhichthesecrucialprocessvariableshavebeenexperiencedandthewaysinwhichtheyareengagedintheirstudying:thenationalsurveyofstudentengagement(nsse,2007).Thisquestionnairehasbeenusedwidely–by774universitiesandcollegesin2008alone.ithasalsobeenwidelyusedtomonitorstudentengagementaschangeshavebeenmadetoeducationalpracticesinattemptstoimprovestudentlearning.sufficientstudiesofappropriaterigourhavebeenpublishedtomakeitpossibletobringthedatatogetherandmakegeneralisableconclusions.meta-analysisofstudiesofinnovationandchangesupporttheoriginal


34

identificationbytheearlierresearchofapredictablerangeofeducationalprocessesthat,whenadoptedorenhanced,makeameasurabledifferencetostudentsengagement(carininet al.,2006).

Thethirdstageofthisresearchhasinvolvedvaliditystudies–examiningifthereisarelationshipbetweenengagementandeducationaloutcomes:betweenscoresonthensseandbothstudentperformanceandpsychometricmeasuressuchasofeffectivereasoningandproblemsolving,‘inclinationtoinquire’andlifelonglearning.anumberofstudieshaveshownclearlinksbetweennssescoresandoutcomessuchasfirst-yearsuccess(e.g.lanasaet al.,2007).Pascarellaet al.(2010)havegonefurtherandinamuchlargerstudy,involving19collegesofawidevarietyoftypes,demonstratedstrongrelationshipsbetweennssescalescoresandarangeofeducationalgains,involvingbeforeandaftermeasurements.Theauthorsconcluded:

NSSE results regarding educational practices and student experiences are good proxy measures for growth in important educational outcomes.

inotherwordsifyouwanttoknowthe‘valueadded’bystudents’highereducationexperiencethenthenssewillprovideagoodindicationwithoutneedingtousebeforeandaftermeasuresofwhathasbeenlearnt.

itisinterestingtonote,withreferencetotheself-imposedlimitationsofthenssandceq,thatthescaleonthenssethathastheclosestrelationshipwitheducationalgainsconcerns‘deeplearning’(Pascarellaet al.,2008).

5.6 formativeassessmentandfeedback

Theeducationalinterventioninschoolsthathasmoreimpactonstudentlearningthananyotherinvolvesimprovingformativeassessmentandespeciallytheprovisionofmore,betterandfasterfeedbackonstudentwork(blackandwiliam,1998;hattieandTimperley,2007).‘Goodpracticeprovidespromptfeedback’isoneoftheevidence-based‘sevenprinciplesofgoodpracticeinundergraduateeducation’(seeabove).ondegreeprogrammeswherethevolumeofformativeassessmentisgreater,studentstakeadeepapproachtotheirstudiestoagreaterextent(Gibbsanddunbar-Goddet,2007)anddeepapproachisagoodpredictoroflearningoutcomes(seesection5.5.2above).enhancedfeedbackcanalsoimprovestudentretention(yorke,2001).

Thenumberofoccasionsduringathree-yearbachelorsprogrammeintheuKonwhichstudentsarerequiredtoundertakeanassignmentpurelyforthepurposeoflearning,withfeedbackbutwithoutmarks,varieswidelybetweeninstitutions.onestudyhasfoundarangefromtwiceinthreeyearsatoneenglishuniversitytoover130timesatanother(Gibbsanddunbar-Goddet,2009).inanotheruKstudyusingthesameassessmentauditmethodology(TesTa,2010),thevolumeof

dimensionsofqualiTy

35

writtenfeedbackonassignmentsoverthreeyearsvariedfrombelow3,000wordsperstudenttoabove15,000words,andfororalfeedbackvariedfrom12minutesperyearperstudenttoovertenhoursperyear(Jessopet al.,2010).Thesearemuchwidervariationsbetweeninstitutionsthanexistintheirfundingperstudent,theirssrs,theirclasscontacthoursortheirindependentstudyhours.Theissueaddressedbythenssthatrevealsthegreatestareaofstudentdisquietisfeedback.

asresourcesperstudenthavedeclinedtherehavebeeneconomiesofscaleinteachingthataredifficulttoachieveinassessment:assessmentcostsgoupprettymuchinproportiontothenumberofstudents.Thisplacesenormoustimepressuresonteachers.qualityassurancesystemsinmostinstitutionshavenotpreventedthevolumeofformativeassessmentfromdecliningsubstantially,despitetheqaa Code of practice.anexceptionisTheopenuniversitywherethenumberofassignmentspermodule,andthevolumeandqualityoftutorfeedbackonallassignmentshavebeenmaintainedover30years.Thishasbeenachievedacrossallcoursesbyformalrequirementsoftheircourseapprovalprocessandbyseveralqualityassuranceprocesses.Theopenuniversityhasexceptionallyhighnssscoresforassessmentandfeedback.

5.7 otherprocessdimensionsofquality

5.7.1. reputation

seekingtheviewsofresearchpeersisacommonmethodusedtojudgeadepartmentoruniversity’sresearchqualityandthesamemethodologycouldinprinciplebeusedtojudgeeducationalquality.Thehighlyinfluentialuniversityrankingsystemintheusprovidedbytheusnewsandworldreport,‘america’sbestcolleges’,investsheavilyinsurveysofdeansandPresidentsinestablishingcollegereputations.however,thereputationalrankingthatderivesfromthesesurveyscorrelatescloselywiththesizeofinstitution’sfederalresearchgrants(GrahamandThompson,2001)andcanalsobepredictedbyundergraduateselectivity,perstudentexpenditureandnumberofdoctoralawardingdepartments(astin,1985),noneofwhichpredicteducationalgains.reputationaldatahaveaverypoorreputationasavalidindicatorofeducationalquality.

5.7.2. Peerratings(wherethesefocusonprocessdimensions)

manyqualityassurancesystemsmakeuseofexpertpeerjudgementofthequalityofeducationalprovisioninadegreeprogramme,atthetimeofaperiodicreviewofsomekind,basedonawiderangeofevidenceanddocumentationandsometimesincludingobservationofteaching.Therelationshipbetweentheseratingsandtheevidenceonwhichtheyarebasedisnoteasytoestablishastheyareinherently


36

subjectiveandglobal,andbasedondifferentcombinationsofevidence,withdifferentweightings,indifferentcontexts,bydifferentgroupsofpeers.however,theremaybepotentialfortheapplicationofprofessionalexpertiseinsuchsubjectivejudgementstoreachmorevalidconclusionsthancouldbeachievedmerelyonthebasisofindividualquantitativemeasures.ThisiswhatTeachingqualityassessment(Tqa)ratingsattemptedtoprovideinaquantitativeway.inleaguetablesinenglandthesixfour-pointratingscalesinvolvedinTqahaveusuallybeencombinedintoasinglescoreoutof24,andinstitutionalaveragesoutof24havebeenusedasindicatorsofeducationalquality.subsequentanalysisofaverageTqascoresforinstitutionshasrevealedthattheyareverylargelypredictableonthebasisofstudententrystandards(a-levelpointsscores)andresearchperformance(raescores),togetherorseparately,withoutreferencetoanymeasuresofeducationalprocess(drennanandbeck,2001;yorke,1997,1998).inotherwords,Tqascoreslargelyreflectreputationalfactors.Thiswouldnotbeaterminalproblemifresearchperformanceandqualityofstudentswerevalidindicatorsofeducationalqualitybut,aswehaveseenabove,theyarenot.Theinabilityofreputationalfactorstoprovideavalidindicatorofeducationalqualityishighlightedaboveinsection5.7.1.Theinabilityofpeerjudgementstobeimmunefromreputationalfactorsunderminestheircredibility.Tqascoreswerealsosubjecttootherconfoundingvariables,suchasinstitutionalsize,whichhavenotbeentakenintoaccounteitherinmoderatingoverallscores,orinleaguetablesbasedonTqascores(cooket al.,2006).

5.7.3. studentsupport

Theqaahavehighlighted,intheirreviewsofwhathasbeenlearntfrominstitutionalaudits(qaa,2003),theimportantroleplayedbyadequatestudentsupportservicesofvariouskinds:studyskillsdevelopment,counselling,englishlanguagesupport,supportforstudentswithspecialneeds,andsoon.Thereanumberofreasonswhyitisdifficulttoestimatetheextenttowhichstudentservicesplayaroleineducationaleffectivenessorgain.supportservicesareconfiguredinmanydifferentways,forexamplesubsumedwithinacademicrolesorcentralisedingenericserviceunits.Theyaredescribedusingdifferentterminology:forexample,therearefewusequivalentsoftheuK’straditionalpersonaltutorrole,andfewuKequivalentsoftheroleof‘studentadvising’intheus.Thismakescollatingevidenceacrosscontexts,orcomparinglikewithlike,somewhatchallenging.dataconcerningthepositiveimpactofstudentsupportfromlargeusstudiesaredifficulttorelatetothenatureofuKprovision.Theimpactofsuchservicesalsorelatescloselytothenatureofstudentintake.slenderprovisionatoneinstitutionmightbeperfectlyadequatebecauseitonlyhasazephyrofdemandtodealwith,whileatanotherinstitutionevenextensiveandprofessionallyrunsupportservicesmayfaceagaleofdemandandexpectationsandsomayfallshortdespiteextensiveinstitutionalcommitment.Thereisclear

dimensionsofqualiTy

37

evidenceoftheroleofvariouskindsofstudentsupport,forexampleconcerningtheimpactonstudentperformanceofthedevelopmentofstudents’studyskills(hattieet al.,1996).however,whatsupportservicesareappropriate,andhowtheymightbestbedelivered,canbehighlycontext-anddiscipline-specific.forthisreasonnogeneralempiricalconclusionswillbedrawnhere.

5.7.4. qualityenhancementprocesses

muchofthepastfocusofattentionofthecouncilfornationalacademicawards,andtodaythequalityassuranceagency,hasbeenonqualityprocesses,suchastheoperationoftheexternalexaminersystemandtheuseofstudentevaluationofteaching,thatareintendedtoassurequality.Theassumptionisthatifsuchprocessesaresecurelyinplace,thenanadequatelevelofqualitycanbemoreorlessguaranteed.Thereissomeevidencetosupportthiskindofassumption.aswasdiscussedinsection4.1above,ininstitutionswherestudentengagementisfoundtobehighandeducationalgainsarehigh,onefindsahigherthanaverageinvestmentofresourcesinqualityenhancementprocessessuchasfacultydevelopmentandteachingandlearningcentres(Gansemer-Topfet al.,2004).Thereisalsoevidencethatsomeoftheprescribedqualityenhancementprocesseshaveapositivemeasurableimpact,butonlyundercertaincircumstances.forexample,collectingstudentfeedbackonteachinghaslittleornoimpactonimprovingteaching(weimerandlenze,1997)unlessitisaccompaniedbyotherprocesssuchastheteacherconsultingwithaneducationalexpert,especiallywhenprecededbytheexpertobservingteachingandmeetingstudents(Piccininet al.,1999).

Theextentofinstitutionaladoptionofqualityenhancementprocessesthroughteachingandlearningstrategieshasbeendocumentedforenglishinstitutions,(hefce,2001;Gibbset al.,2000),butthereiscurrentlynoevidencethattheextentofadoptionoftheseprocessesrelatestoanyothermeasuresofprocessorproduct.


38

6. ProducTdimensionsofeducaTionalqualiTy

6.1 studentperformanceanddegreeclassif ications

intheuKthemeasuremostcommonlyusedtoindicatethequalityoftheoutcomeofhighereducationistheproportionofstudentsgaininguppersecondclassorfirstclassdegrees.Theproportionofstudentswhogain‘gooddegrees’hasincreasedverymarkedlyovertime,althoughunevenlyacrossinstitutionsandsubjects(yorke,2009).atthesametimepresageandprocessindicatorsofquality(suchasfundingperstudent,thequalityofstudentintake,classsize,ssrs,amountofclosecontactwithteachersandamountoffeedbackonassignments)havedeclined.yorke(2009)suggestsawholelistofreasonswhythiscounter-intuitivephenomenonhasoccurred.forexample,theproportionofassessmentmarksderivedfromcourseworkhasincreasedandcourseworkusuallyproduceshighermarksthanexaminations(Gibbsandlucas,1997).mostofthepossibleexplanationscurrentlylackdatathroughwhichtheycouldbetested.

Thekeyproblemappearstobethattherehasbeenlittletostopgradeinflation.Theexternalexaminersystemhasnotprovedcapableofmaintainingthestandardsthatareappliedbymarkerstowhateverqualityofstudentworkisbeingassessed.asaconsequencedegreeclassificationscannotbetrustedasindicatorsofthequalityofoutcomes.awholeraftofunjustifiablevariationsexistsinthewaystudentdegreeclassificationsaregenerated.forexample,amathsstudentsismorethanthreetimesaslikelytogainafirstclassdegreethanahistorystudent(yorkeet al.,2002;bridgeset al.,2002)andthereareidiosyncraticinstitutionalalgorithmsforaddingmarksfromdifferentcourses(yorkeet al.,2008)thatcanmakeasmuchasadegreeclassificationdifferencetoindividualstudents(armstronget al.,1998).Thebestpredictorofthepatternofdegreeclassificationsofaninstitutionisthattheyhaveproducedthesamepatterninthepast(Johnes,1992),andinstitutions’historicalpatternsarenoteasilyexplicable.

ithasbeenarguedthatthereisnolongeranymeaningfulsenseinwhichdegreestandardsarecomparable(brown,2010).Therehasbeenpersistentcriticismofthemeaningandinterpretabilityofdegreeclassificationsasindicatorsofeducationaloutcomes(e.g.houseofcommons,2009)andtheseargumentshavebeenlargelyaccepted,e.g.bytheqaa(2006),andsotheargumentswillnotberehearsedhere.whatisclearisthatdegreeclassificationsdonotcurrentlyprovideasoundbasisforindicatingthequalityofeducationaloutcomesofauKinstitution.

dimensionsofqualiTy

39

6.2 studentretentionandpersistence

Theopenuniversityandtheuniversityofoxfordhavecomparablenssstudentratingsfortheperceivedqualityoftheireducationalprovision,butareatoppositeendsofrankingsintermsofstudentretention,withabout98%ofenteringundergraduatescompletinginthreeyearsatoxford,almostdoubletheproportionofnewstudentscompletingaten-monthcourseatTheopenuniversity.studentretention(inrelationtopersistingfromoneyeartothenextandcompletionrateswithinnormaltimeframes)varyveryconsiderablyfromoneinstitutiontoanotherevenwheneducationalprovisionisjudgedtobesimilarlyexcellentorsimilarlypoor.institutionalcomparisonsaremadedifficultbythevariednatureofstudentcohorts.broadly,nationalretentionratesvaryininverserelationtoageparticipationrates(oecd,2000):thebroadertherangeofstudentabilityisenteringhighereducation,theloweristheoverallretentionrate.inaddition,differentinstitutionstaketheirstudentsfromdifferentsubsetsoftheoverallabilityrange.

studentsvarynotjustintermsoftheirrecordofpasteducationalsuccess,butinothervariablesknowntoaffectretentionsuchaswhethertheyliveoncampus(chickering,1974)andwhethertheyareundertakingpaidworktosupporttheirstudies(Paton-saltzbergandlindsay,1993).

intheusitisnolongerthecasethatthemajorityofstudentsgainthecreditstheyneedforaqualificationfromasingleinstitution.so‘drop-out’isnotonlythenormbutis,formanystudents,expectedandevenplannedforastheyaccumulatecreditswhereverandwheneverisconvenient.ThisisnotyetthenormintheuK,but‘drop-out’doesnothavethesamemeaningorsignificanceforanincreasingproportionofstudentsasitdoesforpolicymakers(woodley,2004).itisnotsimplythatpart-timestudentscompleteatdifferentratesthandofull-timestudents,butthat‘retention’hasadifferentsignificanceforthem.

avariableknowntoinfluenceretentioniswhetherstudentsaresociallyandacademicallywellintegrated(Tinto,1975).socialandacademicintegrationisaffectedbylivingoffcampus,livingathome,andtakingtimeouttoearnenoughtocontinuestudying.Theprevalenceofthesevariablesisveryvariedacrossinstitutions,anditisdifficulttotakeallsuchvariablesfullyintoaccountinjudginginstitutionalretentionperformance.

studentvariablesalsoaffectretentionwithininstitutionsandaresoinfluentialthatintheuscommercialcompanies(suchasthenoel-levitzorganisation)offerservicestoinstitutionstocollectmanagementinformationandotherstudentdataconcerningtheireducationalqualifications,preparednessandattitudes,inordertopredictwhichstudentsaremostlikelytodropoutsothatscarceadditionalsupportcanbedirectedatthestudentsmostlikelytobenefit.amathematical,data-drivenapproachofthiskindatTheopenuniversityhasidentifiedverywidedifferencesbetweenenteringstudentsinrelationtotheprobabilityofthemcompletingasinglecourse.Thispredictionhasbeenusedtodecidewhichstudentstocontactandsupport,withmeasurablepositive


40

consequencesforoverallretention(simpson,2003;Gibbset al.,2006).Thetypesofstudentvariablesthatpredictdrop-outgowellbeyondthekindofdatathathefcehaveavailabletocalculateinstitutionalbenchmarksforretention.soeventheextenttowhichinstitutionsexceedorfallshortoftheirretentionbenchmarkscanonlybeacrudeandincompletemeasureoftheireducationalquality.

notalloftheinstitutionalvariationinretentionisduetostudentvariables.effortstoimproveretentionhavebeenevaluatedfor30yearsintheus,andwhileoverallretentionrateshaveremainedlargelystatic,thishidessubstantialprogressinimprovingretentioninsomeinstitutions.agooddealisnowknownaboutwhatkindsofinstitutionaleffortsarelikelytoimproveretentionandpersistenceintheus(barefoot,2004)and,withamuchlesserevidencebase,intheuK(yorke,1999).makinggooduseofthoroughinformationaboutstudentssoastotargettimelyindividualisedsupportandinterventionisoneofthemosteffectivepractices.othereffectivepracticescloselyresemblethoseidentifiedasimprovingstudentperformanceandeducationaloutcomesingeneral(lanasaet al.,2007),discussedinsection5.5.3above.inparticular,collaborativeandinteractivelearningandclosecontactwithteachersincreasessocialandacademicintegration.aspointedoutabove,suchinterventionshaveagreaterimpactonlessablestudents.

ifvariationsbetweenstudents,andespeciallypsychologicalvariablessuchasmotivationandcommitment,andsocialvariables,suchaswherestudentsliveandhowmuchtimetheyhaveavailabletostudy,couldbefullytakenintoaccount,thenretentionperformancecouldbeusedasanindicatorofeducationalquality.however,withthedatacurrentlyavailablethisisnotyetpracticable.

6.3 employabilityandgraduatedestinations

Theextenttowhichgraduatingstudentsareabletoobtainemploymentreasonablyquickly,ingraduatejobs,infieldsrelevanttotheirdegreesubject,andwithasalarythatjustifiestheirinvestmentoftimeandmoneyintheirhighereducation,isacommonlyuseddimensionofquality.Thedifficultywithemployabilitydata,aswithretentiondata,istheirinterpretation.differentmethodsofcollectingdata,andinparticularthetimingofthedatacollection,makesaconsiderabledifference,andtheprocessusuallyreliesonsurveysinvolvingstudentself-reporting.howeveremployabilitydataarecollected,interpretingdifferencesbetweeninstitutionsisproblematicforawidevarietyofreasons(smithet al.,2000):

— itisaffectedbydegreeclassification,andstudentperformanceiscloselylinkedtostudents’prioreducationalqualifications(whichalsoaffectsemployability),whichvariesgreatlybetweeninstitutions(seesection4.4above).inaddition,degreeclassificationsvaryintheirmeaningsacrossinstitutions(seesection6.1above).

dimensionsofqualiTy

41

— itisgreatlyaffectedbyinstitutionalreputation,whichisaverypoorindicatorofeducationalquality(seesection5.7.1above).Thereislittleevidencethatemployershaveanaccurateandup-to-datepictureoftheeducationalqualityoftheinstitutionstheyemploygraduatesfrom,especiallyiftheyderivethispicturefromcurrentlyinvalidleaguetables.

— itisaffectedbyregionandlocality,duetovariationsinlocalemploymentmarketsandtheproportionofstudentswholiveathomeandwhoarelesslikelytomoveawayeithertostudyortofindemployment.

— itchangesovertime,duetochangesintheemploymentmarket,andprobablydifferentiallybetweeninstitutions,withoutanychangesintheeffectivenessofhighereducationinstitutions.

— itisaffectedbystudents’socialclass,andthemixofsocialclassvariesbetweeninstitutions.

— itisaffectedbystudents’age,andageprofilesvarybetweeninstitutions.— itisaffectedbystudentaffluence,withsomestudentsneedingtotakeanyemploymentquicklywhileotherscanaffordtowaitforgraduate-levelandsubject-relevantemployment,ortocontinuetheireducation.

— itisaffectedbysubjectmix,withhigherlevelsofgraduateunemployment,non-graduate-levelemployment,andemploymentinfieldsunrelatedtothedegreesubject,moreofteninsomesubjectsthaninothers.

interpretingthedifferencesthatexistbetweenemployabilityindicatorsintheuKandmainlandeuropeisalsodifficultbecausethehighereducationsystemsaredifferentincrucialways.forexample,abouthalfofuKgraduatesfromnon-vocationalsubjectsfeltthattheirfirstjobswerenotappropriatetoadegree-leveleducationcomparedwithaboutaquarterofsuchgraduatesfromothereuropeancountries.forvocationalsciencegraduatestheproportionwasonly17%and10%respectively,butstillhigherfortheuKthanfortherestofeurope(brennanandTang,2008a,2008b).Theoveralldifference,acrossallsubjects,isprobablybecauseeuropeangraduatestendtobeolder,muchmorelikelytohavemasters-levelqualificationsbythetimetheyenterthejobmarket,andmorelikelytohaveexperiencedwork-basededucationrelevanttotheirsubjectduringtheirmoreextendededucation.ThereisalsoadifferentsubjectmixbetweentheuKandeuropewithmorenon-vocationalartsgraduatesintheuK(withunemploymentratestwicethatofsomeothersubjects).ThisemployabilitydifferencebetweentheuKandeuropedisappearsaboutfiveyearsaftergraduation,atwhichpoint96%ofuKgraduatesareinemployment(ibid.),whichsupportstheinterpretationthatthereareinfluentialdifferencesbetweentheuKandmainlandeuroperegardinggraduates’ageandvocationalexperienceatthetimeofgraduation.

Thereareusexamplesofresearchthatvalidateinstitutionalemployabilitymissionsbyidentifyinglong-termconsequencesforthewaygraduatesdemonstrate,intheirprofessionallives,abilitieslearntatcollege(e.g.mentkowskianddoherty,


42

1984).ThereareonlyverymodeststudiesofthiskindintheuK(e.g.Jenkinset al.,2001),andcertainlynotenoughtomakeinstitutionalcomparisonsoreventovalidateinstitutionalclaimsabouttheefficacyoftheiremployabilitymissions.

Thehighereducationstatisticsagencyisabletotakeintoaccountsomevariables(subjectofstudy,qualificationsonentryandageonentry)insettinginstitutionalperformancebenchmarksforemployability,butnotothers.smithet al.(2000)havemadeamoremathematicallysophisticatedattempttotakemorevariablesintoaccount,butstillleaveoutcrucialvariablesaboutwhichdataarenoteasytoobtain.interpretinganinstitution’sgraduateemploymentperformanceinrelationtohefcebenchmarksisfraughtwithproblems.

finally,theloosefitthatcharacterisestheuK’shighereducationanditsjobsmarkethasbeeninterpretedbysomecommentatorsnotasaproblem,butasprovidingflexibilityforgraduatestocopewithafluidemploymentmarketthatisconstantlychanginginrelationtothecapabilitiesthatarerequired.Thisissueconcernsthedifferencebetweenexpertiseforefficiency,whichiswhatemployersrecruitinggraduatesnormallydemand,andadaptableexpertise,thatenablesanindividualtooperateeffectivelyinunpredictablenewsituations(schwartzet al.,2005).ittakesverydifferentkindsofeducationalprocesstodevelopthesetwoformsofexpertise.Thereisalackofevidenceaboutthelong-termconsequencesforgraduateemploymentofeithernarrowlyfocusedvocationaleducationoreducationthatemphasisesefficiencyingeneric‘employabilityskills’,ratherthanemphasisingthehigherorderintellectualcapabilitiesinvolvedinadaptableexpertise.Thismakesrelyingonhesa’sveryshort-termemploymentdataariskythingtodo.

dimensionsofqualiTy

43

7. summaryandconclusions

7.1 Theimportanceofprocessvariables

muchofthisreportdemonstrateswhatcommentatorsintheushavebeenarguingformanyyears.Presagevariablessuchasfunding,researchperformanceandthereputationthatenablesaninstitutiontohavehighlyselectiveentry,donotexplainmuchofthevariationbetweeninstitutionsinrelationtoeducationalgains.measuresofeducationalproductsuchasgradesandcareerearningsreflectthesepresagevariables,becausethebeststudentscompetetoenterthebestfundedandmostprestigiousinstitutionsandthequalityofstudentsisthebestpredictorofproducts.measuresofproductsuchasretentionandemployabilityarestronglyinfluencedbyaraftofvariablesthatmakeinterpretinganinstitution’sperformanceextremelydifficult.

Themostimportantconclusionofthisreportisthatwhatbestpredictseducationalgainismeasuresofeducationalprocess:whatinstitutionsdowiththeirresourcestomakethemostofwhateverstudentstheyhave.Theprocessvariablesthatbestpredictgainsarenottodowiththefacilitiesthemselves,ortodowithstudentsatisfactionwiththesefacilities,butconcernasmallrangeoffairlywell-understoodpedagogicalpracticesthatengenderstudentengagement.

intheuKwehavefewdataabouttheprevalenceoftheseeducationalpracticesbecausetheyarenotsystematicallydocumentedthroughqualityassurancesystemsandnorarethey(inthemain)thefocusofthenss.Thebestmeasureofengagement,thensse,isusedonlytoaverylimitedextentintheuK.

7.2 Theimportanceofmultivariateanalysis

muchoftheuKdataaboutrelationshipsbetweenpresageandprocessvariables,orbetweeneitherpresageorprocessvariablesandproductvariables,looksatonepairofvariablesatatime–forexample,therelationshipbetweenameasureofresearchperformance(e.g.therae)andameasureofteachingquality(e.g.Tqascores).suchrelationshipsareinvariablyconfoundedwithrelatedvariables,forexamplewiththequalityofstudentsattractedtothehigh-statusinstitutionsthathavehighresearchperformance.asaconsequencefewrelationshipsbetweentwovariablescanbeinterpretedwithconfidence.ThefewuKstudiesthathaveexaminedanumberofvariablesatatimeusingsomeformofmultivariateanalysis(e.g.drennanandbeck,


44

2001;yorke,1998)haveconfirmedthatapparentlystrongrelationshipsbetweenpairsofvariables(e.g.betweenameasureofresearchandameasureofteaching)areconfoundedbyothervariablesthatcouldequallyberesponsibleforapparentrelationships(e.g.ameasureofqualityofstudentintake).intheustherehavebeenfarmore,largerandmorecomplex,multivariateanalysesthattakeintoaccountawholeraftofvariablesatthesametimeandwhich,asaconsequence,areabletoteaseoutthosevariablesthatareconfoundedwithothersandthosethatarenot.wearethereforelargelydependentonusdataandanalysesforourunderstandingofthecomplexrelationshipsbetweendimensionsofquality.someofthenecessarydatathatwouldallowamorecomprehensivemultivariateanalysisintheuKhavealreadybeencollectedandcollated(forexamplebyhefce,hesa,thenssandbyhePi),butitcurrentlyresidesindifferentdata-bases.itwouldbehelpfultocombinethesedatabasessoastoallowmultivariateanalysis,andtoaligndatacollectionmethodstomakethiseasiertodo.

7.3 Theimportanceofeducationalgain

whilesomeuKdataincludemeasuresofeducationalproduct,thereareveryfewuKstudiesthathaveincludedmeasuresofeducationalgain.Thismattersbecausethebestpredictorofproductisthequalityofstudentsenteringtheinstitution,andthequalityofstudentsvariesgreatlybetweeninstitutions,sothatifyouonlyhaveameasureofproduct,suchasdegreeclassifications,ratherthanofgains,thenyoucannoteasilyinterpretdifferencesbetweeninstitutions.whenuKstudiesdoattempttomeasuregaintheyinvolvedifferentmeasuresonentrythanonleavinghighereducation(forexamplea-levelpointscoresanddegreeclassifications,respectively).furthermorethemostcommonmeasureofproduct,degreeclassification,variesinitsmeaningandstandardacrosssubjectsandacrossinstitutions(yorke,2009).itisthereforedifficulttointerpreteventhesecomparativemeasuresofgain.studiesintheusincontrastarefarmorelikelytousepsychometricmeasuresofgenericeducationaloutcomes(suchasatestofcriticalthinking)withthesamemeasure,andwiththesamestandardsbeingusedacrossdifferentsubjectsandinstitutions,andalsousingthesamemeasurebothbeforeandafterexperiencingthreeorfouryearsofcollege.inthiswayareliablemeasureofeducationalgain,andcomparisonbetweeninstitutionsinrelationtoeducationalgain,ispossible.againweareheavilydependentonusstudiesforevidenceofwhichqualitydimensionspredicteducationalgain,andespeciallyonthevaststudies,andreviewsofevidence,undertakenbyastin(1977,1993)andPascarellaandTerenzini(1991,2005).

7.4 dimensionsofqualityindif ferentkindsofinstitutions

relyingonusdatamightnotmatterifinstitutionsandeducationalprocesseswereessentiallythesameoneithersideoftheatlantic.however,itseemslikelythat

dimensionsofqualiTy

45

thedimensionsthatdefinequalityinavalidwayaredifferentindifferentkindsofinstitutions.forexample,evenwithintheusthequalityindicatorsthatappearvalidforlarge,national,researchuniversities(inthesensethattheypredicteducationalperformancetolerablywell)donotworkaswell,oratall,inregionalschoolsandnon-selectivecolleges(schmitz,1993).similarlythenormallackofarelationshipbetweenanemphasisonresearchandanemphasisonteachingdoesnotseemtoapplytoasmallgroupofwell-endowedliberalartscollegesthatemphasiseclosecontactbetweenteachersandstudents(astin,1993).differentpedagogicalphenomenaarelikelytobesalientindifferentcontexts,withsomewhatdifferentpatternsofrelationshipsbetweenprocessandproduct,dependentoncontext.

itisnotjustthatdifferenteducationalprocessesmighthavemoreinfluenceoneducationalgainsinsometypesofinstitutionthaninothers.measuresofeducationalgainthemselvesmightalsoneedtobedifferentbetweeninstitutionsiftheyaretohavemeaning.institutionalmissionsvary,particularlywithregardtotherelativeimportanceofemployabilityandsubjectknowledge.itwouldbesurprisingifthesamemeasuresofeducationalgainwereequallyappropriateinalluKinstitutions.forexample,Theopenuniversity’smission,emphasisingopenness,meansthatitwouldnotseektoincreasestudentretentionandperformancethroughincreasingselectivitybecausethatwouldreduceitsopenness.itsownindicatorsofqualityaredistinctive,andaredifferentevenfromthoseusedbyhefceindeterminingitsfunding.Theproblemhereisthatfundingmechanismsaredrivenbyindicatorsofqualitythatcutacrossinstitutions’missions.

usresearchhasdonewelltoidentifyanyconsistentpatternsatallacrossvariedcontexts.however,thelimitsofwhatispossibletoconclude,onaverage,havebeenhighlightedbythoseconductingtheresearch(Pascarella,2001).ThesamecautionshouldaccompanyextrapolationoffindingsaboutkeyindicatorsofqualityfromvarieduscontextstovarieduKcontexts.

7. 5 d imens ion s o f qu a l i t y i n d i f f e ren t dep ar tmen t s

muchoftheliteraturecitedabove,andmostofthedebate,hasfocusedoninstitutionaldifferencesinquality.however,itisclearthatdepartmentscandifferhugelywithinthesameinstitution.regardingnssscores,thereareinstitutionsthathavethehighest-rateddepartmentinenglandinonesubjectandthelowestratedinanothersubject,despitesharingthesameinstitutionalqualityindicators.educationalleadershipofdepartmentsmakesadifference,creatingculturesthatvalueteaching,thatengageinaconstantprocessofimprovingteaching,andthatcreaterichandengaginglearningenvironments,tosomeextentwhatevertheinstitutionalenvironmentandpresagevariables(ramsden,1998;Gibbset al.,2008b).

interestinglythetwoinstitutionsfrequentlyreferredtointhisreport,andthatappearatthetopofthenssranking,theuniversityofoxfordandThe


46

openuniversity,have‘institutionalpedagogies’:commonpatternsofteachingandassessmentacrossalldepartments.alltheirdepartmentsscoresimilarlyhighlyonthenss.Thismaybeinpartduetodeliberatepoliciestoprotecttheinstitutionalpedagogy(whichisknowntoworkwell)fromoutsidepressures,withtheeffectoflimitingdepartmentalvariationinteachingquality.Princetonuniversitysimilarlyhasanapproachtoqualityassurancethroughitscourseapprovalprocessthatisintendedtomaintain‘thePrincetonmodel’ratherthanencouragediverseinnovation.

muchoftheemphasisintheuKonrecognisingandrewardingqualityinteachinghasfocusedonindividualteachers(e.g.throughthenationalTeachingfellowshipscheme)orinstitutions(throughleaguetablesofonekindoranother).Thereareexampleselsewhereofnationalandinstitutionalschemestoidentifyeducationalquality(e.g.infinlandandattheuniversityofoslo,respectively)thatfocusondepartments,or‘learningenvironments’atthelevelofadegreeprogramme(Gibbs,2008).

7.6 dimensionsofqualityindif ferentsubjects

ThecarnegiefoundationfortheadvancementofTeachingestablishedalarge-scaleinitiativeontheassumptionthatthepedagogiesofdisciplinesaredifferent:that,forexample,educationalqualityisachievedthroughdifferentstrategiesandpracticesinthecreativeartsthaninthesciences,anddifferentlyinenglishthaninlaw(huberandmorreale,2002).atasufficientlevelofabstractiontherearesimilarunderpinningeducationalprinciplesacrossalldisciplines(Gibbs,1999),buttheyareembodiedinsuchvariededucationalpractices,andaresalienttosuchdifferentextentsindifferentdisciplines,thatdisciplinesineffectachieveeducationalqualityindifferentways.ifyouthenattempttomeasurequalityacrossdisciplines,forexamplebyusingthecourseexperiencequestionnaire,youfindthatsomedisciplinesemergeconsistentlybetterthanothers,acrossdifferentstudiesanddifferentinstitutions.eitheronehastoacceptthatcertainsubjectsarealwaystaughtlesswellthanothers,whichseemshighlyunlikely,orthatdifferentmeasuresofqualityarebetteralignedwiththeconsequencesofsome(disciplinary)pedagogicpracticesthanwithothers.Productmeasuresareparticularlysusceptibletodisciplinaryvariation.ThedistributionofdegreeclassificationsismarkedlydifferentbetweensubjectsintheuK(yorkeet al.,2002;bridgeset al.,2002).employabilitymeansdifferentthings,andcomesaboutindifferentways,followingthestudyofdifferentdisciplines.comparingqualitybetweendisciplinesisfraughtwithdifficulties.

itseemslikelythatmanystudentschoosesubjectsfirstandinstitutionssecond,nottheotherwayround.currentinstitutionalqualitydataandrankingsarethewrongplacetostartifinformingpotentialstudentsisthepriority.furthermorethedefinitionofa‘subject’intheinformationavailabletostudentsoftendoesnotcorrespondverycloselywiththeprogrammeinwhichstudentsmaybeinterested.brownet al.(2009)giveanexampleofa‘subjectcategory’usedforreporting

dimensionsofqualiTy

47

nssratingsthatinoneinstitutionisanaveragedrawnfrom11differentdegreeprogrammes.studentsneedgooddataaboutprogrammesmorethantheydoaboutinstitutionsorevenaboutbroad‘subjects’,andthensscurrentlydoesnotprovidethat,fortechnicalreasonsthatwillbedifficulttoovercome.Politicaldemandsfor‘betterinformationforcustomers’cannotbemetwithcurrentdatagatheringandanalysismethodspartlybecausetheyaggregatedataintoocoarseaway.oncedataareaggregatedinafineenoughwaytobeuseful,therearethenboundtobeproblemswithsamplesizes.Thisproblemmaybeintractableandisoneofanumberofsimilarproblemsthatmakeitdifficulttoprovideinformationaboutqualityinaccessibleandusableformsevenwhenithasbeencollated(brown,2007).

Thequalityofindividualcoursesormodulesalsovarieswithindegreeprogrammes,andtheextentofthisvariationmayberelatedtodegreecoherence.Thisreporthasfocusedoninstitutionsanddegreeprogrammesratherthanonvariablesthatprimarilyaffectindividualcourses.

7.7 dimensionsofqualitythataredif f iculttoquantify

mostofthisreporthasfocusedondimensionsofqualitythatarefairlyreadilyoperationalisableinawaythatenablesthemtobemeasuredquantitatively,sothatstatisticalrelationshipscanbeestablishedwithotherdimensionsthataresimilarlyeasytomeasure.Thereareotherdimensionsofqualitythatareimportant,atleastinsomecontexts,butthataredifficultorimpossibletoquantify.forexample,throughoutliteratureinvolvingcasestudiesofexcellentteachingatdepartmentleveltherearereferencestoaspectsofdepartmentalculture:whetherteachingisvaluedandrewarded,whetherteachersregularlytalktoeachotheraboutteachinganditsimprovement,whetherinnovationinteachingissystematicallysupportedandfunded,whethereducationaleffectivenessisthesubjectofseriousscholarlyevaluation,andsoon(hannanandsilver,2000).qualitiesofdepartmentalleadershipofteachingmakeaconsiderabledifference(ramsden,1998;Gibbset al.,2008a),andsomeeffortshavebeenmadetomeasureteachers’perceptionsbothofdepartmentalleadershipofteachingandoftheteachingenvironmentthatframesthekindofteachingandlearningthatislikelytotakeplace(e.g.ProsserandTrigwell,1997;martinet al.,2003).

sometimeshighlyeffectiveeducationalsystemsaredrivenalmostentirelybyvalues,suchas‘likingyoungpeople’,virtuallyindependentlyofthepedagogicpracticesemployedortheresourcesavailable.inaninternationalstudyofdepartmentsthatwereidentifiedbytheirinstitutionasofexceptionallyhighqualityinrelationtoteaching,studentsinoneofthedepartmentssaidthattheirteacherswerenotespeciallygoodbut that it didn’t matter becausetheyfeltincludedinanexcitingcommunityofscholars(Gibbset al.,2008a).studiesatoxfordbrookesuniversityconcerningwhysomesubjectsregularlyproducedbetterstudentperformancethanothersfoundnodifferencesinanyquantitativemeasureofpresagevariables.however,aqualitative


48

follow-upstudyfoundthatthehighperformingsubjectswerecharacterisedbyhealthy‘communitiesofpractice’involvingmuchdiscussionofhowtosolveteachingproblemssoastomaketheentireprogrammeworkwellforstudents.incontrast,subjectswithconsistentlylowaveragemarkswerecharacterisedbyacorrespondinglackoftalkingaboutteaching,andafragmentedfocusonindividualcourses(havnes,2008).itmaybedifficultorimpossibletomeasuresuchinfluentialvariablesinwaysthatallowsafecomparisonbetweencontexts,althoughitmaybepossibletomeasuretheirconsequences,forexampleinrelationtostudentengagement.

7.8 evidenceoftheproductsoflearning

amongthemosttellingofallindicatorsofthequalityofeducationaloutcomesmustbestudents’final-yeardissertationsandprojectreports.itisadistinctivefeatureofuKhighereducation(andinthepastarequirementofthecnaaforhonoursdegreeclassification)thatstudentsundertakeaverysubstantialpieceofindependentstudyintheirfinalyear.evenatusivyleagueinstitutionsundergraduatestudentswouldusuallyneedtotakeafourth,honours,yeartotacklesuchachallengingpieceofwork.itisoftenaculminationandintegrationofalltheyhavelearnt,especiallyinappliedandcreativefieldsofstudy.Thereisanalmosttotallackofevidenceconcerningtherelativequalityofsuchproductsacrossinstitutions,withinsubjects.anattempt,forthisreport,toobtainsuchevidencefromsubjectcentreselicitednotasingleexample,andthefewpublishedstudiesillustratetheembryonicnatureofefforts(e.g.woolfet al.,1999).dissertationsandprojectreportsareoftenarchivedandareavailableforstudy–althoughcurrentlynotcomprehensivelyacrossallinstitutions.suchproductswouldbeamenabletosystematicpeerreviewwithineachsubject’sacademiccommunity,inawaythattheexternalexaminersystemsignallyfailstodo(warren-Piper,1994).suchproductswouldalsobeamenabletoreviewbyeducationalresearchersusingagenericframeworkforcategorisingthequalityoflearningoutcomessuchasthesolo(structureoftheobservedlearningoutcome)taxonomy(biggsandcollis,1982),whichiscapableofdistinguishinglevelsofqualityacrossdifferentformsofassessmentproductwithinsubjects,andevenacrosssubjects.

7.9 Thepotentialforimprovedquality,andtheevaluationofimprovedquality

Thelackofarelationshipbetweenresearchperformance,funding,ssrsandstudentselectivity,ontheonehand,andstudentengagementandeducationalgainsontheother,thatmakesthesepresagevariablessuchpoorindicatorsofquality,isnotinevitable–itisnotlikeaninvariantphysicallaw,i.e.itdoesnotapplytoallcircumstances,foralltime.itisinpartaconsequenceofcomparativelywell-funded,

dimensionsofqualiTy

49

selective,research-orientedinstitutionsnotexploitingtheirpotentialadvantagestothefull.iftheyweretochangetheirpracticesinlinewithwhatisknownabouteducationaleffectiveness,inotherwordsemphasiseprocessvariablesratherthanrelyingonpresagevariablestodothejobontheirown,thenthisrelationshipcouldchange,providedthattheirpotentialadvantagesenabledthemtoimplementtheprocessesmorethoroughlythanothers.

Thereisarapidlygrowingevidencebaseintheusabouttheimpactoneducationalgainsofchangesinevidence-basededucationalprocesses(cf.carininet al.,2006).itisbecomingeverclearer,asaconsequence,whateducationalprocessesitissensibletointroducewithareasonablelikelihoodofimprovingeducationalgains.ThereisnosuchevidencebaseintheuK,orevenamethodologyforbuildingone.whileinenglandtherearesomedeliberateinstitutionaleffortstoimprovenssscorestheydonotalwaysappeartobebasedonanyvalidconceptualframeworkconcerningwhatprocessesarelikelytoproducewhatoutcomesandthenssdoesnotcurrentlyhavevalidityasanindicatorofqualityinthesensethatitpredictsoutcomes,letalonegains.anexampleofanalternativeapproachintheuKistheTesTaprojectbasedattheuniversityofwinchester(TesTa,2010),whichemploysestablishedmethodologiesforauditingassessmentpracticesandformeasuringstudentlearningresponsestothosepractices,bothbeforeandafterevidence-basedchangesinassessment,involvingwholedegreeprogrammes,acrossanumberofinstitutions.Progresshasbeenmadeintheuspartlybecausealargenumberofinstitutionshavebeenpreparedtoadoptthesameevidence-basedconceptualframeworkandusethesamevalidevaluationtools.Thisallowsmeaningfulevaluationdatatobepooled,andsoenablingconclusionstobedrawnthatstandoutfromthenoiseofcontextualvariation.Thehighereducationacademyisstartingtotakealeadinidentifyinganddisseminatingsuitableevaluationtoolsandmethodologies,andcreatinganevidenceinfrastructurewithinwhichdatafromlocallyconductedstudiescouldbecollated.


50

8. abouTTheauThor

GrahamGibbshasspent35yearsinresearchanddevelopmentworktoimprovethequalityofteaching,studentlearningandassessmentinhighereducation.

hehasbeencentrallyinvolvedinaseriesofnationalteachingdevelopmentinitiatives,includingtheTeachingmorestudentsProjectandhefce’sinstitutionallearningandTeachingstrategyinitiative,andintheco-ordinationofthefundforthedevelopmentofTeachingandlearning.heisthefounderoftheimprovingstudentlearningsymposiumandoftheinternationalconsortiumforeducationaldevelopmentinhighereducation.hehasbeenawardedanhonorarydoctoratebysheffieldhallamuniversityforhisleadershipofthedevelopmentofteachingintheuK,andbytheuniversityofutrechtforhisinternationalleadershipofeffortstoimproveuniversityteaching.

heretiredfromhispositionasProfessoranddirectoroftheoxfordlearninginstitute,attheuniversityofoxford,in2007.

dimensionsofqualiTy

51

9. acKnowledGemenTs

inundertakingbackgroundresearchforthisreportireceivedsupportfrombahrambekhradnia,Johnbrennan,davidwatsonandmantzyorke,andalsofromrogerbrownwhoinadditionprovidedinvaluableguidanceinrevisingdrafts.Theyhelpedmetolocateitsfocusinrelationtoexistingliteratureandinrelationtothenatureofcurrentdebatesaboutqualityandstandardsinhighereducation.Theirwisdomisgreatlyappreciated.

iwouldalsoliketothankhighereducationacademystafffortheirsupport,andinparticulardrrachelsegalforhereditorialguidance


52

10. references

abrami,P.c.,d’appolonia,s.andcohen,P.a.(1990)validityofstudentratingsofinstruction:whatweknowandwhatwedonot. Journal of Educational Psychology. 82(2),pp219–231.

‘adjwilson’.(2010)ifthecapfits[online].london:educationGuardian.availablefrom:www.guardian.co.uk/education/2010/may/25/education-letters-gibb-adhd-tuition-fees[12august2010].

armstrong,m.,clarkson,P.andnoble,m.(1998)Modularity and credit frameworks: the NUCCAT survey and 1998 conference report. newcastle-upon-Tyne:northernuniversitiesconsortiuumforcreditaccumulationandTransfer.

ashby,e.,Plant,K.,ericsson,a.,hill,l.andasberg,K.(2005)whystudytimedoesnotpredictgradepointaverageacrosscollegestudents:implicationsofdeliberatepracticeforacademicperformance.Contemporary Educational Psychology.30(1),pp96–116.

astin,a.(1977)Four critical years. sanfrancisco:Jossey-bass.

astin,a.w.(1985)Achieving educational excellence: a critical assessment of priorities and practices in higher education.sanfranciscoandlondon:Jossey-bass.

astin,a.(1993)What matters in college. sanfrancisco:Jossey-bass.

bales,r.f.,strodtbeck,f.l.,mills,T.m.androseborough,m.(1951)channelsofcommunicationinsmallgroups.American Sociological Review.16(4),pp461–468.

barefootb.o.(2004)highereducation’srevolvingdoor:confrontingtheproblemofdrop-outatuscollegesanduniversities.Open Learning, 19,(1),pp65–77.

barrie,s.andGinns,P.(2007)Thelinkingofnationalteachingperformanceindicators

toimprovementsinteachingandlearninginclassrooms.Quality in Higher Education.13(3),pp275–286.

bauer,K.w.andbennett,J.s.(2003)alumniPerceptionsusedtoassessundergraduateresearchexperience.Journal of Higher Education. 74(2),pp210–230.

bergren,m.,snover,l.andbreslow,l.(2007)undergraduateresearchopportunitiesatmiT.Illuminatio.spring2007,pp6–8.

biggs,J.b.(1993)fromtheorytopractice:acognitivesystemsapproach. Higher Education Research and Development. 12(1),pp73–85.

biggs,J.andcollis,K.f.(1982)Evaluating the quality of learning: the SOLO Taxonomy (Structure of the observed learning outcome). newyork:academicPress.

black,P.andwiliam,d.(1998)assessmentandclassroomlearning.Assessment in Education. 5(1),pp7–74.

bloom,b.s.,englehart,m.d.,furst,e.J.,hill,w.h.andKrathwohl,d.r.(1956)Taxonomy of educational objectives. Handbook 1: Cognitive domain.newyork:davidmcKay.

bound,J.andTurner,s.(2005)cohortcrowding:howresourcesaffectcollegiateattainment.Journal of Public Economics.91(5–6),pp877–899.

bowden,r.(2000)fantasyhighereducation:universityandcollegeleaguetables. Quality in Higher Education.6(1),pp41–60.

brennan,J.andTang,w.(2008a) The employment of UK graduates: comparisons with Europe. REFLEX Report to HEFCE No 1.bristol:hefce.

brennan,J.andTang,w.(2008b) Subject differences in graduate employment across Europe. REFLEX Report to HEFCE No 2. bristol:hefce.

brennan,J.,Patel,K.andTang,w.(2009) Diversity in the student learning experience and time devoted to study: a comparative analysis of the UK and European evidence.bristol:hefce.

dimensionsofqualiTy

53

bridges,P.,cooper,a.,evanson,P.,haines,c.,Jenkins,d.,scurry,d.,woolf,h.andyorke,m.(2002)courseworkmarkshigh,examinationmarkslow:discuss.Assessment and Evaluation in Higher Education.27(1),pp35–48.

brown,r.(2006)leagueTables–dowehavetolivewiththem?Perspectives: Policy and Practice in Higher Education. 10(2),pp33–38.

brown,r.(2007) The information fallacy.oxford:highereducationPolicyinstitute.availablefrom:www.hepi.ac.uk/484-1291/The-information-fallacy.html[June2010].

brown,r.(2010) Comparability of degree standards? oxford:highereducationPolicyinstitute.availablefrom:www.hepi.ac.uk/455-1838/comparability-of-degree-standards.html[June2010].

brown,r.,carpenter,c.,collins,r.andwinkwist-noble,l.(2009)recentdevelopmentsininformationaboutprogrammequality.Quality in Higher Education. 13(2),pp173–186.

carinin,r.,Kuh,G.andKlein,s.(2006)studentengagementandstudentlearning:testingthelinkages.Research in Higher Education. 47(1),pp1–32.

carney,c.mcneish,s.andmccoll,J.(2005)Theimpactofpart-timeemploymentonstudents’healthandacademicperformance:ascottishperspective. Journal of Further and Higher Education. 29(4),pp307–319.

cheng,y.c.andTam,w.m.(1997)multi-modelsofqualityineducation. Quality Assurance in Education. 5(1),pp22–31.

chickering,a.w.(1974)Commuting versus resident students: Overcoming the educational inequities of living off campus. sanfrancisco:Jossey-bass.

chickering,a.w.andGamson,Z.f.(1987a)Seven Principles for Good Practice in Undergraduate Education.racine,wi:TheJohnsonfoundationinc.

chickering,a.w.andGamson,Z.f.(1987b)sevenprinciplesforgoodpracticeinundergraduateeducation.AAHE Bulletin. 39(7),pp3–7.

chickering,a.w.andGamson,Z.f.(1991)Applying the seven principles for good practice in undergraduate education.sanfrancisco:Jossey-bass.

clarke,m.(2002)someguidelinesforacademicqualityrankings. Higher Education in Europe. 27(4),pp443–459.

coffey,m.andGibbs,G.(2000)TheevaluationofthestudentevaluationofeducationalqualityquestionnaireinuKhighereducation.Assessment and Evaluation in Higher Education. 26(1),pp89–93.

cook,r.,butcher,i.andraeside,r.(2006)recountingthescores:ananalysisoftheqaasubjectreviewgrades1995–2001.Quality in Higher Education.12(2),pp135–144.

curtis,s.andshani,n.(2002)Theeffectoftakingpaidemploymentduringterm-timeonstudents’academicstudies.Journal of Further and Higher Education.26(2),pp129–138.

curtis,s.andwilliams,J.(2002)Thereluctantworkforce:undergraduates’part-timeemployment.Education and Training. 44(1),pp5–10.

dochy,f.,segers,m.,vandenbossche,P.andGijbels,d.(2003)effectsofproblem-basedlearning:ameta-analysis.Learning and Instruction. 13(5),pp533–568.

drennan,l.T.andbeck,m.(2001)TeachingqualityPerformanceindicators–keyinfluencesontheuKuniversities’scores.Quality Assurance. 9(2),pp92–102.

dunbar-Goddet,h.andTrigwell,K.(2006)A study of the relations between student learning and research-active teachers. Paperpresentedatthe14thinternationalimprovingstudentlearningsymposium,bath,4–6september.

eccles,c.(2002)TheuseofuniversityrankingsintheunitedKingdom.Higher Education in Europe. 27(4),pp423–432.

ehrenberg,r.G.(2006) What’s Happening in Public Higher Education?westport,cT:Praeger.

ewell,P.(2008)nocorrelation:musingsonsomemythsaboutquality.Change.november-december2008,40(6),pp8–13.

fearnley,s.(1995)classsize:theerosiveeffectofrecruitmentnumbersonperformance.Quality in Higher Education. 1(1),pp59–65.

feldman,K.(1984)classsizeandcollegestudents’evaluationsofteachersandcourses:acloserlook. Research in Higher Education.21(1),pp45–116.

finnie,r.andusher,a.(2005)Measuring the Quality of Post-secondary Education: Concepts, Current Practices and a Strategic Plan. Kingston,on:canadianPolicyresearchnetworks.


54

ford,J.,bosworth,d.andwilson,r.(1995)Part-timeworkandfull-timehighereducation.Studies in Higher Education.20(2),pp187–202.

Gansemer-Topf,a.,saunders,K.,schuh,J.andshelley,m.(2004)A study of resource expenditure and allocation at DEEP colleges. ames,ia:educationalleadershipandPolicystudies,iowastateuniversity.

Gardiner,l.f.(1997)Redesigning higher education: producing dramatic gains in student learning. ASHE-ERIC Higher Education Report 7. washingtondc:associationforthestudyofhighereducation.

Gibbs,G.(1995)nationalscalefacultydevelopmentforteachinglargeclasses.in:wright,a.(ed.)Teaching Improvement Practices.newyork:anker.

Gibbs,G.(1999)arethepedagogiesofthedisciplinesreallydifferent?in:rust,c.(ed.)Improving Student Learning Through the Disciplines. oxford:oxfordcentreforstaffandlearningdevelopment.

Gibbs,G.(2008)Designing teaching award schemes.york:highereducationacademy.

Gibbs,G.(2010)The assessment of group work: lessons from the literature.oxford:assessmentstandardsKnowledgeexchange.availablefrom:www.brookes.ac.uk/aske/documents/brookes%20groupwork%20Gibbs%20dec%2009.pdf[June2010].

Gibbs,G.andcoffey,m.(2004)Theimpactoftrainingofuniversityteachersontheirteachingskills,theirapproachtoteachingandtheapproachtolearningoftheirstudents.Active Learning in Higher Education.5(1),pp87–100.

Gibbs,G.anddunbar-Goddet,h.(2007)The effects of programme assessment environments on student learning. york:highereducationacademy.

Gibbs,G.anddunbar-Goddet,h.(2009)characterisingprogramme-levelassessmentenvironmentsthatsupportlearning.Assessment and Evaluation in Higher Education. 34(4),pp481–489.

Gibbs,G.andJenkins,a.(eds.)(1992)Teaching Large Classes: maintaining quality with reduced resources. london:KoganPage.

Gibbs,G.andlucas,l.(1997)courseworkassessment,classsizeandstudentperformance:1984–94. Journal of Further and Higher Education.21(2),pp183–192.

Gibbs,G.,habeshaw,T.andyorke,m.(2000)institutionallearningandteachingstrategiesinenglishhighereducation.Higher Education. 40(3),pp351–372.

Gibbs,G.,Knapper,c.andPicinnin,s.(2008a)Departmental leadership for quality teaching: an international comparative study of effective practice. london:leadershipfoundation.availablefrom:www.lfhe.ac.uk/research/projects/gibbsoxford.html[may2010].

Gibbs,G.,Knapper,c.andPicinnin,s.(2008b)disciplinaryandcontextuallyappropriateapproachestoleadershipofteachinginresearch-intensiveacademicdepartmentsinhighereducation.Higher Education Quarterly.62(4),pp416–436.

Gibbs,G.,lucas,l.andsimonite,v.(1996)classsizeandstudentperformance:1984–94. Studies in Higher Education. 21(3),pp261–273.

Gibbs,G.,morgan,a.andTaylor,e.(1982)areviewoftheresearchofferencemartonandtheGoteborgGroup:aphenomenologicalresearchperspectiveonlearning.Higher Education. 11(2),pp123–145.

Gibbs,G.,regan,P.andsimpson,o.(2006)improvingstudentretentionthroughevidencebasedproactivesystemsattheopenuniversity(uK).College Student Retention. 8(3),pp359–376.

Glass,G.v.andsmith,m.l.(1978) Meta-Analysis of Research on the Relationship of Class Size and Achievement. sanfrancisco:farwestlaboratoryforeducationalresearchanddevelopment.

Glass,G.v.andsmith,m.l.(1979)meta-analysisofresearchontherealationshipofclass-sizeandachievement.Evaluation and Policy Analysis. 1,pp2–16.

Graham,a.andThompson,n.(2001)brokenranks:usnews’collegerankingsmeasureeverythingbutwhatmatters.andmostuniversitiesdon’tseemtomind.Washington Monthly. 33(4),pp9–14.

Grunig,s.G.(1997)research,reputationandresources:theeffectofresearchactivityonperceptionsofundergraduateeducationandinstitutionalresourceacquisition. Journal of Higher Education. 33(9),pp9–14.

dimensionsofqualiTy

55

hannan,a.andsilver,h.(2000) Innovating in Higher Education: teaching, learning and institutional cultures.buckingham:Thesocietyforresearchintohighereducation/openuniversityPress.

harvey,l.andGreen,d.(1993)definingquality.Assessment and Evaluation in Higher Education. 18(1),pp9–34.

hathaway,r.s.,nagda,b.a.andGregerman,s.r.(2002)Therelationshipofundergraduateresearchparticipationtograduateandprofessionaleducationpursuit:anempiricalstudy. Journal of College Student Development.43(5),pp614–631.

hattie,J.andmarsh,h.w.(1996)Therelationshipbetweenresearchandteaching:ameta-analysis.Review of Educational Research.66(4),pp507–542.

hattie,J.andTimperley,h.(2007)Thepoweroffeedback.Review of Educational Research.77(1),pp81–112.

hattie,J.,bibbs,J.andPurdie,n.(1996)effectsoflearningskillsinterventionsonstudentlearning:ameta-analysis.Review of Educational Research.66(2),pp99–136.

havnes,a.(2008),There is a bigger story behind. An analysis of mark average variation across Programmes. europeanasssociationforresearchintolearningandinstructionassessmentconference.universityofnorthumbria.

hefce(2001)Analysis of Strategies for Learning and Teaching. Report 01/37a.bristol:highereducationfundingcouncilforengland.

hePi(2006)The Academic Experience of Students in English Universities (2006 Report). oxford:highereducationPolicyinstitute.

hePi(2007) The Academic Experience of Students in English Universities (2007 Report). oxford:highereducationPolicyinstitute.

hochschul-informations-system(2005)Eurostudent 2005: Social and Economic conditions of student life in Europe 2005. hannover:his.

hoskins,s.,newstead,s.e.anddennis,i.(1997)degreePerformanceasafunctionofage,Gender,Priorqualificationsanddisciplinestudied.Assessment & Evaluation in Higher Education. 22(3),pp317–328.

houseofcommonsinnovations,universities,scienceandskillscommittee(2009)Students and Universities. Eleventh Report

of Session 2008-09. Volume 1.london:Thestationeryoffice.

huber,m.T.andmorreale,s.(2002)Disciplinary styles in the scholarship of teaching and learning: exploring common ground. washingtondc:americanassociationforhighereducationandthecarnegiefoundationfortheadvancementofTeaching.

hunt,a.,lincoln,i.andwalker,a.(2004)Term-timeemploymentandacademicattainment:evidencefromalarge-scalesurveyofundergraduatesatnorthumbriauniversity.Journal of Further and Higher Education.28(1),pp3–18.

innis,K.andshaw,m.(1997)howdostudentsspendtheirtime?Quality Assurance in Education. 5(2),pp85–89.

Jenkins,a.(2004)A guide to the research on teaching-research relations. york:highereducationacademy.

Jenkins,a.,Jones,l.andward,a.(2001)Thelong-termeffectofadegreeonGraduatelives.Studies in Higher Education. 26(2),pp147–161.

Jessop,T.&el-hakim,y.(2010)Evaluating and improving the learning environments created by assessment at programme level: theory and methodology. europeanassociationforresearchintolearningandinstructionassessmentconference,universityofnorthumbria,June2010.

Johnes,G.(1992)Performanceindicatorsinhighereducation:asurveyofrecentwork.Oxford Review of Economic Policy. 8(2),pp19–33.

Jongbloed,b.w.a.andJ.J.vossensteyn(2001)KeepingupPerformances:aninternationalsurveyofperformancebasedfundinginhighereducation.Journal of Higher Education Policy and Management. 23(2),pp127–145.

Kehm,b.m.andstensaker,b.(2009) University rankings, diversity and the landscape of higher education. rotterdam:sensePublishers.

Kuh,G.d.andPascarella,e.T.(2004)whatdoesinstitutionalselectivitytellusabouteducationalquality?Change.september-october2004,36(5),pp52–58.

lanasa,s.,olson,e.andalleman,n.(2007)Theimpactofon-campusstudentgrowthonfirst-yearengagementandsuccess. Research in Higher Education. 48(8),pp941–966.


56

lindsay,r.andPaton-saltzberg,r.(1987)resourcechangesandacademicperformanceatanenglishPolytechnic. Studies in Higher Education. 12(2),pp213–27.

lindsay,r.,breen,r.andJenkins,a.(2002)academicresearchandteachingquality:theviewsofundergraduateandpostgraduatestudents.Studies in Higher Education. 27(3),pp309–327.

lucas,l.,Jones,o.,Gibbs,G.,hughes,s.andwisker,G.(1996)The effects of course design features on student learning in large classes at three institutions: a comparative study.Paperpresentedatthe4thinternationalimprovingstudentlearningsymposium,bath.

marsh,h.w.(1982)seeq:areliable,validandusefulinstrumentforcollectingstudents’evaluationsofuniversityteaching.britishJournal of Educational Psychology. 52,pp77–95.

marsh,h.w.(1987)students’evaluationsofuniversityteaching:researchfindings,methodologicalissues,anddirectionsforfutureresearch.International Journal of Educational Research.1(3),(entireissue).

martin,e.,Trigwell,K.,Prosser,m.andramsden,P.(2003)variationintheexperienceofleadershipofteachinginhighereducation.Studies in Higher Education. 28(3),pp247–259.

marton,f.andwenestam,c.(1978)qualitativedifferencesintheunderstandingandretentionofthemainpointinsometextsbasedontheprinciple-examplestructure.in:Gruneberg,m.m.,morris,P.e.andsykes,r.n.(eds.)Practical aspects of memory. london:academicPress.

marton,f.,hounsell,d.andentwistle,n.(1984)The experience of learning.edinburgh:scottishacademicPress.

mentkowski,m.anddoherty,a.(1984)Careering After College: Establishing the Validity of Abilities Learned in College for Later Careering and Professional Performance. Final Report to the National Institute of Education.milwaukee,wi:alvernocollege.

nasr,a.,Gillett,m.andbooth,e.(1996)lecturers’teachingqualificationsandtheirteachingperformance. Research and Development in Higher Education. 18,pp576–581.

nchems(2003)Do DEEP institutions spend more or differently than their peers? boulder,co:nationalcentreforhighereducationmanagementsystems.

nus(2008)NUS Student Experience Report. london:nationalunionofstudents.

oecd(2000)Education at a Glance 2000. Paris:organisationforeconomiccooperationanddevelopment.

olcott,d.(2010)Parforthecourse.Times Higher Education.8april,p32.

Pascarella,e.T.(1980)student-facultyinformalcontactandcollegeoutcomes.Review of Educational Research.50(4),pp545–595.

Pascarella,e.T.(2001)identifyingexcellenceinundergraduateeducation:areweevenclose?Change.33(3),pp19–23.

Pascarella,e.T.andTerenzini,P.(1991) How college affects students.sanfrancisco:Jossey-bass.

Pascarella,e.T.andTerenzini,P.(2005)How college affects students: a third decade of research, Volume 2. sanfrancisco:Jossey-bass.

Pascarella,e.T.,cruce,T.,umbach,P.,wolniak,G.,Kuh,G.,carini,r.,hayek,J.,Gonyea,r.andZhao,c.(2006)institutionalselectivityandgoodpracticesinundergraduateeducation:howstrongisthelink?Journal of Higher Education.77(2),pp251–285.

Pascarella,e.T.,seifert,T.a.andblaich,c.(2008)Validation of the NSSE benchmarks and deep approaches to learning against liberal arts outcomes. Paperpresentedattheannualmeetingoftheassociationforthestudyofhighereducation,Jacksonville,fl.availablefrom:www.education.uiowa.edu/crue/publications/index.htm[march2010].

Pascarella,e.T.,seifert,T.a.andblaich,c.(2010)howeffectivearethenssebenchmarksinpredictingimportanteducationaloutcomes?Change. January-february2010,42(1),pp16–22.availablefrom:www.changemag.org/index.html[march2010].

Paton-saltzberg,r.andlindsay,r.(1993)The Effects of Paid Employment on the Academic Performance of Full-time Students in Higher Education. oxford:oxfordPolytechnic.

Patrick,J.P.andstanley,e.c.(1998)Teachingandresearchqualityindicatorsandtheshapingofhighereducation. Research in Higher Education.39(1),pp19–41.

dimensionsofqualiTy

57

Perry,w.G.(1970)Forms of intellectual and ethical development in the college years: a scheme.newyork:holt,rhinehartandwinston.

Piccinin,s.,cristi,c.andmccoy,m.(1999)Theimpactofindividualconsultationonstudentratingsofteaching. International Journal of Academic Development. 4(2),pp75–88.

Prosser,m.andTrigwell,K.(1997)Perceptionsoftheteachingenvironmentanditsrelationshiptoapproachestoteaching.British Journal of Educational Psychology. 67,pp25–35.

qaa(2003)Learning from Subject Review 1993–2001. Gloucester:qualityassuranceagencyforhighereducation.

qaa(2006) Background Briefing Note: The classification of degree awards. Gloucester:qualityassuranceagencyforhighereducation.

qaa(2009)Thematic enquiries into concerns about academic quality and standards in higher education in England. Gloucester:qualityassuranceagencyforhighereducation.

ramsden,P.(1979)studentlearningandperceptionsoftheacademicenvironment.Higher Education. 8(4),pp411–427.

ramsden,P.(1998) Learning to lead in higher education. london:routledge.

ramsden,P.(1999)aperformanceindicatorofteachingqualityinhighereducation:thecourseexperiencequestionnaire.Studies in Higher Education. 16(2),pp129–150.

ramsden,P.andmoses,i.(1992)associationsbetweenresearchandteachinginaustralianhighereducation.Higher Education.23(3),pp273–295.

säljö,r.(1979)learningaboutlearning.Higher Education. 8(4),pp443–451.

sastry,T.andbekhradnia,b.(2007)The academic experience of students in English universities. london:highereducationPolicyinstitute.

schmitz,c.(1993)assessingthevalidityofhighereducationindicators. Journal of Higher Education. 64(5),pp503-521.

schomburg,h.andTeichler,u.(2006)Higher education and graduate employment in Europe: Results from graduate surveys from twelve countries. dordrecht:springer.

schwartz,d.l.,bransford,J.d.andsears,d.(2005) Efficiency and innovation in transfer. stanford,ca:stanforduniversity.availablefrom:www.stanford.edu/~danls/efficiency%20and%20innovation%204_2004.pdf[april2010].

simpson,o.(2003)Student retention in open and distance learning. london:routledgefalmer.

smith,m.l.andGlass,G.v.(1979)Relationship of class-size to classroom processes, teacher satisfaction and pupil affect: a meta-analysis. sanfrancisco,ca:farwestlaboratoryforeducationalresearchanddevelopment.

smith,J.andnaylor,r.(2005)schoolingeffectsonsubsequentuniversityperformance.Economics of Education Review. 24(5),pp549–562.

smith,J.,mcKnight,a.andnaylor,r.(2000)Graduateemployability:policyandperformanceinhighereducationintheuK.The Economic Journal. 110(464),pp382–411.

stinebrickner,r.andstinebrickner,T.(2008)ThecausaleffectofstudyingonacademicPerformance.The B.E. Journal of Economic Analysis and Policy. 8(1),article14.

svanum,s.andbigatti,s.m.(2006)TheinfluencesofcourseeffortandoutsideactivitiesonGradesinacollegecourse. Journal of College Student Development.47(5),pp564–576.

svensson,l.(1977)onqualitativedifferencesinlearning:iii–studyskillandlearning.British Journal of Educational Psychology. 47,pp223–243.

Terenzini,P.T,andPascarella,e.T.(1994)livingmyths:undergraduateeducationinamerica.Change. 26(1),pp28–32.

TesTa(2010)Transforming the Experience of Students Through Assessment. availablefrom:www.winchester.ac.uk/studyhere/excellenceinlearningandTeaching/research/Pages/TesTa.aspx[12august2010].

Tinto,v.(1975)dropoutfromhighereducation:atheoreticalsynthesisofrecentresearch.Review of Educational Research. 45(1),pp89–125.

Thompson,n.(2000)Playingwithnumbers:howusnewsmis-measureshighereducationandwhatwecandoaboutit.Washington Monthly. 32(9),pp16–23.

Trigwell,K.(2005)Teaching–researchrelations,cross-disciplinarycollegialityandstudentlearning.Higher Education. 49(3),pp235–254.

Trigwell,K.andashwin,P.(2004)Undergraduate students’ experience at the University of Oxford.oxford:oxfordlearninginstitute.availablefrom:www.learning.ox.ac.uk/oli.php?page=365[april2010].


58

Trigwell,K.&Prosser,m.(2004)ThedevelopmentanduseoftheapproachestoTeachinginventory.Educational Psychology Review, 16,4,pp409–424.

usher,a.andsavino,m.(2006)A world of difference: a global survey of university league tables. Toronto,on:educationalPolicyinstitute.availablefrom:www.educationalpolicy.org/pdf/world-of-difference-200602162.pdf[12august2010].

vanrossum,e.J.,deijkers,r.andhamer,r.(1985)students’learningconceptionsandtheirinterpretationofsignificanteducationalconcepts.Higher Education.14(6),pp617–641.

vos,P.(1991) Curriculum Control of Learning Processes in Higher Education.13thinternationalforumonhighereducationoftheeuropeanassociationforinstitutionalresearch.edinburgh.

warren-Piper,d.(1994) Are Professors Professional? Theorganisationofuniversityexaminations.london:JessicaKingsley.

weimer,m.andlenze,l.f.(1997)instructionalinterventions:areviewofliteratureoneffortstoimproveinstruction.in:Perry,r.P.andsmart,J.c.(eds.) Effective Teaching in Higher Education: Research and Practice. newyork:agathonPress.

wood,K.,limsky,a.s.,andstraus,m.a.(1974).classsizeandstudentevaluationsoffaculty.Journal of Higher Education.43,pp524–34.

woodley,a.(2004)conceptualisingstudentdrop-outinparttimedistanceeducation:pathologisingthenormal?Open Learning.19(1),pp47–63.

woolf,h.,cooper,a.,bourdillon,b.,bridges,P.,collymore,d.,haines,c.,Turner,d.andyorke,m.(1999)benchmarkingacademicstandardsinhistory:anempiricalexercise.Quality in Higher Education. 5(2),pp145–154.

yorke,m.(1997)agoodleaguetableguide?Quality Assurance in Higher Education. 5(2),pp61–72.

yorke,m.(1998)TheTimes‘leagueTable’ofuniversities,1997:astatisticalappraisal.Quality in Education.6(1),pp58–60.

yorke,m.(1999)Leaving early: undergraduate non-completion in higher education.london:falmer.

yorke,m.(2001)formativeassessmentanditsrelevancetoretention.Higher Education Research and Development. 20(2),pp115–126.

yorke,m.(2009)Trends in honours degree

classifications, 1994–95 to 2006–07, for England, Wales and Northern Ireland. york:highereducationacademy.

yorke,m.,bridges,P.andwoolf,h.(2000)markdistributionsandmarkingpracticesinuKhighereducation;somechallengingissues.Active Learning in Higher Education. 1(1),pp7–27.

yorke,m.,barnett,G.,bridges,P.,evanson,P.,haines,c.,Jenkins,d.,Knight,P.,scurry,d.,stowell,m.andwoolf,h.(2002)doesgradingmethodinfluencehonoursdegreeclassification? Assessment and Evaluation in Higher Education. 27(3),pp269–279.

yorke,m.,woolf,h.,stowell,m.,allen,r.,haines,c.,redding,m.,scurry,d.,Taylor-russell,G.,Turnbull,w.andwalker,w.(2008)enigmaticvariations:honoursdegreeassessmentregulationsintheuK.Higher Education Quarterly. 63(3),pp157–180.

dimensionsofqualiTy

59


60

dimensionsofqualiTy

61

www.heacademy.ac.uk

The Higher Education Academy supports the sector in providing the best possible learning experience for all students. It does this by:

— providing national leadership in developing and disseminating evidence-informed practice about enhancing the student learning experience

— operating as an independent broker, enabling expertise to be shared across institutions and subject areas

— working at multiple levels, with individual academics, subject communities, departments, faculties and institutions

— working across all parts of the UK, recognising the distinctive policy contexts and priorities of the devolved administrations but also provising opportunities to share expertise among them.

The Academy is an independent organisation funded by grants from the four UK higher education funding bodies, subscriptions from higher education institutions, and grant and contract income for specific initiatives.

Date post:	21-May-2020
Category:	Documents
Upload:	others
View:	22 times
Download:	0 times

Dimensions of quality - sparqs Dimensions of Quality.pdf · dimensions of qualiTy by Graham Gibbs 2...

Documents