Date post: | 23-Dec-2014 |
Category: |
Business |
Upload: | paul-greatrix |
View: | 1,322 times |
Download: | 1 times |
League Tables:valuable market information
or dangerous nonsense?
Dr Paul Greatrix, Registrar,The University of Nottingham
Dr Tony Rich, Registrar and Secretary,University of Essex
League Tables
• Background
• Who wants to know?• Who are the providers of this valuable
information?
• Mad, bad and dangerous
• Doing it yourself
• Measuring the unmeasurable?
Background
• The US experience
• Regulatory interest
• The Times (1992)
• They sell papers...• ...and, some would suggest, create perverse
incentives
Who wants to know?
• The Government
• The State
• The Funding Councils
• Potential Students
• Alumni
• Parents, teachers, advisors
• Employers
• Overseas sponsors
• Jo Public
• The Universities
Who’s responsible for providing this valuable data?
• The Times
• Sunday Times
• Financial Times
• The Higher
• Daily Telegraph
• The Guardian
• Shanghai Jiao Tong University
• HEFCE
• BBC Online
• others...
• and, of course, Government
Invaluable information
• For all stakeholders
• Intelligent decision-making
• Better than prejudice
• Reflecting the realities of the market place
• We have a right to know
• We aren’t stupid
League tables are a bad thing...
“The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”
But we’ll use them anyway!
“Having said that, I’m not ashamed to report that we came a very creditable 79th overall, with my own department rating a particularly good score for research
- and as I remarked to the Dean, you can’t get much better than that.”
Dangerous...
• Criteria used do not reflect quality of education
• Historical data
• Variation over time
• Scores are institutional averages – mask strengths
• Distorting effect of weightings and scalings and data manipulation
• Many of criteria used are inter-related (ie not independent)
...extremely dangerous...
• Apples and elephants and paperclips
• Perverse incentives
• Hugely political – Government interest
• Open to manipulation
• Spurious precision – there’s no such thing as a good league table
• Serious consequences for universities, departments, staff and students
The Times 2006
1 Oxford2 Cambridge3 Imperial4 LSE5 UCL6 Loughborough7 Bristol8 Warwick9 Bath10 Durham
11 Edinburgh12 Royal Holloway13 Aston14 Nottingham15 York16 Cardiff17= UEA17= King’s19= SOAS, Leicester, St Andrews
Sunday Times 2006
1 Cambridge2 Oxford3 LSE4 Imperial5 UCL6 Warwick7 York8 Durham9 Bristol10 St Andrews
11 Bath12 Nottingham13 King’s14 Edinburgh15 Manchester16 Southampton17 Loughborough18 Exeter19 UEA20 Sheffield
The Guardian 2006
1 Cambridge2 Oxford3 LSE4 UCL5 Imperial6 SOAS7 King’s8 Warwick9 Bath10 Edinburgh
11 Nottingham12 Surrey13 Bristol14 Manchester15 York16 Birmingham17 Leeds18 Goldsmiths19 Aston20 Royal Holloway
Financial Times 2003
1 Cambridge2 Oxford3 Imperial4 LSE5 UCL6 Warwick7 Bristol8 York9 Nottingham10 King’s
11 UMIST12 Bath13 Loughborough14 Edinburgh15 Manchester16 Birmingham17 SOAS18 Leicester19 St Andrews20 Southampton
‘Poll of polls’(Daily Telegraph 2003)
1 Cambridge2 Oxford3 Imperial4 LSE5 Warwick6= Nottingham6= UCL8 York9 Bristol10 Manchester
11 SOAS12 Bath13 Birmingham14 Loughborough15 King’s16 = Durham16= St Andrews18= Southampton18= Edinburgh20 Sheffield
Times HigherWorld Rankings 2006
2 Cambridge (3)3 Oxford (4)9 Imperial (13)17 LSE (11=)25 UCL (28)33= Edinburgh (30)40 Manchester (35)46= King’s (73=)
64= Bristol (49)70= SOAS (103=)73 Warwick (77=)81 Glasgow (101=)90= Birmingham
(143)85 Nottingham (97)99= Queen Mary
(112)
Shanghai Jiao Tong 2006
2 Cambridge (2)10 Oxford (10)23 Imperial (23)26 UCL (26)50 Manchester (53)52 Edinburgh (47)
62 Bristol (64)69 Sheffield (65)79 Nottingham (83)83 King’s (80)90 Birmingham (98)
And a completely different approach:UEL: TQ in the South East 2005
Surplus of good practice over recommendations(in QAA audit reports)
1 East London 42 King's College London 23 Brunel 24 Queen Mary, London 05 Kent 06 Hertfordshire 07 Royal Holloway -18 London South Bank -39 Greenwich -610 Essex -611 Anglia Polytechnic -9
Different approaches 1
• The Times– 9 indicators, z-scoring used– Teaching weighted @ 1.5 (now using only NSS)– Research weighted @ 1.5
• The Financial Times (if they ever do again)– 15 indicators, z-scoring used– Weighting of T and R, both 15%
• The Sunday Times– 9 indicators - student satisfaction weighted x1.5 (NSS);
teaching excellence x1.0– A level scores weighted x2.5 and research weighted x2– Also – Heads’ assessment x1 (up from 0.5 in 2005) and drop-
out rate (variable)
Different approaches 2
• The Guardian– 7 indicators in subject tables (research ratings do not
feature, no longer using TQA but not adopted NSS)– T staff qualifications weighted @ 15% in each subject
table– Entry qualifications and SSR @ 20%– Spend per student and Value-added @ 10%– Destinations @ 17%– ‘inclusiveness’ @ 8%– Overall table in 2006 is an average of subject tables
• The Daily Telegraph– just the one indicator (keeps things nice and simple)
International approaches 1
Shanghai Jiao Tong: Ranking of World Universities
– 6 indicators covering: quality of education; quality of faculty; research output; performance relative to size
– Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science; articles in citation indices.
International approaches 2
Times Higher World University Rankings
– Peer review: 40%
– Citations per Faculty member and SSR: each 20%
– Recruiter review: 10%
– Proportions of international Faculty and students: each 5%
The critical factors(by frequency of appearance in the tables)
• Subject review/teaching quality results/NSS
• A level scores
• Staff:Student Ratios
• Graduate employment rates
• RAE results
• Spending on library and computing
• Degree classifications
• Completion rates
One-offs(only appeared in one UK table)
• Spending on facilities
• Access
• Value-added
• %age of graduates entering professional training
• %age of graduates entering higher degree
• %age of postgraduate research students
• %age of taught postgraduate students
• Income from industry
• Income from research
• Applications to places
• Inclusiveness
An indicator too far(not covered in the UK tables - yet)
• Alumni giving
• Academic staff pay
• Percentage of full-time academic staff
• Citations
• Brand impact
But international tables will increasingly influence methodologies of UK tables
Conclusions
• They aren’t going to go away
• The international dimension will become increasingly significant
• They can and will be used by many different groups – but can be dangerous in the wrong hands
• Handle with great care!