Impact, explosion, fallout: what have we learned
from the 2014 Research Excellence Framework?
Roger Watson
Purpose of the REF• The REF is a process of expert review
• It replaces the RAE as the UK-wide framework for assessing research in all disciplines
• Its purpose is:
- To inform research funding allocations by the four UK HE funding bodies (approximately £2 billion per year)
- Provide accountability for public funding of research and demonstrate its benefits
- To provide benchmarks and reputational yardsticks
Overview:
Overview of the processREF assessed the
quality of research in all UK universities, in all
disciplines. It was carried out by 36 expert panels, grouped into 4
main panels.
2011-12PreparationPanels were appointed.Guidance and criteria were published.
2012-13SubmissionsUniversities made submissions in whichever subjects they chose to.
2014Assessment36 expert panels reviewed the submissions, guided by the 4 main panels.
Main Panel A: Medical and life sciences
Main Panel B: Physical sciences and engineering
Main panel C: Social sciences
Main Panel D: Arts and humanities
Expert Panels• Submissions were assessed by 36 Sub-panels working under
the guidance of four Main Panels
Each Main Panel comprised:• The chair
• Chairs of each sub-panel
• International members
• User members
Each Sub-Panel comprised:• The chair and deputy chair
• Panel members
• Additional assessors (for outputs and
impact)
• On average ~30 people
• The Equalities and Diversity Panel (EDAP) reviewed complex staff circumstances
Working methods
• Work was allocated to members and assessors according to expertise
• Outputs were cross-referred to other sub-panels where necessary
• Strict procedures were in place to avoid conflicts of interest• Significant attention was given to ensure comparability of
assessment standards, at sub- and main panel level: − Calibration exercises− Ongoing review of emerging scores
• Where used, citation data played a minor role in informing peer review of the outputs
Development of sub-profiles• Each outputs was graded on the scale:
4* : World-leading
3* : Internationally excellent
2* : Recognised internationally
1* : Recognised nationally
U : Below the standard for 1* or deemed ineligible• Each impact case study, the impact template, and each section
of the environment template was graded on a scale from 4* - U, using ‘half-marks’ for borderline judgments.
• Each sub-profile for each submission was collectively agreed by the sub-panel and recommended to the main panel for approval.
• The overall quality profile for each submission aggregates the three sub-profiles, according to the standard weightings.
EXAMPLE - 2014 Research Excellence Framework ResultsQuality profiles for all submissions (sample)
Institution name
Main panel
Unit of assessment name Profile
FTE Category
A staff submitte
d 4* 3* 2* 1*unclassified
University X A Biological Sciences Outputs 50.45 12.8 32.8 43.0 11.4 0.0University X A Biological Sciences Impact 50.45 20.0 45.0 35.0 0.0 0.0University X A Biological Sciences Environment 50.45 0.0 40.0 40.0 20.0 0.0University X A Biological Sciences Overall 50.45 12 37 41 0 0University X B General Engineering Outputs 65.20 25.9 43.1 27.0 4.0 0.0University X B General Engineering Impact 65.20 17.9 60.1 21.0 1.0 0.0University X B General Engineering Environment 65.20 10.0 70.0 20.0 0.0 0.0University X B General Engineering Overall 65.20 22 51 24 3 0University X A Architecture Outputs 40.00 17.0 60.0 20.0 2.0 1.0
The overall quality profile is weighted 65% for Outputs, 20% for Impacts, 15% Environment
Published results include the three sub-profiles and the overall quality profile for each submission
For the first time, REF has demonstrated the impact of UK research
• Across the exercise, over 250 research users judged the impacts, jointly with academic panel members.
• Across the exercise, 44% of impacts were judged outstanding (4*) and a further 40% were judged very considerable (3*).
• Impressive impacts were found from research in all subjects.
• REF shows many ways in which research has fuelled economic prosperity, influenced public policy and services, enhanced communities and civic society, enriched cultural life, improved health and wellbeing, and tackled environmental challenges.
HEFCE HEI BUDGET: £4 billion
The assessment framework
Overall quality
Outputs
Maximum of 4 outputs per researcher
Impact
Impact template and case studies
Environment
Environment data and template
65% 20% 15%
Overview:
Submissions • Each HEI may submit in any or all of the 36 units of
assessment (UOAs)
• Each submission in a UOA provides evidence about the activity and achievements of a ‘submitted unit’ including:
- Staff details (REF1a/b/c)
- Research outputs (REF2)
- Impact template and case studies (REF3a/b)
- Environment data (REF4a/b/c)
- Environment template (REF5)
• A submitted unit may, but need not, comprise staff who work within a single ‘department’ or organisational unit
Overview:
Publication of results• The primary outcome of the REF is an ‘overall quality
profile’ to be awarded to each submission:
- Using the same scale as RAE2008, but in steps of 1%
• Further reports and feedback will be provided:
- Overview reports by panels
- Concise feedback on submissions, to the heads of HEIs
- The output, impact and environment sub-profiles for each submission will be published
- A report by the Equality and Diversity Advisory Panel
• Submissions will be published (except for confidential or sensitive information)
Overview:
The results have been published!
Outputs Impact Environment
4* 3* 2* 1* U
20 45 35 0 0
4* 3* 2* 1* U
0 40 40 20 0
65%
Overall Quality Profile
12
4*
0104137
U1*2*3*
4* 3* 2* 1* U
12.8 32.8 43 11.4 0
20% 15%
The overall quality profile is comprised of the aggregate of the weighted sub-profiles produced for outputs, impact and environment.
Quality Level
% of Research Activity
Example of a quality profileOverview:
Main panel working methods• Each main panel has developed a consistent set of
criteria for its group of sub-panels• Each main panel will guide its sub-panels throughout
the assessment phase, ensuring:- Adherence to the published criteria
- Consistent application of the overall standards of assessment
• Main panels will undertake calibration exercises and keep the emerging outcomes under review
• Main panel international and user members will be engaged at key stages across the sub-panels
REF panels:
Sub-panel working methods• Sub-panels will review their expertise to ensure
appropriate coverage
• Work will be allocated to members/assessors with appropriate expertise
• Each sub-panel will run calibration exercises for outputs and impacts, guided by the main panels
• All outputs will be examined in sufficient detail to contribute to the formation of the outputs sub-profiles
• Each case study will normally be assessed by at least one academic and one user
• Graduated sub-profiles will be formed for each aspect of submissions
REF panels:
Definition of impact• Impact is defined broadly for the REF: an effect on,
change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia
• Panels recognise that impacts can be manifest in a wide variety of ways, may take many forms and occur in a wide range of spheres, in any geographic location
• Panels provide examples of impact relevant to their disciplines, intended to stimulate ideas - not as exhaustive or prescriptive lists
Impact:
Some examples of impactImpact:
Public debate has been shaped or informed by research
A social enterprise initiative has been created
Policy debate or decisions have been influenced or shaped by research
A new product has been commercialised
Enhanced professional standards, ethics, guidelines or training
Jobs have been created or protected
Improved business performance
Changes to the design or delivery of the school curriculum
The policies or activities of NGOs or charities have been informed by research
Improved management or conservation of natural resources
Improved forensic methods or expert systems
Production costs have reduced
Levels of waste have reduced
Improved quality, accessibility or efficiency of a public service
Enhanced preservation, conservation or presentation of cultural heritage
Organisations have adapted to changing cultural values
New forms of artistic expression or changes to creative practice
More effective management or workplace practices
Changes to legislation or regulations
Enhanced corporate social responsibility policies
Research has informed public understanding, values, attitudes or behaviours
Improved access to justice, employment or education
Enhanced technical standards or protocols
Improved risk management
Improved health or welfare outcomes
Research has enabled stakeholders to challenge conventional wisdom
Changes in professional practice
Submission requirements• Sets out the submitted unit’s general
approach to supporting impact from its research:
• Approach to supporting impact during the period 2008 to 2013
• Forward strategy and plans
Impact template (REF3a)
• Specific examples of impacts already achieved, that were underpinned by the submitted unit’s research:
• 1 case study per 10 FTE staff submitted (plus 1 extra)
• Impacts during 2008 to 2013; underpinned by research since 1993
Case studies (REF3b)
Impact:
20% of the impact sub-
profile
80% of the impact sub-
profile
Case studies • Each case study should:
- Clearly describe the underpinning research, who undertook it and when
- Provide references to the research and evidence of quality
- Explain how the research led/contributed to the impact
- Clearly identify the beneficiaries and define the impact
- Provide evidence/indicators of the impact
- Provide independent sources of corroboration
• All the material required to make a judgement should be included in the case study
• Submitted case studies need not be representative of activity across the unit: pick the strongest examples
Impact:
Underpinning research• Each case study must be underpinned by research that:
- was produced by staff while working in the submitting HEI
- is evidenced by outputs published between 1 Jan 1993 to 31 Dec 2013
- meets the quality threshold of at least equivalent to 2*
- made a material and distinct contribution to the impact (there are many possible ‘routes’ to impact, but in each case a distinct and material contribution must be shown)
• Once the panel is satisfied that these criteria have been met, it will assess and grade the case study in terms of the ‘reach and significance’ of the impact
Impact:
Evidence of impact• Case studies should provide a clear and coherent
narrative linking the research to the impact
• Including evidence most appropriate to the case being made
• Evidence may take many different forms, including quantitative (where possible) and qualitative. Panels provide examples, which are not exhaustive or prescriptive
• Key claims should be capable of verification. Independent sources of corroboration should listed, to be used for audit purposes
Impact:
Assessment criteria• The criteria for assessing impact are reach and
significance
• In assessing a case study, the panel will form an overall view about the impact’s reach and significance taken as a whole, rather than assess each criterion separately
• ‘Reach’ is not a geographic scale. Sub-panels will consider a number of dimensions to the ‘reach’ as appropriate to the nature of the impact.
• In assessing the impact template, the panel will consider the extent to which the unit’s approach is conducive to achieving impacts of ‘reach and significance’
Impact:
Assessment criteriaThe criteria for assessing impacts are reach and significance*
Four star Outstanding impacts in terms of their reach and significance
Three star Very considerable impacts in terms of their reach and significance
Two star Considerable impacts in terms of their reach and significance
One star Recognised but modest impacts in terms of their reach and significance
UnclassifiedThe impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit
* Each main panel provides descriptive account of the criteria
Impact:
What counts as impact?
Impact includes but is not limited to, an effect on, change or benefit to:
• the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
• of an audience, beneficiary, community, constituency, organisation or individuals
• in any geographic location whether locally, regionally, nationally or internationally
What counts as impact?
Impact includes the reduction or prevention of harm, risk, cost or other negative effects.
What does not count as impact?
For the purposes of the impact element of the REF:
• impacts on research or the advancement of academic knowledge within the higher education sector are excluded
• impacts on students, teaching or other activities within the submitting HEI are excluded.
• other impacts within the higher education sector, including on teaching or students, are included where they extend significantly beyond the submitting HEI
Reach and significance
Impacts will be assessed in terms of their ‘reach and significance’ regardless of the geographic location in which they occurred, whether locally, nationally or internationally.
Common errors in impact statements
• Not following the guidance
• Including inappropriate impact– Research impact– Teaching impact
• Including staff not employed in the correct period
• Using personal letters in support of impact
• URLs that led to no information
• Simply not addressing the issue of impact
Easy shots
• Be familiar with the guidance in detail
• Follow the guidance to the letter
• For named staff, provide dates of employment
• Don’t ‘overuse’ media consultants and journalists
• Don’t use too many inane diagrams
Things to do
• Respond to all and any consultations on 2020 REF
• Build impact into research proposals
• Consider impact at research meetings
• Start gathering evidence now
• Create the evidence now:– Webpages with counters– Research outcomes translated for the general
public– Hold public understanding of science lectures– Use social media effectively
Further informationwww.ref.ac.uk includes:
• The results and submissions
• Summary data and analysis
• Panel overview reports
• Confidential feedback on submissions was provided to institutions in January
• Evaluations of the REF are currently being carried out by the funding bodies, with reports due in March