The Research Excellence Framework

Design Research SocietyNottingham October 2011

Main Panel D Criteria and Working Methods

• Main Panel D covers:

• Area Studies

• Modern Languages and Linguistics

• English Language and Literature

• History

• Classics

• Philosophy

• Theology and Religious Studies

• Art and Design: History, Practice and Theory

• Music, Drama Dance and Performing Arts

• Communication, Cultural and Media Studies; Library and Information Management

REF Sub-panel 34: Art and Design: History, Practice and TheoryChair

Professor Paul Greenhalgh, Sainsbury Centre for Visual Arts University of East Anglia

Deputy ChairAnne Boddington, University of Brighton

Panel SecretaryMs Brenda Purkiss, University of Cambridge

MembersProfessor Oriana Baddeley, University of the Arts London

Professor Naren Barfield, Glasgow School of ArtProfessor Tim Benton, The Open University

Professor Sandy Black, London College of Fashion, University of the Arts LondonProfessor Stephen Boyd Davis

Dr Christopher Breward Victoria and Albert MuseumProfessor Brendan Cassidy, University of St Andrews

Professor Rachel Cooper, Lancaster UniversityDr Colin Cruise, Aberystwyth University

Professor Juan Cruz, Liverpool John Moores UniversityProfessor Stephen Dixon, Manchester Metropolitan University

Dr Beth Harland, Winchester School of Art

Members continued Michael Horsham, Tomato Design Consultants

Professor Deborah Howard University of CambridgeProfessor Nigel Llewellyn, Tate

Professor Judith Mottram, Nottingham Trent UniversityProfessor Magdelana Odundo, University for the Creative Arts

Professor Stephen Partridge University of DundeeProfessor James Roddis Sheffield Hallam University

Professor Irit Rogoff Goldsmiths, University of LondonProfessor Emma Rose, Lancaster University

Professor Paul Seawright, University of UlsterProfessor Penny Sparke, Kingston University

Dr Peter Stewart Courtauld Institute, University of LondonDeyan Sudjic OBE, Design Museum

Professor Sue Walker, University of ReadingProfessor Evelyn Welch, Queen Mary, University of London

REF is not RAE

Key Differences Between RAE and REF

•Impact introduced.

•Increased role of ‘users’.

•Fewer UoA’s with more subject range.

•Role of Main Panel.

•Removal of ‘Esteem’ as a criterion.

•Adjustment of Environment as a criterion.

The overall assessment framework

Overall qualityOverall quality

OutputsOutputs

‘rigour, originality and significance’

‘rigour, originality and significance’

Maximum of 4 outputs per researcher

Maximum of 4 outputs per researcher

EnvironmentEnvironment

‘vitality and sustainability’‘vitality and sustainability’

Template and dataTemplate and data

ImpactImpact

‘reach and significance’‘reach and significance’

Template and case studiesTemplate and case studies

65% 15% 20%

Outputs Environment Impact

4* 3* 2* 1* u/c

0 40 40 20 0

4* 3* 2* 1* u/c

20 45 35 0 0

65%

Overall Quality Profile

12

4*

0104236

u/c1*2*3*

4* 3* 2* 1* u/c

12.8 32.5 43.3 11.4 0

15% 20%

Quality Level

% of the submission

Main Panel D Criteria and Working Methods

• Requests for multiple submissions need to meet the criteria

specified in the Guidance on Submissions

• Requests are expected in:

• Area Studies

• Modern Languages

• Art and Design

• Music, Drama, Dance and Performing Arts

• Communication, Culture and Media Studies, Library and

Information Management

Assessment Criteria: Outputs

Eligible Outputs:

• Any type of output embodying research as defined for the

REF may be submitted

• Sub-panels will not privilege any one kind of output above

another

• Journal rankings will not be used!

Assessment Criteria: OutputsEligible Outputs:

• The output should be submitted without additional material

where that is in itself deemed to constitute sufficient

evidence of the research

• Additional information for practice-based outputs - 300

words describing the research imperatives, research

process and research significance

• Portfolio in cases where the research output is ephemeral, is

one in a series of inter-connected outputs (eg performances

etc) or cannot fully represent its scholarly dimensions

through the evidence provided above.

Assessment Criteria: OutputsCo-authored, co-edited and collaborative outputs:

• May be listed by more than one author from within a single

submitting unit or across submissions from different units

• In all cases submissions are required to provide an

explanation of the nature and scale of the author’s

contribution - not expressed as a %

• Panels may judge that significant differences in the quality of

the respective contributions should be taken into account in

the final grades awarded

Assessment Criteria: OutputsDouble-Weighting:

• To recognise outputs of extended scale and scope = to 2 outputs. No

particular type of output will automatically be double-weighted

• Institutions may identify up to 2 outputs per individual author which

they consider worthy of double weighting and submit a supporting

statement

• Panels will assess the claim for double weighting separately from the

quality of the output (ie double-weighting does not necessarily result

in 2 x 4*)

• For each claim, institutions may submit a reserve output, which will

only be assessed if the claim for double weighting is not justified.

Assessment Criteria: Impact

• Definitions for the criteria for assessing impact are:

Reach: The extent and/or diversity of the organisations, communities and/or individuals who have benefitted from the impact. Significance: The degree to which the impact enriched, influenced, informed or changed the policies, practices, understanding and awareness of organisations, communities and/or individuals.

Assessment Criteria: Impact• The Main Panel believes that the impact of research conducted

in its disciplines is powerful, pervasive and ubiquitous;

challenging imaginations and enriching lives economically,

culturally, spiritually and educationally

• It has provided, as illustration, a range of areas of impact, to help

institutions to think about what case studies in the arts and

humanities might look like

• These are: civil society, cultural life, economic prosperity,

education, policy making, public discourse, public services

• There is no expectation that case studies should be classified in

this way; indeed case studies may well cross the boundaries of

these areas or go well beyond them

Assessment Criteria: Impact

Examples of Impacts:

• A short list of examples of impact is provided in the panel criteria

• These are drawn from lengthy lists put together by sub-panels,

which we would like to publish in due course as an aid to the

sector

Assessment Criteria: Impact

Evidence of Impact

• Main Panel D acknowledges that all potential records of

evidence might not be available and that the integrity, coherence

and clarity of the narrative will be essential to the panels in

forming their judgements, but nonetheless key claims made in

the narrative should be capable of corroboration

• Narratives should articulate the relationship between the

underpinning research and the impact as well as the reach and

significance of the impact itself

• An extensive range of types of evidence that could be used to

support case studies is provided in the criteria to assist

institutions in compiling their case studies

Assessment Criteria: Impact

• The Impact Template has four sections and will comprise

20% of the impact sub-profile:

• Context

• Approach to Impact

• Strategies and Plans

• Relationship to Case Studies

• Main Panel D has explained in its criteria the kinds of

information it would like to see under these headings; not

exhaustive lists

• In particular, it recognises that there is not always a planned,

causal link between research and its subsequent impact and

that pathways to impact may be diffuse and non-linear.

Assessment Criteria: Environment

• Definitions for the criteria for assessing environment are:

Vitality: The extent to which the research environment supports a research culture characterised by intellectual vigour, innovation and positive contribution to the discipline.  Sustainability: The extent to which the research environment ensures the future health and well-being of the unit and the discipline.

Assessment Criteria: Environment

• Data required [REF 4a/b/c]

• Data requirements have been reduced since RAE2008 to

the following three datasets(by year, for the period 1 August

2008 – 31July 2013):

• Doctoral awards

• Research income by source

• Research income-in-kind

• Main Panel D has not asked for any other additional data to be submitted.

• This will be considered alongside the information provided in the environment template

Assessment Criteria: Environment

• Environment template [REF5] (equivalent to RA5)

• Headings: Overview; Strategy; People (covering staffing

strategy and staff development, and research students);

Income, Infrastructure and facilities; collaboration and

contribution to the discipline.

• Panel Criteria specifies the kinds of information sub-panels

would like to see under these headings; these are not

exhaustive lists.

• Word lengths linked to number of ftes submitted.

Assessment Criteria: Working Methods• Interdisciplinary and multidisciplinary research welcomed

and treated equally. Sub-panels members have been

selected to embrace broad-ranging experience to enable

assessment of such work and work that crosses UOA

boundaries.

• Within Main Panel D, cross-referral will be characterised by

dialogue between the relevant SPs.

• Cross-referrals to other Main Panels if necessary.

Assessment Criteria: Working Methods

• Additional assessors (both academic and user) will be

appointed to each sub-panel to assist with the assessment

phase.

• Sub-panels will review institutional Statements of

Submission Intentions to identify gaps in expertise or areas

where the workload will be significantly heavier than

anticipated.

• There will be an appointments process which will take due

regard of advice received from subject associations and

other professional bodies.

Assessment Criteria: Working Methods• Main Panel will work with sub-panels to ensure adherence

to assessment criteria and consistent application of standards. Details defined in Panel Criteria.

• Sub-panels will ensure that submissions are assessed using appropriate expertise: approaches defined in Panel Criteria.

• User members and user assessors will contribute significantly to the assessment of impact.

Assessment Criteria: Working Methods

Reviewing Outputs

“In every submission, all outputs will be examined with a level of detail sufficient to contribute to the formation of a reliable quality profile for all the outputs in that submission.”

 

Further information

• Guidance on submissions (July 2011)

• Draft panel criteria and working methods (July 2011)

www.ref.ac.uk

info@ref.ac.uk