The journey towards Subject-level TEF

Richard Smith

TEF Subject Pilot Manager

Office for Students

Subject-level TEF video series

1. What is the Teaching Excellence and Student Outcomes Framework (TEF)?

2. The journey towards Subject-level TEF

3. Overview of 2018-19 pilot assessment model

4. Overview of panel assessment process

5. Introduction to metrics

6. Benchmarking and significance flags

7. Data limitations

8. Contextual data

9. Attainment and WP data

Key documents

TEF principles

An independent panel of experts carry out the

assessment against criteria that matter to students

Learning

environment:

resources and

activities to support

learning and

improve retention,

progression and

attainment

Teaching quality:

teaching that

stimulates and

challenges

students, and

maximises

engagement with

their studies

Student outcomes

& learning gain: the

extent to which all

students achieve

their educational and

professional goals, in

particular those from

disadvantaged

backgrounds

How is TEF being developed?

A phased approach

How is TEF being developed?

Biennial cycle

Spring 2021: Outcomes published

Spring 2020: Application window

Autumn 2019: Decisions on full subject TEF

Academic year 2018-19

Second subject-level pilot Independent review

First pilot specification

Tested how two models generated TEF ratings at subject level and provider level:

• Explored Model A and Model B and how successfully they would have worked in practice

• Assessed whether the full range of detailed design and delivery aspects of the pilot were fit for purpose (both cross-cutting and model specific)

• Assessed considerations for delivery at scale

• Piloted a measure of teaching intensity

Specification was based on current provider level TEF (metrics and assessment criteria).

The Government consultation also sought views on these two models

Model A – ‘by exception’

Model B – ‘bottom-up’

OfS Pilot findings report

• Published alongside Government response to the subject-level TEF consultation

• No ratings were published in a way that identified individual providers

Main findings:

• Both models featured some elements that were intended to reduce burden but ultimately added to the complexity of the exercise.

• Ratings were successfully generated at provider and subject-level in both models using the existing provider-level framework.

• The method of assessment based on combining metrics and submissions worked well but there were limitations in using the data at subject-level

• Student input into submissions were generally valuable where present but inconsistently present and/or meaningful, efforts will be needed to enhance this in future

OfS Pilot findings report

• Published alongside Government response to the subject-level TEF consultation

• No ratings were published in a way that identified individual providers

Main findings:

• For providers, there was little difference in the costs of the two models

• Assessment by panels each covering a group of subjects worked well – but it will be important to test issues of scalability through the pilots

• Grouping subjects at the second level of the Common Aggregation Hierarchy (CAH2) level worked well, providing the right level of aggregation. Some refinements to the classification could be made to better reflect the diversity of provision in UK higher education – these will be tested through the pilot.

Government consultation outcomes

• Test ‘refined model’ in this year’s pilot

• Maintain CAH2 subject classification system, but test revisions through second year of pilot

• Retain existing TEF criteria, metrics, benchmarking, submissions, independent panels and rating system in subject-level TEF (testing modifications through this year’s pilot)

• Maintain current approach of allowing natural variation in distribution of ratings, and maintain current approach to absolute values

• Test reportability and assessability thresholds through this year’s pilot

• Test approaches to interdisciplinarity (better contextual data; interdisciplinary liaisons on panels)

• TEF awards to last for a minimum of four years, with reapplication possible every two years (final decision taken after this year’s pilot and independent review)

• Remove teaching intensity from TEF (but retain grade inflation)

Student research findings

• The majority of applicants surveyed (59%) in November/December

2017 had heard of TEF. A minority (15%) had used or intended to use

TEF to inform their choice. A majority (>68%) considered that subject-

level awards would be useful.

• The CAH2 subject classification was the one most understood by

applicants (of 3 tested)

• The most important factors in choosing an HE provider for applicants

and students were:• the likelihood of securing a graduate job

• potential exposure to employers, industry and workplaces

• inspiring and engaging staff was the third most important factor for applicants. For current students this was fourth after the qualification received at the end of the course

Model of assessment

2018-19 pilot assessment timeline

2018 2019

Sept Oct Nov Dec Jan Feb Mar Apr May June July

Provider selection and

trainingSubmission window

Provider evaluation

activity

Statements of findings

Feedback and evaluation

Panel recruitment Panel training Assessment

Purpose of the 2018-19 subject-level pilot

Test a revised model of subject-level assessment, which combines

successful features from the two models tested in the first pilot.

It will also develop and test a series of other refinements to improve:

• Meaningfulness for students

• Driving enhancement

• Supporting diversity of provision

• Supporting widening participation and

social mobility

• Robust evidence and assessment processes

• Effects on provider behaviour

• Value for money

The challenge of scale

We currently estimate that a full-scale exercise would require in the region of 3,500 to

4,500 subject assessments.

Drivers:

• the numbers of providers who successfully register with the OfS

• Impact of data limitations

• CAH2 architecture

In recognition of the scale of the first full subject-level TEF exercise, it will be conducted

across two academic years, 2019-20 and 2020-21, to enable it to produce robust

outcomes.

The independent reviewer will be asked to report on:

(a) the process by which ratings are determined under the scheme and

the sources of statistical information used in that process,

(b) whether that process, and those sources of statistical information,

are fit for use for the purpose of determining ratings under the scheme,

(c) the names of the ratings under the scheme and whether those

names are appropriate,

(d) the impact of the scheme on the ability of higher education providers

to which the scheme applies to carry out their functions (including in

particular their functions relating to teaching and research),

(e) an assessment of whether the scheme is in the public interest, and

(f) any other matters that the appointed person considers relevant

Copyright ©

The copyright in this presentation is held either by the Office for Students (OfS) or by the originating authors.

Please contact info@officeforstudents.org.uk for further information and re-use requests.

Thank you for listening

Enquiries

For enquiries about:

• participation in provider-level TEF or the development of subject-level TEF contact: TEF@officeforstudents.org.uk.

• TEF metrics and the TEF extranet contact: TEFmetrics@officeforstudents.org.uk.

• TEF policy contact the Department for Education (DfE): tef.queries@education.gov.uk.

For news and updates on the TEF from the OfS, we are launching a JiscMail list, TEFNEWS.

To subscribe or unsubscribe from this list, visit https://www.jiscmail.ac.uk/TEFNEWS. Subscribers to the list will be kept informed of any updates and news relating to TEF, including publications, consultations, events and opportunities to apply to be a panel member.

Key sources of information

• Results of the first pilot: www.officeforstudents.org.uk/publications/teaching-excellence-andstudent-outcomes-framework-findings-from-the-first-subject-pilot-2017-18/

• The government consultation response: www.gov.uk/government/consultations/teaching-excellence-and-student-outcomes-framework-subject-level

• IFF Research into TEF and student choice: www.gov.uk/government/publications/teaching-excellence-framework-and-informing-student-choice

• Guide to the second year of Subject-level pilots and guide to pilot data: https://www.officeforstudents.org.uk/publications/teaching-excellence-and-student-outcomes-framework-subject-level-pilot-guide/

• OfS TEF webpages: https://www.officeforstudents.org.uk/advice-and-guidance/teaching/