Removing barriers: Financial

transparency, peer effects and

prior attainment

� Simon Burgess, Julia Carey at EFM, University of Bristol

� Catherine Dilnot, Lindsey Macmillan at UCL/IOE

� Raj Chande, Michael Sanders at Behavioural Insights Team

� Also

� Abbi Stoneman, Christine Spencer and Maggi Walton, Project administrators

� Kate Guest from Teach First ran the mentor training

� Amy Butterworth, Deputy Head of Admissions at UoB helped with understanding data and provided admissions data

� All the student mentors.

UoB WP Workshop, Oct. 2016 2

Team:

This talk� Issue

� ‘Model’ and Interventions

� Aims

� Operations

� Analysis:

� Methodology

� Data

� Preliminary results

� Timetable

� Conclusions

UoB WP Workshop, Oct. 2016 3

Issue� Despite huge rise in numbers attending university, the

rate of return is still high for most students.

� How can we help more students from disadvantaged backgrounds attend the best universities?

� It may all be about bursaries … But recent research suggests that other factors may matter as much.

UoB WP Workshop, Oct. 2016 4

‘Model’ and Interventions� We believe there are other important barriers to access:

� Not understanding the true financial costs and benefits of going to a Russell Group university

� Not believing that a Russell Group uni is “for people like me”

� Not having the very top grades to get in

� Our interventions address these

� We train UoB students to go into local schools to deliver: � ‘Inspire’ talks

� Additional small-scale tutoring

UoB WP Workshop, Oct. 2016 5

Aims� The over-arching aim was to:

� improve access to the University of Bristol …

� … and universities more broadly …

� … for disadvantaged children in local schools and colleges.

� We will provide causal evidence on the efficacy of this approach using an RCT methodology.

� We have written a Handbook on managing this scheme.

UoB WP Workshop, Oct. 2016 6

Operations� Intervention from 2013/14 to 2015/16, finishing June 2016.

� Recruit local schools to take part

� ‘Local’ was broad including disadvantaged areas of Weston-super-Mare.

� Included regular state secondary schools and Further Education colleges

� Recruit UoB students to act as mentors:

� Fresher’s Fair and Volunteer’s Fair, and later through email, undertaken each year

� Work with Teach First to train the students for both roles

� Teach First do this for free, undertaken each year.

� Match mentors to schools, and organise the practicalities:� Funded the travel of the mentors but they were volunteers, so not paid.

UoB WP Workshop, Oct. 2016 7

Analysis� Methodology

� Data

� Preliminary Results

UoB WP Workshop, Oct. 2016 8

Methodology� RCT approach

� School-year groups are randomised into Control, ‘Inspire’, ‘Tutoring’.

� Control = “business as usual”.

� ‘Inspire’ talks:

� We treated whole school-year groups

� School-year group level analysis

� Tutoring

� Five pupils selected by school (not random) for tutoring

� Pupil-level analysis using matched pupils in Control schools

UoB WP Workshop, Oct. 2016 9

Data� ‘Inspire’ talks: school-year-calendar year level data on

applications from UCAS:

� UoB

� any RG

� any HE

� Mentoring: pupil level data from NPD:

� GCSE grades

� A level grades

UoB WP Workshop, Oct. 2016 10

� Sample: 43 schools and colleges in the trial:

� 7 are 16+ colleges only (so only appear in KS5 data)

� 10 have no sixth form (so only appear in KS4 data)

� 26 schools are all through, so appear in both

UoB WP Workshop, Oct. 2016 11

Trial schools

UoB WP Workshop, Oct. 2016 12

State schools and colleges: In the Trial All in Bristol,

Gloucestershire and

Somerset

In England

KS4 KS5 KS4 KS5 KS4 KS5

Number of students 6,857 7,594 23,971 16,776 593,435 395,457

Number of schools/colleges 36 33 174 98 3,775 2,392

% female 48.1 53.7 48.6 52.5 48.8 53.2

FSM (% of non-missing) 12.7 7.0 10.3 5.7 14.9 10.7

SEN (% of non-missing) 13.9 8.3 14.3 8.2 17.0 10.4

SSEN (% of non-missing) 2.2 0.9 3.4 0.7 3.8 1.1

EAL (% of non-missing) 7.5 6.3 4.7 4.9 12.7 15.1

Ethnicity: % Asian (of non-missing) 3.6 2.9 2.0 2.3 8.2 10.3

Ethnicity: % Black (of non-missing) 4.0 2.6 1.9 1.8 4.8 5.7

Ethnicity: % Mixed (of non-missing) 4.2 3.9 3.1 3.3 3.8 4.0

Ethnicity: % White (of non-missing) 87.5 89.2 91.8 91.0 80.6 76.9

Ethnicity: % AOEG/Chinese/Unclear (of

non-missing)

0.8 1.4 1.2 1.6 2.6 3.1

GCSE mean capped points score

including equivalents

333 370 332 377 333 375

% with 5 A*-C – GCSE and GNVQ 53.5 79.4 56.8 82.2 55.1 78.3

% with A-levels as main qualification N/A 56.9 N/A 61.4 N/A 57.5

% with NVQ as main qualification N/A 41.3 N/A 36.7 N/A 39.9

% achieving 3 or more A*-E grades in A-

levels/applied A-levels

N/A 42.5 N/A 48.3 N/A 46.6

Mean points from A-levels (across all

students)

N/A 415 N/A 464 N/A 445

Mean number of A-level entries per

student

N/A 2.0 N/A 2.2 N/A 2.1

Mean points per A-level entry N/A 199 N/A 205 N/A 202

Trial schools – extract …

UoB WP Workshop, Oct. 2016 13

State schools and colleges: In the Trial All in Bristol,

Glos & Somerset

KS4 KS5 KS4 KS5

FSM (% of non-missing) 12.7 7.0 10.3 5.7

EAL (% of non-missing) 7.5 6.3 4.7 4.9

% with 5 A*-C – GCSE and GNVQ 53.5 79.4 56.8 82.2

% with A-levels as main

qualification

N/A 56.9 N/A 61.4

% with NVQ as main qualification N/A 41.3 N/A 36.7

% achieving 3 or more A*-E grades

in A-levels/applied A-levels

N/A 42.5 N/A 48.3

Preliminary results� Our research is ongoing and these are preliminary

findings.

� Our determination to give the actual implementation as long as possible to run, and the relative slowness of the process of accessing NPD data means that there is more to do.

UoB WP Workshop, Oct. 2016 14

Results 1: Tutoring� We provided tutoring to 108 students from year 9 to 11

over the three years. NPD matched 106.

� Only 29 appear in KS4 data that we have so far.

� So – small numbers and very preliminary.

� Control group from same LAs as our schools, matched using very local SES score, KS2 and gender.

� Capped GCSE scores as a summary measure:

UoB WP Workshop, Oct. 2016 15

UoB WP Workshop, Oct. 2016 16

Treated

GCSE score

Controls CGSE

score

Difference

Unmatched 373 302 71

ATT 373 342 31

So about an 8% increase in GCSE points PRELIMINARY

Results 2: ‘Inspire’ Talks� Outcome measure is UCAS “applications” to UoB, RG,

All HE.

� Analysis uses the randomisation plus a difference-in-difference approach

� Include school fixed effects and application-year effects.

UoB WP Workshop, Oct. 2016 17

All HE Applications

UoB WP Workshop, Oct. 2016 18

(1)

HE applications

(total)

b/se

treat 14.948

(65.521)

application_cycle=2013 0.000

(.)

application_cycle=2014 50.441

(48.704)

application_cycle=2015 -4.546

(51.549)

application_cycle=2016 29.831

(52.507)

Constant 801.487***

(34.819)

N 95

(About 2% baseline; if each kid applies to 5 unis, that’s almost 3 more kids)

Russell Group Applications

UoB WP Workshop, Oct. 2016 19

(1)

RG applications

(total)

b/se

treat 0.528

(11.030)

application_cycle=2013 0.000

(.)

application_cycle=2014 21.462*

(8.199)

application_cycle=2015 4.872

(8.678)

application_cycle=2016 21.933*

(8.839)

Constant 124.829***

(5.862)

N 95

UoB Applications

UoB WP Workshop, Oct. 2016 20

(1)

UoB applications

(total)

b/se

treat 1.440

(2.411)

Application cycle=2013 0.679

(1.829)

Application cycle=2014 1.804

(1.829)

Application cycle=2015 0.000

(.)

Application cycle=2016 3.363+

(1.772)

Constant 22.439***

(1.358)

N 94

UoB Apps: Longer run of data

UoB WP Workshop, Oct. 2016 21

UoB Apps: “before” and “after”

UoB WP Workshop, Oct. 2016 22

Commentary 1� � Preliminary results �

� No statistical significance, but that might change

� Positive results, very worthwhile but not huge:

� ‘Inspire’ Talks on

� All HE applications

� ‘Local RG’ university = University of Bristol

� Tutoring on

� GCSE points

UoB WP Workshop, Oct. 2016 23

Commentary 2� Policy Lessons?

� To the extent that UoB WP own practices are somewhat similar to ours, suggests that they may be effective too.

� But two caveats:

� Control condition was “business as usual” so that includes the activities of the University’s own WP team. Our intervention raised applications above and beyond their activities, suggesting pupils don’t see them as the same.

� No power to isolate the impact of the different components of our intervention. So cannot know whether the most effective components are or are not part of UoB existing practices.

UoB WP Workshop, Oct. 2016 24

Timetable� Sharpening results on Inspire Talks

� Now through December 2016.

� Revising results with the 2016/17 Admissions data, Spring 2017.

� Undertaking pupil level analysis on mentoring� Spring 2017, using the final release pupil level data for 2016 KS4,

available next April.

� Dissemination� Late Spring 2017

� We have good links with: the Russell Group, individual leading universities, the Cabinet Office, the Department for Education, the Social Mobility Commission

UoB WP Workshop, Oct. 2016 25

Conclusion: Beneficiaries of the scheme� Main focus: local schools and pupils (hopefully)

� University of Bristol students:� the addition to their cvs� the training, experience and ‘warm glow’ of helping out in

their adopted city. � invaluable soft skills improving their future employability.

� University of Bristol: � enhanced student experience� capability to attract talent from a more diverse pool� reputational terms both in the city and nationally � delivered by volunteer students, this fits very well into the

developing ethos of the ‘Bristol Futures’ programme.

UoB WP Workshop, Oct. 2016 26

Discussant

Rachael Tooth,

Head of Strategy and Change,

Office for Fair Access

Lunch