The Three Rs: The Scope for Literacy and Numeracy Policies to Raise Achievement

Stephen Machin* and Sandra McNally**

* University College London; CEE and CEP, London School of Economics

** CEE and CEP, London School of Economics

Background• Long lower tail of education and skills

distribution in some countries: high percentage of population with low skills in literacy and numeracy.

• This is bad for the people concerned, for their children and communities and for national productivity.

• The higher frequency of adults with poor basic skills has been linked to perceived failures in the education systems of these countries – especially poor teaching.

Issues and Questions• How can we ensure that the next generation

of adults does not suffer from the same problems of poor and inadequate basic skills?

• Are there cost effective ways in which teaching of literacy and numeracy can be altered to raise achievement levels?

• Is it better (for children’s schooling outcomes) to tell teachers what to do by means of structured literacy and numeracy programmes?

Structure of Talk• Description of extent to which better literacy

and numeracy skills matter.• Literacy and numeracy strategies.• National trends.• Empirical approach.• Results• Conclusions and implications (and a bit on

what more we will do in future).

Adult Basic Skills and Wages - IALS

Earnings Gains For One SD Increase in Basic Skills [Average of Prose, Document and Quantitative]

[From Denny, Harmon and O’ Sullivan (2004)]

Germany .053 New Zealand .118

Norway .067 Great Britain .137

Denmark .074 Northern Ireland .145

Sweden .081 Ireland .168

Italy .108 United States .176

Wages and Childhood Cognitive Tests- BCS70

Earnings Gains For One SD Increase in Cognitive Skills [From Blanden, Gregg and McMillan (2007)]

EPVT Age 5 .024 (.010)

Copy Age 5 .054 (.010)

Reading Age 10 .035 (.013)

Maths Age 10 .081 (.014)

British Ability Scale Age 10 .021 (.012)

GCSE Performance and Key Stage 2 English and Maths - NPD

Probability of Getting 5 or More A*-C GCSEs If Achieve Level 4 or Higher KS2 English and Maths

[All our own work, 2001-2006 data]

Level 4 or Higher English Level 4 or Higher Maths

.472 (.001) .451 (.001)Includes: Year dummies, primary school controls and fixed effects,

secondary school controls and fixed effects

Includes: Year dummies, primary school controls and fixed effects,

secondary school controls and fixed effects

N = 3,064,428 N = 3,083,336

Intergenerational Correlations in Cognitive Skills – Children of BCS70 and BCS70

Increase in Percentiles of Child Cognitive Skills For One SD Increase in Parent Cognitive Skills

[Blanden (This Morning)]

Reading Maths

9.72 (1.36) 7.81 (1.44)

N = 546 N = 545

Our Earlier Work – Literacy Hour• Highly structured hour of literacy teaching introduced to some

schools through National Literacy Project in in 400 schools in England in September 1996 and September 1997.

• Constituted a discrete, well-defined and systematic change in how literacy is taught within primary schools (and where teaching standards were perceived to be low before).

• Daily literacy hour: 10-15 minutes whole class reading or writing; 10-15 minute session on word level work (e.g. phonics, spelling); 25-30 minutes directed group activities; plenary session at the end.

Our Earlier Work – Literacy Hour• Gives setting where some children exposed to up to two years of the

literacy hour, when other children were not.

• Used a difference-in-difference framework and statistical matching methods to compare outcomes in treatment and control group before and after policy was introduced (and taking care of pre-policy trends).

• Found significant improvement in KS2 English (of the order of 2.7-3.5 percentage points in the probability of achieving Level 4), that there was no evidence of this being explained by pre-policy trends/mean reversion and that the NLP was extremely cost effective.

• [You can do the same exercise for the national numeracy project – you get 3.3-4.1 percentage points on Probability of getting ≥Maths KS2 Level 4]

Literacy and Numeracy Strategies

• ‘Centrally planned’ teaching approach for literacy and numeracy implemented in all primary schools in England (but not other parts of UK).

• Essential components: 1 hour of literacy and numeracy per day; prescribed method and content.

• ‘Projects’ had been introduced by previous (Conservative) government in 1996/97. National roll-out for ‘literacy hour’ was in September 1998; ‘numeracy hour’ in September 1999 (but announced at same time).

Research Questions• Does being exposed to the ‘literacy hour’ or the

‘numeracy hour’ matter for achievement?• Does length of exposure matter?• How does the national strategy compare to the more

targetted ‘pilots’?• How did national performance in England with the

strategies compare to countries without such strategies?

• Are there longer lasting effects beyond primary school?

• Are there any broader effects of the strategies? (e.g. on behaviour; attitudes to school; other subjects?)

Trends in the Percentage of Pupils Achieving the Required (National Curriculum) Standard in Key Stage 2 Tests at the End of

Primary School

Introduction of Literacy and Numeracy Hours

• The ‘literacy hour’ and ‘numeracy hour’ were both initially introduced into some schools in a small number of Local Education Authorities (12/150 for literacy; 13 for numeracy); 3 LEAs exposed to both for overlapping time period (but not schools in those LEAs)

• Within the ‘pilot’ LEAs, schools were introduced in different waves:

- Literacy hour: wave 1 started in 1996/97; wave 2: 1997/98. National Roll-out: Sept. 1998.

- Numeracy hour: wave 1 started in 1996/97; wave 2: 1997/98; wave 3: 1998/99. National roll-out: Sept. 1999.

Methodology (2)• This structure means that a certain number of

schools were exposed to the ‘literacy hour’ or ‘numeracy hour’ at a time when other schools were not and we can make before and after comparisons (so we can do difference-in-differences).

• It also means that there is variation in the length of exposure to these teaching strategies faced by different cohorts of pupils.

• In Wales, there was no ‘national strategy’. Every Welsh local authority drew up a local strategy and scope for more discretion by schools. We can use Wales as a comparison group.

Methodology (3)1) Effects of any exposure (2 years pre-policy: 1995/96; 1996/97; 2

years post-policy (before national roll-out) for literacy hour; 3 years for numeracy hour

Ast = α + βExps*Policy Ont + γExps + δXst + λTt +ηs + εist

Ast: Average Attainment in School s in year t. Exp=1 if school is exposed to ‘literacy hour’ or ‘numeracy hour'; Exp=0 otherwise. X: vector of school characteristics T: set of year dummies η: school fixed effects

Methodology (4)2) Effects of length of exposure

(Ast - Ast-1) = β(Expyrst – Expyrst-1) + δ(Xst - Xst-1) + λTt + µst Ast: Average Attainment in School s in year t. Expyr=years of exposure to literacy or numeracy hour. In some specifications, control for school-specific time trends

3) Differential effects according to whether school was in ‘pilot’

(Ast - Ast-1) = β1Pilot*(Expyrst – Expyrst-1) + β2non-Pilot*(Expyrst – Expyrst-1) + δ(Xst - Xst-1) + λTt + µst

Data

• National Pupil Database: contains census of all pupils in English schools. Test results and school-identifiers from 1995/96 onwards (for age 11 pupils). More personal characteristics from 2002 onwards. Importantly, this contains the pupil’s test result in English and Maths at age 11 (national tests that are externally set and marked – undertaken by all pupils; used to compile ‘performance tables’ for primary schools).

• School-level data: LEA and School Information System (e.g. number of pupils; school type; % eligible for free school meals; % non-white etc.) available for many years. School-level performance data available from 1994/95 onwards.

Data (2)

• Welsh school data available from 1997/98 and pupil-level from 1996/97. Same tests at age 11 up to about 2000 after which there are some differences in the papers actually given to pupils between England and Wales.

• Longitudinal Survey of Young People: data on about 14,000 15-year olds in English schools from 2001. Very rich data and we can link the data with whether the student went to a primary school that was in the ‘literacy’ or ‘numeracy’ projects.

 Literacy Hour – effects of years of exposurePercent Achieving Level 4 Key Stage 2 English

(13890 Primary Schools, 1995-2002, Sample Size 95412) (1) (2) (3) (4) (5) (6)

Years of Exposure 1.440 (0.450)

1.600 (0.458)

1.611 (0.500)

One Year of Exposure 1.713 (0.758)

1.777 (0.769)

1.783 (0.836)

Two Years of Exposure

2.196 (1.034)

2.324 (1.057)

2.332 (1.157)

Three Years of Exposure

3.068 (1.328)

3.476 (1.354)

3.486 (1.486)

Four Years of Exposure

4.988 (1.668)

5.651 (1.696)

5.681 (1.858)

School Fixed Effects Yes Yes Yes Yes Yes Yes Year Dummies Yes Yes Yes Yes Yes Yes Control Variables No Yes Yes No Yes Yes School Specific Trends (P-Value)

No No Yes (P = 0.16)

No No Yes (P = 0.16)

P-Value, Test of Equality Across Years of Exposure

P = 0.02 P = 0.00 P = 0.01

 Numeracy Hour – effects of length of exposurePercent Achieving Level 4 Key Stage 2 Mathematics

(13933 Primary Schools, 1995-2003, Sample Size 123144)

(1) (2) (3) (4) (5) (6) Years of Exposure 1.215

(0.277) 1.172

(0.271) 1.158

(0.295)

One Year of Exposure 1.537 (0.494)

1.527 (0.499)

1.521 (0.540)

Two Years of Exposure

2.982 (0.655)

2.901 (0.649)

2.867 (0.703)

Three Years of Exposure

3.772 (0.841)

3.693 (0.828)

3.643 (0.900)

Four Years of Exposure

4.835 (1.095)

4.661 (1.074)

4.604 (1.166)

School Fixed Effects Yes Yes Yes Yes Yes Yes Year Dummies Yes Yes Yes Yes Yes Yes Control Variables No Yes Yes No Yes Yes School Specific Trends (P-Value)

No No Yes (P = 0.26)

No No Yes (P = 0.26)

P-Value, Test of Equality Across Years

P = 0.01 P = 0.00 P = 0.01

Effects of the Pilots Versus Strategies

(1) Literacy Hour

Dependent variable: English

(2) Numeracy Hour

Dependent variable: Maths Years of Exposure * Pilot Years of Exposure * Strategy School Fixed Effects Year Dummies Control Variables P-Value, Test of Equality, NNP versus NNS Number of Schools Number of observations

2.135 (.469) 1.065 (.475)

Yes Yes Yes

P = 0.00

13890 95412

1.687 (.297) .626 (.281)

Yes Yes Yes

P = 0.00

13933 123144

England Versus Wales: 1997-2000, Regression Models

Literacy Hour Dependent variable: English

Numeracy Hour Dependent variable: Maths

(1) (2) Years of Exposure .762 (.228) 1.396 (0.314) School Fixed Effects Yes Yes Year Dummies Yes Yes Control Variables Yes Yes Number of Schools Number of observations

15692 76164

15662 76072

National data: Proportion achieving expected standard in English

National data: Proportion achieving expected standard in Maths

Difference-in-Difference Estimates By Year (England Versus Wales)

Literacy Hour Dependent variable: English

Numeracy Hour Dependent variable: Maths

(1) (2) England*1998 England*1999 England*2000

0.635 (0.539)

2.524 (0.554)

2.184 (0.590)

-0.465 (0.564)

4.010 (0.577)

4.560 (0.618)

School Fixed Effects Yes Yes Year Dummies Yes Yes Control Variables Yes Yes Number of Observations 61210

61159

Longer Term Achievement – GCSE Performance at End of Secondary School 1

Literacy Hour Dependent variable: English

Numeracy Hour Dependent variable: Maths

(1) (2) (3) (4) England,

2001-2006

England and Wales, 2002-

2006

England, 2001-2006

England and Wales, 2002-

2006

Years of Exposure 1.608 (0.304) 0.195 (0.104) 1.137 (0.160)

0.713 (0.102)

Primary School Fixed Effects Secondary School Fixed Effects

Yes Yes

Yes, Yes

Yes, Yes

Yes, Yes

Year Dummies Yes Yes Yes Yes Control Variables (Primary and Secondary School)

Yes Yes Yes Yes

Number of Primary Schools Number of Observations

13937 3,089,747

15388 2,766,453

13941 3,658,879

13941 3,658,879

Longer Term Achievement – GCSE Performance at End of Secondary School 2

Literacy Hour Dependent variable: English

Numeracy Hour Dependent variable: Maths

(1) (2) (3) (4) England,

2001-2006

England and Wales, 2002-

2006

England, 2001-2006

England and Wales, 2002-

2006

One Year of Exposure Two Years of Exposure Three Years of Exposure Four Years of Exposure

1.158 (0.552) 2.862 (0.682) 4.135 (1.052) 6.869 (1.205)

-1.150 (0.280) 0.124 (0.307) 0.386 (0.345) 1.600 (0.729)

1.312 (0.361) 1.991 (0.388) 4.106 (0.534) 3.890 (0.923)

0.788 (0.235) 1.191 (0.317) 3.144 (0.446) 2.690 (0.601)

Primary School Fixed Effects Secondary School Fixed Effects

Yes Yes

Yes, Yes

Yes, Yes

Yes, Yes

Year Dummies Yes Yes Yes Yes Control Variables (Primary and Secondary School)

Yes Yes Yes Yes

Number of Primary Schools Number of Observations

13937 3,089,747

15388 2,766,453

13941 3,658,879

15360 2,765,322

Longer Term Achievement – GCSE Performance at End of Secondary School 3

• But most of these secondary school effects explained by primary school gains.

• Include KS2 measures and effects mostly go away (small literacy hour effect remains in England comparison; small numeracy hour effect remains in England and Wales comparison).

• The literacy and numeracy hours boosted primary school performance.

LSYPE Analysis of Exposure to Pilots

• Initial analysis of first wave of the Longitudinal Survey of Young People in England with respect to the ‘literacy hour’ finds no evidence for any effect of the literacy hour on behavioural outcomes (e.g. truancy, anti-social behaviour of various types) or attitudes to school work or school in general.

Summary• Both the the literacy and numeracy hours were effective for raising

primary school achievement.

• More years of exposure seems to have generated bigger achievement gains.

• The effect of both policies in ‘pilot’ areas was stronger than when policies were rolled out nationally, but the national strategies show better outcomes than Wales which has no strategy.

• Those who benefited from the literacy and numeracy hours do better in secondary school BUT this is mostly explained by the primary school gains.

Implications• There has been concern about the ‘plateauing

out’ of KS2 performance.

• Our results show the national literacy and numeracy strategies did enhance achievement and importantly that KS2 performance improved more in England under the strategies regime than in Wales which has no national strategy.

• Training teachers to deliver a prescribed form of structured teaching seems to have been beneficial.

Future Work• Separating out time effects and exposure effects

in more detail – tests versus ‘saturated’ model.

• Explore heterogeneity of teaching methods in Wales in more detail.

• Impacts on other subjects and complementarities between English and Maths (but not vice versa).

Year-by-year effects of numeracy hour (where significantly different…previous table)

Numeracy Hour Years of Exposure * non-Pilot 0.218

(0.350)

One Year of Exposure * Pilot 2.471 (0.676)

Two Years of Exposure * Pilot 4.372 (0.730)

Three Years of Exposure * Pilot 5.527 (0.904)

Four Years of Exposure * Pilot 5.882 (1.291)

School Fixed Effects Yes Year Dummies Yes Control Variables Yes Number of Schools Number of observations

13933 123144