PERSPECTIVES ON QUALITY ANDEQUITY FROM LARGE-SCALE

ASSESSMENT STUDIES

John Ainleyand

Eveline Gebhardt

Outline• Properties of large-scale assessments• Perspectives from NAPLAN: 2008 to 2013

– National changes in reading achievement– Improvements in reading for Indigenous students– Differences in changes across jurisdictions

• Messages from PISA– Changes in reading and mathematics

achievement– Student background – Jurisdictional changes in reading and

mathematics– Variation between and within schools

• Summary

Perspectives from NAPLAN

National mean reading scores: 2008-2013

2008 2009 2010 2011 2012 2013 Difference 2008–13

Year 3 401 411 414 416 420 419 19

Year 5 484 494 487 488 494 502 18

Year 7 537 541 546 540 542 541 4

Year 9 578 581 574 580 575 580 2

National mean numeracy scores: 2008-2013

2008 2009 2010 2011 2012 2013 Difference 2008–13

Year 3 397 394 395 398 396 397 0

Year 5 476 487 489 489 489 486 10

Year 7 545 544 548 545 538 542 -3

Year 9 582 589 585 583 584 584 2

The story so far:

• Improvement in reading in years 3 and 5• No change in reading in years 7 and 9• No significant national change in numeracy

at Years 3, 5 (note exception of Queensland), 7 or 9

Changes in mean reading scores:Indigenous and non-Indigenous students

2008 2009 2010 2011 2012 2013 Difference 2008–13

Year 3

Indigenous 314 327 331 332 333 344 30

Non-Indigenous 405 415 419 420 424 423 18

Year 5

Indigenous 403 414 410 410 409 439 36

Non-Indigenous 489 498 491 492 498 506 17

Jurisdictional mean Year 3 reading scores: 2008-2013

2008 2009 2010 2011 2012 2013 Difference 2008–13

QLD 371 386 393 400 409 408 37

NT 307 322 329 323 332 339 33

ACT 421 434 439 443 444 442 21

WA 387 396 399 400 408 406 19

Jurisdictional mean Year 5 reading scores: 2008-2013

2008 2009 2010 2011 2012 2013 Difference 2008–13

NT 405 421 412 403 405 437 32

QLD 466 478 469 469 480 497 31

WA 474 482 478 480 483 495 22

ACT 503 513 509 516 519 519 16

SA 478 484 477 478 484 492 14

VIC 497 506 502 504 504 510 13

The story extended:

• Improvement in reading in years 3 and 5• No change in reading in years 7 and 9• No significant national change in numeracy

at Years 3, 5, 7 or 9• Larger improvement in reading for

Indigenous than non-Indigenous students (Years 3 & 5)

• Differences among jurisdictions with large increases in:– Queensland– Northern Territory

Interpretation

• Emphasis on the early years (school and pre-school)– Programs and resources– Focus on reading in the early school and

pre-school years• Programs intended to improve outcomes for

Indigenous students• Structural change in Queensland

– Introduction of a foundation/preparatory year

– Accompanied by a strong school improvement strategy

Messages from PISA

Change in reading literacy:2000 - 2012

2000 2003 2006 2009 2012Change

2000-2012

Poland 479 497 508 500 518 39

Germany 484 491 495 497 508 24

Portugal 470 478 472 489 488 18

Australia 528 525 513 515 512 -16

Finland 546 543 547 536 524 -22

Sweden 516 514 507 497 483 -33

Change in mathematical literacy:2003 - 2012

2003 2006 2009 2012Change

2003-2012

Poland 490 495 495 518 27

Portugal 466 466 487 487 21

Germany 503 504 513 514 11

Australia 524 520 514 504 -20

Finland 544 548 541 519 -26

Sweden 509 502 494 478 -31

Changes in achievement and student background

Reading 2000-2009 Mathematics 2003-2012

2000 2009 Gender

Females 546 533 Males 513 496 Difference 34 37 Indigenous status

Non-Indigenous 531 518 Indigenous 448 436 Difference 83 82 Language background English at home 535 518 LBOTE 504 509 Difference 31 10 Location

Metropolitan 535 521 Non-metro 518 496 Difference 17 25

2003 2012 Gender

Females 494 491 Males 505 502 Difference -11 -11 Indigenous status

Non-Indigenous 526 507 Indigenous 440 417 Difference 86 90 Language background

English at home 529 506 LBOTE 505 509 Difference 24 -3 Location

Metropolitan 528 511 Provincial 515 486 Remote 493 444 Diff (met-prov) 13 25

Changes in achievement & socioeconomic background (ESCS): overall gradient

Reading 2000-2009Mathematics 2003-2012

2003 2012

Australia 42 42

Canada 34 31

USA 45 35

UK na 41

Finland 33 33

Germany

47 43

2000 2009

Australia

47 46

Canada 38 32

USA 52 42

UK na 44

Finland 25 31

Germany

52 44

The story so far

• Modest declines in reading and mathematics among 15 year-olds

• Extent of decline is similar for reading and mathematics

• Does not appear to be associated with differences in student background.

Jurisdictional change in reading literacy:2000 -2012

PISA 2000

PISA 2009

PISA 2012

Difference 2000–2012

Victoria 516 513 517 1

Queensland 521 519 508 -13

Western Australia 538 522 519 -19

Northern Territory 489 481 466 -23

New South Wales 539 516 513 -26

ACT 552 531 525 -27

Tasmania 514 483 485 -30

South Australia 537 506 500 -37

Jurisdictional change in mathematical literacy: 2003 -2012

PISA 2003 PISA 2012 Difference 2003-2012

Victoria 511 501 -10

Queensland 520 503 -16

New South Wales 526 509 -17

Tasmania 507 478 -30

ACT 548 518 -30

Western Australia 535 516 -32

Northern Territory 496 452 -45

South Australia 548 489 -46

Summary of jurisdictional changes in PISA reading and mathematicsReading change(00-12)

Jurisdiction Change

VIC 1

QLD -13

WA -19

NT -23

NSW -26

ACT -27

TAS -30

SA -37

Mathematics change (03-12)

Jurisdiction Change

VIC -10

QLD -16

NSW -17

TAS -30

ACT -30

WA -32

NT -45

SA -46

The story extended

• Modest declines in reading and mathematics among 15 year-olds

• Extent of decline is similar for reading and mathematics

• Does not appear to be associated with differences in student background

• Differences among jurisdictions in the extent of decline

• Changes in reading and mathematics achievement appear to run parallel

Variation between and within schools• Variance

– dispersion calculated as the mean of the squared deviations of observed values from a mean

– total variance made up of: • variance within schools and • variance between schools’ mean scores

• Percentages of total variance• Within / total• Between / total

• % variance between schools is an indicator of the extent to which schools differ

Between-school variance in reading and mathematics in selected countries

% variance between schools Reading Mathematics PISA 2000 PISA 2009 PISA 2003 PISA 2012 Finland 8% 9% 5% 8% Sweden 9% 18% 9% 13% Canada 20% 22% 17% 20% Australia 20% 26% 22% 28% United States 30% 36% 24% 24% Germany 59% 60% 57% 53% OECD average 36% 37% 33% 35%

Change in mean mathematics score (2003-12) and change in % between-school variance

The story extended a little further• Modest declines in reading and mathematics among

15 year-olds• Extent of decline is similar for reading and

mathematics• Does not appear to be associated with differences in

student background• Differences among jurisdictions in the extent of

decline• Changes in reading and mathematics achievement

appear to run parallel• Declines in Australia associated with larger differences

among schools• In general for mathematics larger changes in between

school variance are associated with larger declines

Speculation

• Differentiation appears to be associated with lower overall achievement (Willms, 2010)

• Differentiation among schools may be associated with differentiation in teaching expertise among schools

• Possibly a reciprocal relationship arising from attractiveness of schools to teachers

• Caution about policies that exacerbate differences.

Thank you

QuestionsDiscussion