OECD EMPLOYER

BRAND

Playbook

1

PISA 2012Evaluating school systems

to improve education

Pablo Zoido

2 PISA in brief

• Over half a million students…– representing 28 million 15-year-olds in 65 countries/economies

… took an internationally agreed 2-hour test…– Goes beyond testing whether students can

reproduce what they were taught…

… to assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations

– Mathematics, reading, science, problem-solving, financial literacy

– Total of 390 minutes of assessment material

… and responded to questions on…– their personal background, their schools

and their engagement with learning and school

• Parents, principals and system leaders provided data on…– school policies, practices, resources and institutional factors that

help explain performance differences .

3 PISA in brief

• A shared learning tool for all involved– ‘Crowd sourcing’ and collaboration

• PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies…

… guided by governments on the basis of shared policy interests

– Cross-national relevance and transferability of policy experiences

• Emphasis on validity across cultures, languages and systems

• Frameworks built on well-structured conceptual understandingof academic disciplines and contextual factors

– Triangulation across different stakeholder perspectives

• Systematic integration of insights from students, parents, school principals and system-leaders

– Advanced methods with different grain sizes

• A range of methods to adequately measure constructs with different grain sizes to serve different decision-making needs

• Productive feedback, at appropriate levels of detail, to fuel improvement at every level of the system .

4

Climbing Mount Fuji

Mount Fuji is a famous dormant volcano

in Japan.

Mount Fuji is only open to the public for

climbing from 1 July to 27 August each

year. About 200 000 people climb

Mount Fuji during this time.

On average, about how many people

climb Mount Fuji each day?

A. 340

B. 710

C. 3400

D. 7100

E. 7400

PISA 2012 Sample Question 1

5

Percent of 15-year-olds who scored Level 2 or AboveS

hang

hai-C

hina

Sin

gapo

reH

ong

Kon

g-C

hina

Kor

eaE

ston

iaM

acao

-Chi

naJa

pan

Fin

land

Sw

itzer

land

Chi

nese

Tai

pei

Can

ada

Liec

hten

stei

nV

ietn

amP

olan

dN

ethe

rland

sD

enm

ark

Irel

and

Ger

man

yA

ustr

iaB

elgi

umA

ustr

alia

Latv

iaS

love

nia

Cze

ch R

epub

licIc

elan

dU

nite

d K

ingd

omN

orw

ayF

ranc

eN

ew Z

eala

ndO

EC

D a

vera

ge

Spa

inR

ussi

an F

eder

atio

nLu

xem

bour

gIta

lyP

ortu

gal

Uni

ted

Sta

tes

Lith

uani

aS

wed

enS

lova

k R

epub

licH

unga

ryC

roat

iaIs

rael

Gre

ece

Ser

bia

Rom

ania

Tur

key

Cyp

rus*

Bul

garia

Kaz

akhs

tan

Uni

ted

Ara

b E

mira

tes

Tha

iland

Chi

leM

alay

sia

Mex

ico

Uru

guay

Mon

tene

gro

Cos

ta R

ica

Alb

ania

Arg

entin

aB

razi

lT

unis

iaJo

rdan

Qat

arC

olom

bia

Per

uIn

done

sia

0

10

20

30

40

50

60

70

80

90

100

PISA 2012 Sample Question 1

6

Revolving DoorA revolving door includes three wings which rotate within a circular-shaped space. The inside diameter of

this space is 2 metres (200 centimetres). The three door wings divide the space into three equal sectors.

The plan below shows the door wings in three different positions viewed from the top.

The two door openings (the dotted arcs in the diagram) are the same size.

If these openings are too wide the revolving wings cannot provide a sealed

space and air could then flow freely between the entrance and the exit,

causing unwanted heat loss or gain. This is shown in the diagram opposite.

What is the maximum arc length in centimetres (cm) that each door

opening can have, so that air never flows freely between the entrance and

the exit?

Maximum arc length: ____________ cm

PISA 2012 Sample Question 4

7

Percent of 15-year-olds who scored Level 6 or AboveS

hang

hai-C

hina

Sin

gapo

re

Chi

nese

Tai

pei

Hon

g K

ong-

Chi

na

Kor

ea

Japa

n

Mac

ao-C

hina

Liec

hten

stei

n

Sw

itzer

land

Bel

gium

Pol

and

Ger

man

y

New

Zea

land

Net

herla

nds

Can

ada

Aus

tral

ia

Est

onia

Fin

land

Vie

tnam

Slo

veni

a

OE

CD

ave

rag

e

Aus

tria

Cze

ch R

epub

lic

Fra

nce

Slo

vak

Rep

ublic

Uni

ted

Kin

gdom

Luxe

mbo

urg

Icel

and

Uni

ted

Sta

tes

Isra

el

Irel

and

Italy

Hun

gary

Por

tuga

l

Nor

way

Den

mar

k

Cro

atia

Sw

eden

Latv

ia

Rus

sian

Fed

erat

ion

Lith

uani

a

Spa

in

Tur

key

Ser

bia

Bul

garia

Gre

ece

Rom

ania

Uni

ted

Ara

b E

mira

tes

Tha

iland

0

5

10

15

20

25

30

PISA 2012 Sample Question 4

Singapore

Hong Kong-ChinaChinese Taipei

Korea

Macao-ChinaJapan LiechtensteinSwitzerland

NetherlandsEstonia FinlandCanada

PolandBelgiumGermany Viet Nam

Austria AustraliaIrelandSlovenia

DenmarkNew ZealandCzech Republic France

United KingdomIceland

LatviaLuxembourg NorwayPortugal ItalySpain

Russian Fed.Slovak Republic United StatesLithuaniaSwedenHungary

CroatiaIsrael

GreeceSerbiaTurkey

Romania

BulgariaU.A.E.KazakhstanThailand

ChileMalaysia

Mexico410

420

430

440

450

460

470

480

490

500

510

520

530

540

550

560

570

580

Mean score

High mathematics performance

Low mathematics performance

… Shanghai-China performs above this line (613)

… 12 countries perform below this line

Average performance

of 15-year-olds in

MathematicsFig I.2.13

US

Change in performance between PISA 2003 and 2012

Indonesia

Thailand

Russian Fed.

United States

Latvia

Spain

NorwayLuxembourg

Ireland

Austria

SwitzerlandJapan

Liechtenstein

Korea

Brazil

Tunisia

Mexico

Uruguay

Turkey

Greece

Italy

Portugal

Hungary

Poland

Slovak Republic

OECD average

Germany

Sweden

France

Denmark

Iceland

Czech Republic

New ZealandAustralia

Macao-China

Belgium

Canada

Netherlands

Finland

Hong Kong-China

-4

-3

-2

-1

0

1

2

3

4

5

350 400 450 500 550 600

Ave

rag

e a

nn

ua

l m

ath

em

ati

cs

sc

ore

ch

an

ge

Average mathematics performance in PISA 2003

Imp

rovin

g p

erfo

rma

nc

eD

ete

riora

ting

pe

rform

an

ce

PISA 2003 performance below the OECD averagePISA 2003 performance

above the OECD average

Fig I.2.189

Mathematics, reading and science Israel, Poland, Portugal, Turkey, Brazil,

Dubai (UAE), Hong Kong-China,

Macao-China, Qatar, Singapore, Tunisia

Mathematics and readingChile, Germany, Mexico, Albania, Montenegro,

Serbia, Shanghai-China

Mathematics and scienceItaly, Kazakhstan, Romania

Reading and scienceJapan, Korea, Latvia, Thailand

Mathematics onlyGreece, Bulgaria, Malaysia,

United Arab Emirates (ex. Dubai)

Reading only Estonia, Hungary, Luxembourg, Switzerland,

Colombia, Indonesia, Liechtenstein, Peru,

Russian Federation, Chinese Taipei

Science onlyIreland

Of the 65 countries 45 improved at least in one subject10

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Ja

pa

n

Ho

ng

Kon

g-C

hin

a

Luxe

mb

ou

rg

Norw

ay

Czech R

ep

ub

lic

Icela

nd

Kore

a

Ind

one

sia

Th

aila

nd

Me

xic

o

De

nm

ark

Lie

ch

tenste

in

Italy

Austr

ia

Ma

ca

o-C

hin

a

Tu

rkey

Belg

ium

Cana

da

Port

ug

al

Pola

nd

Spa

in

OE

CD

ave

rag

e 2

00

3

Sw

itze

rla

nd

Bra

zil

Un

ited

Sta

tes

Gre

ece

Slo

vak R

epu

blic

Neth

erla

nds

Ru

ssia

n F

ede

ratio

n

Hung

ary

Ire

land

New

Ze

ala

nd

Austr

alia

Uru

gua

y

Sw

ede

n

Latv

ia

Fra

nce

Fin

land

Germ

any

Tu

nis

ia

Me

an

in

de

x c

ha

ng

e

Change between 2003 and 2012 in disciplinary climate in schools

In most countries and economies, the disciplinary

climate in schools improved between 2003 and 2012

Disciplinary climate

declined

Disciplinary climate

improved

Fig IV.5.13

Are 15-year-olds learning for school or for life?

Problem Solving PISA 2012

12

13

TRAFFIC

Problem Solving – Sample Question 1

Julio lives in Silver, Maria lives in Lincoln, and Don lives in Nobel.

They want to meet in a suburb on the map. No-one wants to travel

for more than 15 minutes.

Where could they meet?

This is an easy item – Level 1 on the problem-solving scale (below baseline)

All information required is given at the outset: it is a static problem

This item focuses on students’ ability to monitor and reflect on solutions.

14

TICKETS

You plan to take four trips

around the city on the

subway today. You are a

student, so you can use

concession fares.

Use the ticketing machine

to find the cheapest ticket

and press BUY.

Once you have pressed

BUY, you cannot return to

the question;

Problem Solving – Sample Question 2

This is a harder item – Level 5 on the problem-solving scale

Students must engage with the machine, and use the feedback and information uncovered to reach a

solution: it is an interactive problem

This main demand is exploring and understanding (knowledge acquisition)

Sample items can be tried at cbasq.acer.edu.au and www.oecd.org/pisa/test

Au

str

alia

Bra

zil

Ma

ca

o-C

hin

a

En

gla

nd

(U

.K.)

Ita

ly

Unite

d S

tate

s

Se

rbia

Ja

pa

n

Ko

rea

Au

str

ia

Slo

va

k R

ep

ub

lic

Russia

n F

ed

era

tio

n

Po

rtu

ga

l

Sw

ed

en

Can

ad

a

Cze

ch

Re

pu

blic

Chile

Norw

ay

Sin

ga

po

re

Fra

nce

Bu

lga

ria

Sh

an

gh

ai-

Chin

a Po

lan

d

Unite

d A

rab

Em

ira

tes

Hun

ga

ry

Slo

ve

nia Is

rae

l

Uru

gu

ay

Mo

nte

ne

gro

Cro

atia

Sp

ain

Ire

lan

d

Hon

g K

on

g-C

hin

a

Neth

erla

nd

s

Esto

nia

Tu

rkey

Ma

laysia

Ge

rma

ny

Den

ma

rk

Be

lgiu

m

Chin

ese

Ta

ipe

i

Fin

lan

d

OE

CD

ave

rage

Colo

mb

ia

-60

-40

-20

0

20

40

%

Relative performance in problem solving Fig V.2.15

Students' performance in problem solving

is lower than their expected performance

Students' performance in problem solving

is higher than their expected performance

15

200

300

400

500

600

700

800

200 300 400 500 600 700 800

Patterns of relative performance in problem solving

Problem solving performance

Mathematics performance

Fig V.2.16

Fig V.2.17

Average relationship between problem solving

and mathematics performance

The United States and England (UK) perform better-than-expected in problem solving. The difference between observed and expected performance is larger among strong performers

in mathematics

Japan performs better-than-expected in problem solving. The difference between observed and expected performance is larger among low

achievers in mathematicsPoland’s performance is lower-than-expected

in problem solving. The gap between observed and expected performance is similar at all

levels of mathematics performance.

16

Spain’s performance is lower-than-expected in problem solving. The gap

between observed and expected performance is wider among low

achievers in mathematics.

Singapore’s performance in problem solving is as high as

expected at all levels of mathematics performance

Strengths and weaknesses Fig V.3.10

United States

Poland

England

Estonia

Finland

Slovak Rep.

Germany

Austria

Czech Rep.

France

Japan

Turkey

Sweden

Hungary

Australia

Israel

Canada

Ireland

Chile

Belgium

Netherlands

Spain

Denmark

Slovenia

Portugal

Norway

Korea

Italy

Hong Kong-China

Brazil

Uruguay

CroatiaChinese Taipei

Bulgaria

Macao-China

U.A.E.

Montenegro

Singapore

Colombia

Malaysia

Serbia

Russian Fed.

Shanghai-China

OECD average

OECD

avera

ge

Better performance on interactive tasks

Better performance on static tasks

Better performance on knowledge-utilisation tasks

Better performance on knowledge-acquisition tasks

Stronger-than-expected performance on interactive

items, weaker-than-expected performance on

knowledge-acquisition tasks

Weaker-than-expected performance on interactive items , stronger-than-expected performance on

knowledge-acquisition tasks

17

Resources make a difference…

…but only up to a point

18

Spending per student from the age of 6 to 15 and

mathematics performance in PISA 2012

Slovak Republic

Czech RepublicEstonia

Israel

Poland

Korea

Portugal

New Zealand

CanadaGermany

Spain

France

Italy

Singapore

Finland

Japan

Slovenia Ireland

Iceland

Netherlands

Sweden

Belgium

UK

AustraliaDenmark

United States

Austria

Norway

Switzerland

Luxembourg

Viet Nam

Jordan

Peru

Thailand

Malaysia

Uruguay

Turkey

Colombia

Tunisia

MexicoMontenegro

Brazil

Bulgaria

Chile

CroatiaLithuania

Latvia

Hungary

Shanghai-China

R² = 0.01

R² = 0.37

300

350

400

450

500

550

600

650

0 20 000 40 000 60 000 80 000 100 000 120 000 140 000 160 000 180 000 200 000

Ma

the

ma

tic

s p

erf

orm

an

ce

(sc

ore

po

ints

)

Average spending per student from the age of 6 to 15 (USD, PPPs)

Cumulative expenditure per student less than USD 50 000

Cumulative expenditure per student USD 50 000 or more

Fig IV.1.8

-2.00

-1.50

-1.00

-0.50

0.00

0.50

Peru

Costa

Ric

aM

exic

oB

razil

Ind

one

sia

Th

aila

nd

Colo

mb

iaN

ew

Ze

ala

nd

Tu

rke

yA

rgen

tin

aU

nited

Sta

tes

Uru

gua

yA

ustr

alia

Ch

ileV

iet N

am

Jo

rdan

Sha

ngh

ai-

Ch

ina

U.A

.E.

Ro

ma

nia

Sw

ede

nIs

rael

Bulg

aria

Ch

inese

Taip

ei

Ma

laysia

Ire

land

Gre

ece

Tu

nis

iaP

ola

nd

Ca

na

da

Ja

pa

nM

aca

o-C

hin

aO

EC

D a

ve

rag

eL

uxe

mb

ou

rgQ

ata

rR

ussia

n F

ed.

Icela

nd

Belg

ium

Fra

nce

Sw

itze

rla

nd

Port

ug

al

Ho

ng

Kon

g-C

hin

aS

pa

inL

ithu

ania

De

nm

ark

Kaza

kh

sta

nIt

aly

Cze

ch R

ep

ub

licN

eth

erla

nds

Esto

nia

Hung

ary

Slo

ven

iaA

ustr

iaS

inga

po

reL

atv

iaS

lovak R

epu

blic

Mo

nte

neg

roK

ore

aG

erm

any

Serb

iaU

nited

Kin

gd

om

Norw

ay

Cro

atia

Fin

land

Lie

ch

tenste

inA

lban

ia

Me

an

in

de

x d

iffe

ren

ce

Difference between socio-economically disadvantaged and socio-economically advantaged schools

Difference between public and private advantaged schools

Educational resources are more problematic in disadvantaged

schools, also in public schools in most countries

Advantaged and private schools

reported better educational

resources

Disadvantaged and public schools

reported better educational

resources

Fig IV.3.8

Qatar

Greece

Israel

HungarySweden

USASlovak Rep.

SpainItaly PortugalNorway Luxembourg

IcelandUK

FranceCzech Rep.

New ZealandDenmark

SloveniaIreland

AustraliaAustria GermanyBelgium

PolandCanadaFinland

Estonia Netherlands

JapanMacao-China

KoreaHong Kong-China

Singapore

Indonesia

Jordan

Peru

Tunisia

Colombia

Thailand

Montenegro

Uruguay

Bulgaria

Romania

Malaysia

Argentina

Latvia

Chile

Lithuania

Shanghai-China

Croatia

R² = 0.09

R² = 0.05

300

350

400

450

500

550

600

650

20 40 60 80 100 120 140 160 180 200 220

Ma

the

ma

tic

s p

erf

orm

an

ce

(sc

ore

po

ints

)

Teachers' salaries relative to per capita GDP (%)

Among high-income countries

high-performers pay teachers more

Per capita GDP less than USD 20 000

Per capita GDP over USD 20 000

Fig IV.1.10

Among low-income countries a host of other resources are the

principal barriers

In 33 countries schools where a higher share of principals reported that

teacher shortages hinder learning tend to show lower performance

Hong Kong-China

Brazil

Uruguay

Croatia

Latvia

Chinese Taipei

Thailand

Bulgaria

Jordan

Macao-China

UAE

Argentina

Indonesia

Kazakhstan

Peru

Costa RicaMontenegro

Tunisia

Qatar

Singapore

Colombia

MalaysiaSerbia

Romania

Viet Nam

Shanghai-China

USA

Poland

New Zealand

Greece

UK

Estonia

Finland

Slovak Rep.

Luxembourg

Germany

AustriaFrance

Japan

TurkeySweden Hungary

AustraliaIsrael

Canada

Ireland

Chile

Belgium

SpainDenmark

Switzerland

Iceland

Slovenia

PortugalNorway

Mexico

Korea

Italy

R² = 0.19

300

350

400

450

500

550

600

650

700

-0.500.511.5

Ma

the

ma

tic

s p

erf

orm

an

ce

(sc

ore

po

ints

)

Equity in resource allocation(index points)

Countries with better performance in mathematics tend

to allocate educational resources more equitably

Greater

equityLess

equity

Adjusted by per capita GDP

Fig IV.1.11

30% of the variation in math performance across OECD countries is explained by the degree of similarity of

educational resources between advantaged and disadvantaged schools

OECD countries tend to allocate at least an equal, if not a larger, number of teachers per student to disadvantaged schools; but disadvantaged schools tend to have great difficulty in attracting qualified teachers.

Schools with more autonomy perform better than schools with

less autonomy in systems with more accountability arrangements

School data not public

School data public464

466

468

470

472

474

476

478

Less school autonomy

More school autonomy

Score points

School autonomy for curriculum and assessment

x system's level of posting achievement data publicly

Fig IV.1.16

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Port

ug

al

Italy

Belg

ium

(F

lem

ish)

Me

xic

o

Hu

ng

ary

Chile

Ho

ng

Kon

g-C

hin

a

Germ

any

Cro

atia

Kore

a

Ma

ca

o-C

hin

a

Me

an

in

de

x c

ha

ng

e

Change in the index of perseverance that is associated with parents expecting the child to complete a university degree

Parents’ high expectations can foster

perseverance in their child24 Fig III.6.11

-16

-14

-12

-10

-8

-6

-4

-2

0

2

4

Germ

any

Belg

ium

(F

lem

ish)

Me

xic

o

Ch

ile

Port

ug

al

Italy

Ma

ca

o-C

hin

a

Ho

ng

Kon

g-C

hin

a

Cro

atia

Kore

a

Hung

ary

Pe

rce

nta

ge

-po

int

ch

an

ge

Percentage-point change in arriving late for school that is associated with parents expecting the child to complete a university degree

Before accounting for ESCS After accounting for ESCS

Parents’ expectations for their child have a strong

influence on students’ behaviour towards school25 Fig III.6.11

United States

Poland

Hong Kong-China

Brazil

New Zealand

Greece

Uruguay

United Kingdom

EstoniaFinland

Albania

Croatia

Latvia

Slovak RepublicLuxembourg

Germany

Lithuania

Austria

Czech Republic

Chinese Taipei

France

Thailand

Japan

Turkey Sweden

HungaryAustralia

Israel

Canada

IrelandBulgaria

Jordan

Chile

Macao-China

U.A.E.

Belgium

Netherlands

Spain

Argentina

Indonesia

Denmark

Kazakhstan

Peru

Costa Rica

Switzerland

Montenegro

Tunisia

Iceland

Slovenia

Qatar

Singapore

Portugal

Norway

Colombia

Malaysia

Mexico

Liechtenstein

Korea

Serbia

Russian Fed.

Romania

Viet Nam

Italy

Shanghai-China

R² = 0.36

300

350

400

450

500

550

600

650

-0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20

Me

an

ma

the

ma

tic

s p

erf

orm

an

ce

Mean index of mathematics self-efficacy

OE

CD

ave

rag

e

Countries where students have stronger beliefs

in their abilities perform better in mathematics26 Fig III.4.5

-20

-10

0

10

20

30

40

Colo

mb

iaC

osta

Ric

aP

eru

Isra

el

Luxe

mb

ou

rgC

hile

Tu

nis

iaS

lovak R

epu

blic

Lie

ch

tenste

inIt

aly

Kore

aS

pa

inA

rgen

tin

aB

razil

Port

ug

al

Gre

ece

Ja

pa

nA

ustr

iaU

rug

ua

yM

exic

oH

ong

Kon

g-C

hin

aB

ulg

aria

Tu

rkey

Ind

one

sia

Hung

ary

Vie

t N

am

Un

ited

Sta

tes

Rom

ania

U.A

.E.

Ch

inese

Taip

ei

Cana

da

Ire

land

Belg

ium

Kaza

kh

sta

nC

ze

ch R

ep

ub

licO

EC

D a

ve

rag

eC

roa

tia

Fra

nce

Sha

ngh

ai-

Ch

ina

Mo

nte

neg

roP

ola

nd

Serb

iaM

ala

ysia

Esto

nia

Qata

rM

aca

o-C

hin

aN

eth

erla

nds

Ne

w Z

eala

nd

Norw

ay

Lithu

ania

Slo

ven

iaD

enm

ark

Jo

rdan

Sw

itze

rla

nd

Austr

alia

Germ

any

Latv

iaR

ussia

n F

ed.

Sw

ede

nS

inga

po

reU

nited

Kin

gd

om

Th

aila

nd

Fin

land

Icela

nd

Sc

ore

-po

int

dif

fere

nc

e (

bo

ys

-gir

ls)

Gender gap among the highest-achieving students (90th percentile)

Gender gap adjusted for differences in mathematics self-efficacy between boys and girls

Gender gap

Greater self-efficacy among girls could shrink the gender gap in mathematics

performance, particularly among the highest-performing students 27 Fig III.7.12

Do you have an idea on how to use OECD

data to improve education in your country?

Would you like to work with us

to develop that idea?

Apply to the Thomas J. Alexander

fellowship programme

http://www.oecd.org/edu/thomasjalexanderfellowship.htm