22 April 2009

Benefits and limitations of international

comparative research in science and

mathematics education

AMSTEL – SeminarPauline Vos

Universiteit van Amsterdam

AMSTEL Instituut

2

International comparison Use identical instruments in different countries and look at differences between countries.

My own international experience: Went to school in NL, D and F worked in NL, Zimbabwe, Mozambique &

South Africa Worked at the Dutch research centre for

TIMSS-1999 (University of Twente) PhD on mathematics curriculum in NL using

TIMSS-instruments

3

Many variations

Bilateral studies

Affect in science: ROSE (Relevance of Science Education), 40 countries

Teacher competencies: TEDS(M), 17 countries

4

Two abbreviations…PISA = Project for International Student Assessment

organised by OECD(Organisation for Economic Cooperation and Development)

TIMSS = Trends in International Maths and Science Studypreviously: Third Internat. Maths & Science Study

organised by IEA (Internat. Assoc. for the Evaluation of Educational Achievement)

5

Translate and define

Mathematics Basic calculations and further

Science physics, chemistry, biology, physical

geography, astronomy, ecology, etc

Students all students (not just pre-university)

6

NL USA Fla Jap Maroc Russ.

38 49 38 47 16 37

7

NL USA Fla Jap Maroc Russ.

45 44 46 31 16 38

8

NL USA Fla Jap Maroc Russ.

82 76 76 77 33 74

9

NL USA Fla Jap Maroc Russ.

44 36 66 80 31 55

10

NL 36

USA 19

Fla 21

Jap 44

Maroc

2

Russ. 9

11

NL USA Fla Jap Maroc Russ.

81 74 71 78 45 76

12

Some history…1964 A comparison of 12 countries, Husén & Bloom1971 FISS1980-82 SIMS 1984 SISS

1995 TIMSS (later known as “TIMSS-1995” )

1999 TIMSS-Repeat (later known as “TIMSS-1999”)

2003 TIMSS-2003

2007 TIMSS-20072008 TIMSS Advanced

TIMSS-2011

2000 PISA-2000

2003 PISA-2003

2006 PISA-2006

PISA-2009

22 April 2009

To what extent do countries differ in the education of science and mathematics?

How can you do research on the question:

14

Hours of instruction for science (grd 4)

15

Teacher characteristics math (grd 8)

22 April 2009

Both studies focus on a test

17

Complex studies ..Assessment frameworkLarge number of itemsLong design process for instrumentsLarge random sampleQuality control trend items (retained) coding and re-coding excluding countries when criteria were

not met

18

Pisa framework for science

19

Pisa framework for science competencies

20

Pisa framework for science

21

Pisa framework for science

22

Pisa science item ‘Acid Rain’

23

Pisa science item ‘Acid Rain’

24

Pisa science item ‘Acid Rain’

25

Complex studies… (1)Not just measuring country scores in one discipline, but also: Sub-disciplines (algebra, ecology, ..) Sub-competencies

(reproduce, explain, communicate, ..) Background variables

Not just measuring students’ scores, but also trying to explain: Differences between countries Differences between schools Differences between students

26

Complex studies… (2)PISA: 85 exercises distributed over 13 test booklets

PISA-2000 Main topic: Literacy (reading) Additional: science, maths

PISA-2003: Main topic: Mathematical Literacy Additional: literacy, science, problem solving

PISA-2006 Main topic: Science Additional: literacy, maths

27

Complex studies… (3)TIMSS: 300 exercises distributed over 12 booklets

TIMSS-95: grades 3, 4, grades 7, 8 Performance Assessment (= test in a lab-like environment) grade 12: math/science literacy; advanced maths/science video study (USA, Jap, Germ.)

TIMSS-99: grade 8 TIMSS-R Video study (USA, Jap, NL, Cz, Switz, Austr, HongK.)

TIMSS-2003 grade 4, grade 8

TIMSS-2007 grade 4, grade 8

TIMSS-2008 grade 12

28

Percentage Distribution Math Items PISA - TIMSS

15-year-olds

PISAGrade 8

TIMSSNumber 22% 33%

Measurement 18% 14%

Geometry 12% 18%

Data 40% 11%Algebra 11% 23%Classified to multiple strands 2% 0%

29

Math Item Formats

Multiple Choice

Short Constructed

Response

Extended Constructed

Response

TIMSS 71% 16% 13%

PISA* 33% 42% 25%

*PISA definitions:Multiple Choice includes regular and “complex” multiple-choice items.Short Constructed Response includes “short answer” and “closed response” items.Extended Constructed Response includes “open response” items.

30

Mathematical Complexity

LowModerat

eHigh

TIMSS 51% 46% 3%

PISA 29% 64% 7%

31

PISA-2006 science country ranking

32

PISA-2006 science country ranking

33

TIMSS-2007country ranking

Maths grade 4

34

TIMSS-2007 multi-country comparison

35

Horse race - thinkingboth in PISA and TIMSS

Ranking is simplisticSample: intrinsically inaccurateScores of some countries are very close. With s.e.>3 are small differences not significantNL position in 2003: math: on shared positions 2-7 in PISA

and shared positions 6-9 in TIMSS science: on shared positions 4-8 in PISA

and shared positions 7-11 in TIMSS

36

Differences PISA and TIMSSPopulation:

PISA: age cohort: 15-year olds (indep. of grade) TIMSS: grade (indep. of age)

Countries: PISA: economically developed (OECD) TIMSS: whole world (incl. Saudi Arabia, Botswana)

Cyclus: PISA: 3-yearly with different thematic foci TIMSS: 4-yearly with different populations

Content: PISA: applicable knowledge (situations): reading, math.

literacy, scientific literacy TIMSS: school knowledge (US curriculum): mathematics,

scienceJustification of the study:

PISA: knowledge has economical importance TIMSS: knowledge about education is needed

37

Country score and GDP

38

39

TIMSS-1999 Video StudyGrade 8seven countries: Australia, Hong Kong, Japan, NL, Czech Rep, USA, SwitzerlandAim: describe patterns in lessonsquestion: what is typical for the math lessons in a certain country?

in each country 85 random lesson filmed, transcribed & coded

Points of analysis:•Structure of the lesson: e.g. who speaks? How many exercises?•Mathematical content: complexity, procedures•Char of exercise: symbols, contexts

40

TIMSS-Video study: use of materials:Percentage lessons in which a calculator is used

41

TIMSS-Video study: structure of lessonsPercentage lessons in which a goal is stated

42

TIMSS-Video study: structure of lessonsPercentage lessons, in which a summary statement is made

43

TIMSS-Video study: exercisesPercentage exercises per lesson with a context of symbols only

44

TIMSS-1999 Video Study

USA

video USA

Grade 8 (14 yrs) Heterogeneous

non-problem segment:the teacher shows the class examples of angles formed by secants and tangents intersecting inside, on, or outside a circle.

45

TIMSS-1999 Video Study

Netherlands

video NL

Grade 8 (14 yrs) Homogeneous (pre-university)

non-problem segment:the teacher briefly presents historical background related to the Pythagorean theorem (i.e., Pythagoras was a Greek gentleman). She also connects ideas learned previously (i.e., the definitions of various types of triangles) to the present lesson.

46

TIMSS-1999 Video Study

Hong Kong: grd 8, heterogeneousExplanation of a new concept.

video HongKong

Pre-knowledge: students can solve linear equations

What preceded the scene:Solving2x+10=2(x+5)2x+10=2x+10 0 = 0Then different values were insterted for x .

left hand sideright hand side“expanded form”“factorized form”

47

Next in this theatre…

TEDS(M)report in December 2009

TIMSS-2008 Advancedreport in December 2009

Pisa 2009report in 2010

48

References for PISA and TIMSSwww.pisa.oecd.org

http://timss.bc.edu

49

remains the question….

what are the benefits and limitations?