Challenges in
International Large-Scale Assessments
Higher School of Economics, Moscow, Russia, May 16, 2013
Challenges
1. Using information to improve quality2. Measuring beyond the academics3. Reaching out of school population4. What to measure5. Difficulty level of the tests
Challenge 1: Using Assessment information to
improve education quality
Challenge 1: Using ILSA to improve education quality
A success Story: The case of Poland
Monitoring impact of reforms in Poland
Source: The World Bank. Knowledge Brief. Nov 2010, Vol 34
Poland: PISA Reading Scores
Challenges for effective use of assessments
• Political environment• Dissemination• Local capacity• Assessment culture
Challenge 2: Measuring beyond the academics
Success requires a combination of academic, personality and social skills
Personality is important to predict success
Personality Psychology and Economics. 2011Almlund, M., Duckworth, A., Heckman, J., Kautz, T.
Self-Discipline Outdoes IQ in Predicting Academic Performance of Adolescents
Duckworth, A. L. & Seligman, M. E. P. (2005). Psychological Science, 16, 939-944
Grit Scale
GRIT: Perseverance for achieving long-term goals
Grit Scale
Men1st-9th percentile: < 2.5010th-19th percentile: 2.51 - 2.8220th-29th percentile: 2.83 - 3.0530th-39th percentile: 3.06 - 3.2440th-49th percentile: 3.25 - 3.3750th-59th percentile: 3.38 - 3.5360th-69th percentile: 3.54 - 3.7470th-79th percentile: 3.75 - 3.9180th-89th percentile: 3.92 - 4.0090th-99th percentile: 4.21-5.00
Women1st-9th percentile: < 2.5010th-19th percentile: 2.51 - 2.8720th-29th percentile: 2.88 - 3.1230th-39th percentile: 3.13 - 3.2440th-49th percentile: 3.25 - 3.4950th-59th percentile: 3.50 - 3.6260th-69th percentile: 3.63 - 3.7870th-79th percentile: 3.79 - 3.9980th-89th percentile: 4.00 - 4.2490th-99th percentile: 4.25-5.00
Background Questionnaires• TIMSS 2011
– Attitudes towards math and science: Like, value, confidence
• Results– Younger students have more positive attitudes– Correlation with achievement
• Strong and positive within countries• Weak among countries
Discussion
• Should we move towards measuring personality traits? How?
Challenge 3:Reaching out of school population
Challenge 3:Reaching out of school population
73%
72%84%
91%
96%
n.a.
The PISA 2009 profiles by country/economy.http://stats.oecd.org/PISA2009Profiles/#
PISA 2006 Sample Coverage
Ferreira, F. & Gignoux, J. (2011). The Measurement of Educational Inequality. Achievement and Opportunity. Policy Research Paper 5873. Development Research Group. The World Bank
Cov
ered
Pop
/ T
otal
Pop
PISA 2006 Sample Coverage
Ferreira, F. & Gignoux, J. (2011). The Measurement of Educational Inequality. Achievement and Opportunity. Policy Research Paper 5873. Development Research Group. The World Bank
Cov
ered
Pop
/ T
otal
Pop
390
513
438
Science Scores
Which country did better?
Country Y:
Country X:
70%
90%
2ary school enrollment 400
375
Did the country stay the same?
2009
2000
80%
70%
2ary school enrollment 400
400
Country X
Discussion
• Should we compare countries with different proportions of out-of-school population?
Challenge 4: What to measure?
The traditional view in assessment
Emphasis on opportunities to learn
The “new” view in assessment
Society
PISA approach: to what extent students can apply their knowledge to real-life situations and be equipped for full participation in society.
Measuring 21st Century Skills
21st Century Learning Framework
• Core subjects and interdisciplinary themes– Language, math, sciences, – Finances, health, civics
• Skills– Creativity and innovation– Critical thinking and problem solving– Communication and collaboration– Information, media, and ICT literacy
Challenge 5:Difficulty level of the tests
Measuring what student should know and be able to do?
Measuring what student actually know and can do?
Difficulty Level of the tests
Difficulty Level of the Tests
Focus on and Retrieve Explicitly Stated Information
Make Straightforward Inferences
Interpret and Integrate Ideas and Information
Examine and Evaluate Content, Language, and Textual Elements
Recognize letters
Recognize words
Read short sentences
PIRLS & PrePIRLS
EGRA, Provinha
PrePIRLS Example questionsRetrieve explicitly stated information
ProvinhaExample Question
Identify Letters
Source: Provinha March 2012. Caderno do Professor.
X
Administrator reads test questions to the students.
Test questions are only included in the Test Administrator booklet
Provinha Example question
Read words
Source: Provinha March 2012. Caderno do Professor.
X
Difficulty Level of the Tests
Focus on and Retrieve Explicitly Stated Information
Make Straightforward Inferences
Interpret and Integrate Ideas and Information
Examine and Evaluate Content, Language, and Textual Elements
Recognize letters
Recognize words
Read short sentences
PIRLS & PrePIRLS
EGRA, Provinha
Opposed Forces
More low ability students expected to take the tests in more developing countries
Push for more difficult “higher order thinking skills” questions
How to address this?
new
newnew new
newnew
newnew PISA for Development
PrePIRLS
Universal scale
Discussion
What challenges do you foresee?How can they be addressed?