PISA for Development

Technical Strand 1:Contextual questionnaires

Overview of PISA instruments(constructs, indices and variables)

EDU/DCD

PISA for DevelopmentContextual questionnaires

2

Purpose of PISA contextual questionnaires

How are they used?

How they are developed

What will PISA for Development seek to do?

Questions for discussion

Student and school questionnaires are part of the core assessment design of PISA

Countries may also opt for additional questionnaires:

- Parents, Teachers, Health, Time Use, ICT familiarity, Career…

Student questionnaires: ~ 30 minute* stand-alone document (students respond after test questions)

School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing)

* Rotated questionnaires in 2012 to cover additional material

PISA for DevelopmentPurpose of PISA contextual questionnaires

Contextualise the performance results in reading, mathematics and science*

Allow analyses of equity issues across sub-groups and populations

Provide indications of possible policy interventions

Some information collected is used to scale cognitive assessment data

Country-specific questions can be included for additional analysis

* Additional domains such as financial literacy and problem-solving

PISA for DevelopmentPurpose of PISA contextual questionnaires

Have been used in 74 countries/economies In more than 40 languages Publicly available

PISA 2009 Student QuestionnairePISA 2012 Student Questionnaire

PISA 2012 School QuestionnairePISA 2009 School Questionnaire

PISA for DevelopmentPurpose of PISA contextual questionnaires

National educ., social and economic context

Structures, resource allocation

and policies

Social & economic outcomes of education

Community and school

characteristics

Student learning, teacher working

conditions

Socio-economic background of

learners

Antecedentscontextualise or

constrained policy

The learning environment at

school

Teaching, learning practices and

classroom climate

Individual attitudes,

engagement and behaviour

Output and performance of

institutions

Quality of instructional

delivery

Quality and distribution of

knowledge & skills

Policy Leversshape educational

outcomes

Outputs and Outcomes

impact of learning

Individual learner

Instructional settings

Schools (other institutions)

Country or system

This is the PISA Analytic Framework that guides data collection and analysis

Leve

ls o

f an

edu

cati

on s

yste

mPISA Contextual QuestionnairesHow they are used

Outcomes Processes Antecedents

Students

• General recurring variables (all cycles)

• Domain recurring variables (main domain every 3 years)

• Extension variables (specific cycles)

• System-level data

Classrooms

Schools

Country

Main data collectionStudent Questionnaire, School Questionnaires and System Level*

This is how the PISA Analytic Framework is operationalised for data collection and analysis

Variables included in instruments:

* Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career

PISA Contextual QuestionnairesHow they are used

8

PISA Contextual QuestionnairesSystem-level data collection

• Ratio: students / teaching staff• Spending /student• Instruction time (by age)• Teachers’ salaries

(by education, experience)• Evaluation & accountability• Educational stratification• Decision-making

at different levels• School choice• Changes in education

policy• Others (…)

Standards and procedures for data collection in PISA Questionnaires for non-OECD economies Panel of experts to complete some data

collection

• Within-country disparities• Quality, supply and

distribution of educational materials and resources

• (Extra-)tuition• Parental support• Contractual arrangements

of school teaching staff

Sample of relevant areas to address in developing

contexts

These are examples of the types of information collected at the system level:

9

• Grades served• Public-Private• Funding sources• Community type• Enrollment (boys, girls)• Average class size• Non-native speakers• Grouping/tracking• Teachers, by qualifications• Teacher shortages • Physical building limitations• Instructional materials• Teacher attitudes, expectations,

relations• Computer/network/web access

• Absenteeism, disruptions• Teacher morale• Student morale• Teaching environment• Time for special programmes• Non-native speaker

offerings• Admissions requirements• Assessment types• Uses of assessment• Assessment of teachers• Decision making

authority• External decision makers

School resources, policies and practicesThis is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources

These are the constructs that will be reviewed as part of the technical work

of PISA for Development

PISA Contextual QuestionnairesSchool questionnaires

10

PISA Contextual QuestionnairesStudent questionnaires

General and domain-related processesSense of belonging at schoolStudent-teacher relations at schoolDisciplinary climate in classesTeacher support in subject-specific classesOut-of-school activities (homework, etc...)

Self and domain-related cognitionsAnxiety towards “study subject”Instrumental motivation to learn “study subject”Interest in and enjoyment of “study subject”Self-efficacy in “study subject”Self-concept in “study subject”

Learning strategiesPreference for competitive learning situationsPreference for co-operative learning situationsControl strategiesPerseverance

Others

Health status, Time use

(…)

These are examples of the types ofnon-cognitive outcomeson which PISA collects information

These are the

constructs that will be reviewed as part of the technical work of PISA for

Development

Student attitudes, behaviours and approaches to learning

11

… How are questionnaire data used for relevant analysis and comparative policy insights?

(a few examples from PISA results)

PISA Contextual QuestionnairesExamples of analysis

12

Sha

ngha

i-Chi

na

Sin

gapo

re

Mac

ao-C

hina

Dub

ai (U

AE

)

Uni

ted

Ara

b E

mira

tes

Qat

ar

Chi

nese

Tai

pei

Lith

uani

a

Cos

ta R

ica

Bul

garia

Rom

ania

Tha

iland

Col

ombi

a

Mon

tene

gro

Jord

an

Arg

entin

a

Pan

ama

Aze

rbai

jan

Indo

nesi

a

Geo

rgia

Him

acha

l Pra

desh

-Indi

a

Kyr

gyzs

tan

300

350

400

450

500

550

6002009 2006 2003 2000Reading per-

formance Low-income economies ($1,025 or less)Lower-middle-income economies ($1,026 to $4,035)Upper-middle-income economies ($4,036 to $12,475)High-income economies ($12,476 or more)

Examples of analysisRange of performance among non-OECD countries

PISA Results for different years

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2200

250

300

350

400

450

500

550

600Score

OECD average

OEC

D a

vera

geStu

dent

per

form

ance

AdvantagePISA Index of socio-economic backgroundDisadvantage

School performance and students’ socio-economic background within schools

Student performance and schools’ socio-economic background Private school Public school in rural area Public school in urban area

Mexico

Schools with a similar socio-economic profile

PISA 2009 ResultsExamples of analysisSchool performance and socio-economic background

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2200

250

300

350

400

450

500

550

600Score

OEC

D a

vera

ge

Stu

dent

per

form

ance

AdvantagePISA Index of socio-economic backgroundDisadvantage

Private school Public school in rural area Public school in urban area

OECD average

Examples of analysisSchool performance and socio-economic backgroundSchool performance and students’ socio-economic background within schools

Student performance and schools’ socio-economic background

Kyrgyzstan

PISA 2009 Results

Examples of analysisPolicy findings from contextual information

Policies and practices

Learning climate

Discipline

Teacher behaviour

Parental pressure

Teacher-student relationships

Dealing with heterogeneity

Grade repetition

Prevalence of tracking

Expulsions

Ability grouping (all subjects)

Standards /accountability

Nat. examination

Standardised tests

PolicySystem

RSchool

REquity

E

Policy briefs on these issues can be found in the PISA in Focus series

Examples of analysisPolicy insights using contextual information

Commitment to

universal achievement

Goals, gateways,

instructional systems

Capacity at point of delivery

Incentives and

accountability

Resources where they yield most

A learning system

Coherence

Lessons from PISA from successful

education systems (equitable and improvers)

Based on the analysis of cognitive outcomes

and contextual information

17

PISA Contextual QuestionnairesHow they are developed

‘Crowd sourcing’ and collaboration– PISA draws together leading expertise and institutions from

participating countries to develop instruments and methodologies…

… guided by governments on the basis of shared policy interests

Cross-national relevance and transferability of policy experiences

– Emphasis on validity across cultures, languages and systems– Frameworks built on well-structured conceptual

understandingof assessment areas and contextual factors

Continuous review and updating of assessment frameworks (cycles)

– Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA

– Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors

• Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions)

• School questionnaire: 33 questions (5 additional questions on financial education at school)

• Parental questionnaire: 25 questions• ICT questionnaire: 12 questions• Education careers: 14 questions

PISA Contextual QuestionnairesHow they are developed

PISA 2012 Questionnaires

Linkages between cognitive and non-cognitive outcomes

• Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountry DIF, gender DIF)

• Validation of psychometric equivalence of translated instruments (and adaptations from source versions) issues addressed

• Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study

• Determine cross-national validity of the questions in the questionnaire

• Technical standards ensure that data from participating countries are internationally comparable

PISA Contextual QuestionnairesHow they are developed

Purpose of the PISA Field Trial (for Main Study)

Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results

Adapt existing constructs (and indices and variables)

Identify and introduce new constructs (indices and variables)

Confirm and validate enhancements through field trials and main data collection in participating countries

PISA Contextual QuestionnairesWhat will PISA for Development seek to do?

Student Questionnaire

Section 1: ABOUT YOUSection 2: YOUR FAMILY AND YOUR HOME Section 3: YOUR READING ACTIVITIESSection 4: LEARNING TIMESection 5: YOUR SCHOOLSection 6: YOUR SCHOOL LESSONS AND SUBJECTSSection 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTSSection 8: YOUR VIEWS ON <BROAD SCIENCE> Section 9: CAREERS AND <BROAD SCIENCE>Section 10: YOUR MATHEMATICS EXPERIENCES

Examples of variables and indices

PISA Contextual QuestionnairesWhat will PISA for Development seek to do?

Student QuestionnaireSECTION 2: YOUR FAMILY AND YOUR HOME2009-ST08: Family structure2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ)2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED)2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED)2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ)2009-ST13: Father’s main job (ESCS, HISEI, BMFJ)2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED)2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED)2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ)2009-ST17: Country of birth for student and parents (IMMIG)2009-ST18: Age at arrival in country of test2009-ST19: Language spoken at home2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS)2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH)2009-ST22: Books in home (ESCS, HOMEPOS)

PISA Contextual QuestionnairesStudent questionnaires

Examples of variables and indices

Comparability across countries (not a country-specific income indicator)

PISA index of Economic, Social and Cultural Status(ESCS)

HISEIHighest occupational status of parents

PAREDHighest educational level of parents

HOMEPOSHome possessions

WEALTH

CULTPOSS

HEDRES

# of books in home

PISA Contextual QuestionnairesStudent questionnaires – examples of indices

WEALTH

CULTPOSS

HEDRES

# of books in home

Index of family wealtha room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms witha bath or shower (ST21).

Index of cultural possessionsclassic literature, books of poetry and works of art (ST20)Index of home educational resourcesFor example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20)

PISA index of Economic, Social and Cultural Status(ESCS)

PISA Contextual QuestionnairesStudent questionnaires – examples of indices

2006-ST17: Self-efficacy in science (SCIEEFF)

2003-ST31: Self-efficacy in mathematics (MATHEFF)

Self-efficacy in science and mathematics

PISA Contextual QuestionnairesStudent questionnaires – examples of indices

Relationship between constructs, indices and variables (questions)

Stays the same in PISA 2012

06-ST17Q42 How easy do you think it would be for you to

perform the following tasks on your own?(Please darken only one circle in each row.)

I could do this easily

I could do this

with a bit of effort

I would struggle to do this

on my own

I couldn’t do this

a) Recognise the science question that underlies a newspaper report on a health issue

●1 ●2 ●3 ●4

b) Explain why earthquakes occur more frequently in some areas than in others

●1 ●2 ●3 ●4

c) Describe the role of antibiotics in the treatment of disease

●1 ●2 ●3 ●4

d) Identify the science question associated with the disposal of garbage

●1 ●2 ●3 ●4

e) Predict how changes to an environment will affect the survival of certain species

●1 ●2 ●3 ●4

f) Interpret the scientific information provided on the labels of food items

●1 ●2 ●3 ●4

g) Discuss how new evidence can lead you to change your understanding about the possibility of life on Mars

●1 ●2 ●3 ●4

h) Identify the better of two explanations for the formation of acid rain

●1 ●2 ●3 ●4

Self-efficacy in science

PISA Contextual QuestionnairesStudent questionnaires – examples of indices

2006-ST17: Self-efficacy in science (SCIEEFF)

2006-ST35: Instrumental motivation to learn science (INSTSCIE)

06-ST35

Q46 How much do you agree with the statements below?(Please darken only one circle in each row.)

Strongly agree Agree Disagree

Strongly disagree

a) Making an effort in my science class(es) is worth it because this will help me in the work I want to do later on

●1 ●2 ●3 ●4

b) What I learn in my science class(es) is important for me because I need this for what I want to study later on

●1 ●2 ●3 ●4

c) I study science because I know it is useful for me ●1 ●2 ●3 ●4

d) Studying science is worthwhile for me because what I learn will improve my career prospects

●1 ●2 ●3 ●4

e) I will learn many things in my science class(es) that will help me get a job ●1 ●2 ●3 ●4

Instrumental motivation to learn science

PISA Contextual QuestionnairesStudent questionnaires – examples of indices

Identify constructs in need of adaptation and important constructs that are missing

Obtain input and advice from experts in the field (grounded in developing-country contexts)

Obtain input and guidance from participating countries

Draw on existing work and empirical evidence

Test the constructs (indices and variables) to ensure comparability with PISA scales

PISA for DevelopmentHow will we do this?

PISA for Development

Technical Strand 1:Contextual questionnaires

Are the intentions and rationale of this strand of work clear and understood?

Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation?

Where do you see the largest hurdle to overcome?

What existing studies and evidence should contribute to this strand of work?

PISA for DevelopmentQuestions for discussion