DATA-DRIVEN POLICY ANALYSIS AND INNOVATION
IN EDUCATION: AN OECD PERSPECTIVE
Carlos González-Sancho
OECD Directorate for Education and Skills
Centre for Educational Research and Innovation (CERI)
25 October 2018
Dushanbe, Republic of Tajikistan
CENTRAL ASIA SYMPOSIUM ON ICT IN EDUCATION 2018
Strengthening Education Management Information Systems
to monitor SDG4
Outline
1. Role of data for education policy planning and evaluation
OECD work on SDG 4 monitoring
Some examples of analysis with PISA data
2. Longitudinal information systems in education in OECD countries: current state and future directions
Insights from the OECD/CERI survey of information systems
A typology: four model approaches to using longitudinal systems
Some challenges
Key messages
OECD is strongly engaged in 2030 Agenda and SDG 4 monitoring
Education SDG architecture, TCGs, reporting, data collection, capacity building
EMIS development should enable a wider range of uses of education data
From statistical reporting and evaluation, to research and innovation for improvement
Requires enhancing the capacities of information systems, most importantly with
student longitudinal identifiers, new types of data and more flexible access
Longitudinal education information systems in OECD countries provide
examples of what is possible for EMIS going forward
Many good models and solutions can already be found in the education sector
No need to restart from scratch – strong longitudinal information systems can often
be built from legacy systems
OECD role in SGD 4
framework and monitoring
Not all OECD countries are at the same level
when it comes to meeting the SDG 4 targets
0
10
20
30
40
50
60
70
80
90
100
4.a.1.% of studentswith access to
computers andInternet
4.2.2.Enrolment rate
a year beforeprimary entry
age
4.c.7.% of teacherswho received
in-servicetraining
4.7.5.Proficiency of
15-year-olds inscience
4.6.1.Adult
proficiency inliteracy andnumeracy
4.1.1.Proficiency of
15-year-olds inmaths and
reading
4.a.2.% of studentsexperiencing
bullying
4.1.5.Out-of-school
rate
Indicators for which higher values are desirable Indicators for which lower
values are desirable
General overview of selected SDG indicators
OECD’s support for the Education SDG Action
Plan
Leverage OECD indicators and collect
data for the SDG 4 UN database
Joint validator of SDG 4 indicators and advisor
within the SDG 4 framework
Reporter of progress towards SDG 4
SGD lens to education strategies and support for education policy-
making at country level
Some indicators produced and/or with input by OECD teams:
PISA: SDG Target 4.1
Early Learning and Child Well-being Study: SDG Target 4.2
PIAAC: SDG Target 4.6
TALIS: SDG Target 4.c
UOE questionnaires and additional data collections: e.g. on Indicator 4.a.1 on the infrastructure of schools
Collecting data for the UN database (UIS) for OECD and partner countries
About 90% coverage of global indicators currently
1. Leverage OECD data to analyse progress on
SDG4 and collect data for UN database
A common SDG 4 database requires agreement on methodology and sources for calculation of SDG indicators
TCG Working Group 1 on Indicator Development
Provide feedback on relevant indicators, esp. refinement
Examples classified as “requires further development”: 4.3.1; 4.6.3; 4.7.1; 4.7.2; 4.a.2; 4.a.3.
TCG Working Group 3 on Data Reporting, Validation and Dissemination
Terminology document
Data validation package
2. Joint validator of SDG 4 indicators and advisor
within the SDG 4 framework
3. Reporter of progress towards SDG 4
EAG : a vehicle for reporting on SDG 4 on OECD and partner countries, with a dedicated chapter
EAG 2018 focus on equity SDG4 Target 4.5
Dedicated sections in other publications as well
4. Apply SDG lens to OECD education strategies
and policy tools, including capacity building
PISA for Development is enhancing PISA to make it relevant for low-and-middle-income countries
Successful pilot: Bhutan, Cambodia, Ecuador, Guatemala, Honduras, Panama, Paraguay, Senegal and Zambia
Mainstreamed in PISA from 2021 onwards
Assistance to countries in building national assessment and data-collection systems
Peer-to-peer partnerships (e.g. Korea and Cambodia)
Integrate SDG4 and its targets and indicators in on-going and future support for education policy-making at the country level, e.g. country reviews
some examples of analysis
with PISA data
100
80
60
40
20
0
20
40
60
80
100
Vie
t N
am
Ma
ca
o (
Ch
ina
)E
sto
nia
Ho
ng K
on
g (
Chin
a)
Sin
gapo
reJapa
nC
anad
aF
inla
nd
Ch
ine
se
Taip
ei
Kore
aS
loven
iaIr
ela
nd
De
nm
ark
B-S
-J-G
(C
hin
a)
Pola
nd
Germ
any
La
tvia
Port
ug
al
Un
ite
d K
ing
dom
Ne
w Z
eala
nd
Austr
alia
Ru
ssia
Spain
Sw
itzerl
and
Ne
therl
and
sN
orw
ay
Belg
ium
Un
ite
d S
tate
sC
zech R
epu
blic
Austr
iaO
EC
D a
vera
ge
Sw
ede
nF
rance
CA
BA
(A
rgentina
)Ita
lyC
roa
tia
Lithu
ania
Icela
nd
Lu
xe
mbo
urg
Hu
nga
ryS
lovak R
epub
licIs
rael
Ma
lta
Gre
ece
Ch
ileB
ulg
aria
Ro
man
iaU
rug
uay
Alb
ania
Un
ite
d A
rab E
mira
tes
Mo
ldova
Turk
ey
Trin
ida
d a
nd T
obag
oC
osta
Ric
aT
ha
iland
Me
xic
oC
olo
mbia
Jord
an
Qata
rG
eo
rgia
Mo
nte
neg
roIn
don
esia
Bra
zil
Peru
Le
ban
on
FY
RO
MT
un
isia
Kosovo
Alg
eria
Do
min
ican
Rep
ublic
Level 1a Level 1b
Below Level 1b Level 2
Level 3 Level 4
Level 5 Level 6
Comparing average system performance:
Students’ proficiency in science
PISA 2015, Figure I.2.15
%
Students at or
above Level 2
Students below
Level 2
Across OECD countries, disadvantaged students are 3 times more likely to not attain baseline
proficiency in science – in France and Singapore, about 4 times
1
2
3
4
5
6
7
Do
min
ican
Rep
ublic
CA
BA
(A
rgentina
)P
eru
Sin
gapo
reF
rance
Hu
nga
ryB
-S-J
-G (
Ch
ina
)Lu
xe
mbo
urg
Ch
ileB
ulg
aria
Belg
ium
Czech R
epu
blic
Slo
vak R
epub
licG
erm
any
Sw
itzerl
and
Ch
ine
se
Taip
ei
Ne
w Z
eala
nd
Spain
Austr
iaJapa
nP
ort
ug
al
Pola
nd
Austr
alia
Isra
el
Uru
guay
OE
CD
avera
ge
Ma
lta
Irela
nd
Gre
ece
Jord
an
Le
ban
on
Ro
man
iaS
loven
iaC
osta
Ric
aIta
lyM
exic
oF
inla
nd
Geo
rgia
Ne
therl
and
sS
wede
nB
razil
Mo
ldova
Lithu
ania
Ca
nad
aQ
ata
rU
nite
d S
tate
sD
enm
ark
Co
lom
bia
Indon
esia
Kore
aN
orw
ay
Tun
isia
Un
ite
d A
rab E
mira
tes
Un
ite
d K
ing
dom
Ru
ssia
Cro
atia
Trin
ida
d a
nd T
obag
oF
YR
OM
Vie
t N
am
Turk
ey
Esto
nia
Ho
ng K
on
g (
Chin
a)
La
tvia
Mo
nte
neg
roK
osovo
Icela
nd
Tha
iland
Ma
ca
o (
Ch
ina
)A
lge
ria
Odds ratio Increased likelihood of students in the bottom quarter of ESCS scoring below Level 2in science, relative to non-disadvantaged students (3 other quarters of ESCS)
Comparing the relative strength of the socio-economic
gradient on student performance
PISA 2015, Figure I.6.9
Comparing the variation between and within
schools in student performance (in science)
120
100
80
60
40
20
0
20
40
60
80
Ne
therl
and
s
114
B-S
-J-G
(C
hin
a)
119
Bulg
aria
1
15
Hu
nga
ry 1
04
Trin
ida
d a
nd T
obag
o 9
8B
elg
ium
1
12
Slo
ven
ia 1
01
Germ
any 1
10
Slo
vak R
epub
lic
10
9M
alta
1
54
Un
ite
d A
rab E
mira
tes 1
10
Austr
ia
10
6Is
rael 1
26
Le
ban
on
91
Czech R
epu
blic
1
01
Qata
r 10
9Japa
n 9
7S
witzerl
and
1
10
Sin
gapo
re 1
20
Ita
ly 9
3C
hin
ese
Taip
ei 1
11
Lu
xe
mbo
urg
1
12
Turk
ey 7
0B
razil
89
Cro
atia
8
9G
reece
94
Ch
ile 8
3Lithu
ania
9
2O
EC
D a
vera
ge 1
00
Uru
guay 8
4C
AB
A (
Arg
entina
)
82
Ro
man
ia
70
Vie
t N
am
6
5K
ore
a
10
1A
ustr
alia
1
17
Un
ite
d K
ing
dom
1
11
Peru
6
6C
olo
mbia
7
2T
ha
iland
6
9H
ong K
on
g (
Chin
a)
72
FY
RO
M 8
0P
ort
ug
al 94
Do
min
ican
Rep
ublic
5
9In
don
esia
5
2G
eo
rgia
9
2Jord
an
7
9N
ew
Zeala
nd 1
21
Un
ite
d S
tate
s
108
Mo
nte
neg
ro 8
1T
un
isia
4
7S
wede
n 1
17
Me
xic
o 5
7A
lba
nia
6
9K
osovo 5
7M
aca
o (
Ch
ina
) 74
Alg
eria
54
Esto
nia
8
8M
old
ova 8
3C
osta
Ric
a 5
5R
ussia
7
6C
anad
a 9
5P
ola
nd
92
De
nm
ark
9
1La
tvia
7
5Ir
ela
nd
8
8S
pain
8
6N
orw
ay
10
3F
inla
nd
103
Icela
nd
9
3
% Between-school variation Within-school variation
Total variation as a
proportion of the OECD
average
OECD average 69%
OECD average 30%
PISA 2015, Figure I.6.11
Comparing how learning time is associated with
student performance in science
6
7
8
9
10
11
12
13
14
15
16
0
10
20
30
40
50
60
70
Fin
land
Germ
any
Sw
itzerl
and
Japa
nE
sto
nia
Sw
ede
nN
eth
erl
and
sN
ew
Zeala
nd
Austr
alia
Czech R
epu
blic
Ma
ca
o (
Ch
ina
)U
nite
d K
ing
dom
Ca
nad
aB
elg
ium
Fra
nce
No
rwa
yS
loven
iaIc
ela
nd
Lu
xe
mbo
urg
Irela
nd
La
tvia
Ho
ng K
on
g (
Chin
a)
OE
CD
avera
ge
Ch
ine
se
Taip
ei
Austr
iaP
ort
ug
al
Uru
guay
Lithu
ania
Sin
gapo
reD
enm
ark
Hu
nga
ryP
ola
nd
Slo
vak R
epub
licS
pain
Cro
atia
Un
ite
d S
tate
sIs
rael
Bulg
aria
Kore
aR
ussia
Ita
lyG
reece
B-S
-J-G
(C
hin
a)
Co
lom
bia
Ch
ileM
exic
oB
razil
Co
sta
Ric
aT
urk
ey
Mo
nte
neg
roP
eru
Qata
rT
ha
iland
Un
ite
d A
rab E
mira
tes
Tun
isia
Do
min
ican
Rep
ublic
Score
poin
ts in s
cie
nce p
er
hour
of to
tal le
arn
ing t
ime
HoursIntended learning time at school (hours) Study time after school (hours)
Score points in science per hour of total learning time
PISA 2015, Figure II.6.23
longitudinal education information
systems in OECD countries:
current state and future directions
Longitudinal information systems: a general-purpose
technology supporting the innovation ecosystem
Longitudinal systems maintain and link
individual-level data over time, provide
detailed information on students’ learning
outcomes, schooling environments and
demographics; and facilitate access to data
through reporting and analysis tools
Next-generation systems integrate
statistical data with learning management
systems, including banks of digital resources
Data-driven innovation in education:
mainly about transforming information in
actionable knowledge – much more
than a technical issue
The opportunities around longitudinal information
systems in education
Improve efficiency and reduce administrative costs
Creation of a better data infrastructure for educational research
Faster and richer feedback to stakeholders:
New conversations around evidence on the impact of policies and practices
More applications around formative assessment and instruction
Platforms to access and share digital resources to support teachers and
learners – and develop a stronger educational industry
Mobilise practical knowledge - networks of educators and schools with
similar concerns (learning communities)
the OECD/CERI survey of
information systems in education
The OECD/CERI survey on information systems
in education
As of 2016, it covers 67 systems from 32 countries/economies
Australia [3], Austria [2], Belgium [2], Brazil [2], Canada [2], Chile [2], Czech Republic, Estonia, France, Germany, Hungary, India, Israel, Italy, Japan, Korea [2], Lithuania, Mexico, Netherlands [3], New Zealand, Norway [2], Portugal, Slovak Republic [2], Slovenia [2], South Africa, Spain [2], Sweden [2], Turkey, UK [2], US [20]
Administered to systems managers
US state-wide systems: from DQC
Survey sections
1. Goals of the system
2. Data model
3. Coverage and frequency of collection
4. Data linkages
5. Quality processes
6. Access and privacy
7. Comparison possibilities
8. Accountability usage
9. Instructional support, networking facilities and PD
10. Other features
Longitudinal identifiers and linkages
All systems have school-level identifiers, and 4 in 5 can track students longitudinally
Fewer systems provide teacher and course identifiers
Student- and school-level data matched, but teacher and student data linked only by a third of the systems, mainly US
Some cases where link does not exist despite availability of both identifiers
81%
61% 61%
39%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
School ID Longit StudentID
Longit TeacherID
All 3 Longit student-teacher link
Speed of feedback
Timeliness of feedback is a critical condition to maintain data value
Many systems take more than 1 month to make data available, regardless of access rights
– many impose >6 months delays
Cited reasons for delay include data cleaning and anonymization
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Administrator Schoolprincipals
Teachers Parents Students Researchers
Real Time <1 month 1-3 M 4-6 M > 6 M
Many strengths but also areas of improvement for
current information systems in OECD countries
No single model –wide variation in goals, data elements and functionalities enabled
Unique, student longitudinal identifiers are the most critical feature of effective information systems. Linkages to teacher data as well as to data from other agencies (e.g. labour market) would open more possibilities for innovative uses of data
Cover a broader range of student outcomes. Summative and subject-based indicators fall short of capturing the set of skills that students are expected to develop
Access policies remain highly restrictive. Generally open to policy makers and administrators, but not to researchers and educators
Faster feedback is needed. Many take too long to report back and make data available. Feedback delays are at odds with aim of supporting timely decision-making
More user-friendly analysis and visualisation tools needed. Compatible with tiered access policies and important to prevent that valuable data remain underused
Integration with digital educational resources and automated analysis and recommendations will be important features of next-generation systems
a typology of information systems
in education
Current features and uses suggest 4 ideal-types or approaches:
1. Reporting and research data systems
2. E-government data systems
3. School improvement data systems
4. Expert data systems
A typology of information systems in education
1. Reporting and research approach
Statistical and evaluation approach – from the traditional focus on reporting and accountability requirements
Accountability of systems and school performance cards enriched thanks to longitudinal, individual-level data
Reports seek to inform mainly policy makers and the public
In some cases, also designed to develop research capacity about educational issues
Ontario (Canada): Ontario School Information System
(OnSIS)
Examples: Board Interface reports (left) and Ontario Notable Education Trends (right)
Mexico: Sistema Integral de Resultados de las
Evaluaciones (SIRE)
2. e-Government data systems
Inspired by e-government approach promoting automated data integration across government agencies
Takes advantage of data trails generated by the use of digital ID-cards and digital signatures
Major objectives include making administrative processes more efficient (e.g. school transfer, school choice, university application, etc.) and informing resource allocation (e.g. school funds)
Great potential for linkage of education data with data from other sectors (e.g. labour market, taxation, health, etc.)
Estonia: Estonian Education Information System
(EHIS)
Korea: National Education Information System (NEIS)
3. School improvement data systems
Systems designed to support school improvement efforts by putting data in the hands of principals and teachers
Key features include customisable school reports and visualisation tools such as dashboards
Enable new « improvement routines » (data teams, enquiry teams, etc.) and digital communities of practice
Try to provide information at the individual level and with a granularity that makes data more relevant to teachers
England: RAISEonline - now replaced by the new
Analyse School Performance (ASP) system)
Portugal: Escola 360° (E-360°)
4. Expert data systems
Aim to help personalise teaching and learning and to provide real-time feedback to teachers, students and principals
Combine administrative data with process and formative assessment data from learning management systems
Learning analytics and other diagnosis techniques
Allow adjustments in ongoing instruction cycles – vs. end-of-year feedback
Advanced features: links to banks of educational resources, recommendations and networking platforms for teachers
Colorado (US) state-wide longitudinal system and
SchoolView website
some challenges
The interoperability challenge
The data ecosystem is highly fragmented: “silo” systems that cannot
communicate with each other
Legacy systems, designed for specific functions (accounts, registration, VLEs…)
Inconsistent definitions, formats, coding procedures, etc.
Interoperability: capacity to combine and use data from disparate
systems and content platforms with ease, coherence and efficiency
Technical layer: software and connectivity
Semantic layer: data models, consistent definitions and coding rules
European Interoperability Framework (EIF)
The privacy challenge
Greater data integration and increasing involvement of technology and data
service providers raise stakes for privacy protection
Potential harms: profiling and discrimination, commercial uses, etc.
Blurring distinction between personal and non-personal data: more
possibilities for re-identification
Informed consent and over-restrictive access are inefficient solutions
Need to combine data-focused and governance-focused solutions
Anonymization techniques – make re-identification more difficult
Control access and use for legitimate purposes (e.g. research)
Towards a new generation of systems: from statistical
reporting to timely and actionable feedback
Old/current data systems Next-generation systems
Privacy protected by limited access and data redaction
Risk assessment, tiered access, privacy-enabling technologies
Cross-sectional snapshots Longitudinal perspective
Data silos Interoperability
End-of-year feedback “Real time” feedback
Statistical reports Learning analytics and suggestions
Use by administrators mainlyExtended to educators, students, and researchers
Aggregate-level indicators Individual-level indicators
Benchmarking, contextual dataLocal, isolated data points
Carlos González-Sancho
OECD Directorate for Education and Skills
http://www.oecd.org/edu
Centre for Educational Research and Innovation (CERI)
http://www.oecd.org/edu/ceri
Thank you for your attention
