The Use of ICT for the Assessment of Key Competences
Irish Presidency Conference
Better Assessment and Evaluation to Improve Teaching and Learning
19-20 March 2013
Christine Redecker Yves Punie
IPTS: Part of Joint Research Centre of
the EC: 7 Research Institutes across
Europe
Mission: “to provide customer-driven
support to the EU policy-making
process by developing science-based
responses to policy challenges that have
both a socio-economic as well as a
scientific/technological dimension”
IPTS- Who we are
Future Learning Spaces in 2020
2006 2007 2008 2009 2010 2011 2012 2013
Learning 2.0 in formal E&T
Learning 2.0 in informal settings
Creativity & Innovation in
schools in EU
eLearning in the EU10 MS
- Digital Competence
- Active Ageing & Learning
Tell-Net Teacher Networking
Review ICT
Impact
IPTS Studies on ICT for Learning
The Future of Learning
ICT for the Assessment of Key Competences
Digital Competence
Mainstreaming Creative Classrooms
Open Education in Europe
1to1 Learning initiatives in Europe
Mapping Technologies for Learning
DC Follow up
Where we stand
Multiple Choice
Short Answer
Generation 1
Automated administration
and/or scoring of
conventional tests
Adaptive Testing
Generation 2 Generation 4
Intelligent Tutors
Automated Feedback
Personalised
feedback and
tutoring
Learning 2.0
Generation 3
Continuous
integrated
assessment
Data Mining & Analysis
Behavioural tracking
Generation Re-invention Transformative strategy: use technology to
change test formats
Peer assessment
Collaborative multimedia learning
environments
Online collaboration Virtual laboratories
Games
Simulations
Virtual Worlds
Replication of complex
real life situations within a
test
Bu
nd
erso
n, In
ou
ye
& O
lsen
(1
98
9).
Th
e fo
ur
gen
erat
ion
s o
f co
mp
ute
rize
d e
du
cati
on
al
mea
sure
men
t. I
n R
. L
. L
inn
(E
d.)
, E
du
cati
on
al m
easu
rem
ent
(Th
ird
ed
., p
p. 3
67
-40
7).
New
Yo
rk:
Mac
mil
lan
Ben
net
t, R
. E
. (2
01
0).
Tec
hn
olo
gy f
or
Lar
ge-S
cale
Ass
essm
ent.
In
P. P
eter
son
, E
. B
aker
& B
.
McG
aw (
Ed
s.),
In
tern
ati
on
al E
ncy
clo
ped
ia o
f E
du
cati
on
(3
rd e
d.,
Vo
l. 8
, p
p. 4
8-5
5).
fo
rd:
Els
evie
r.
1990 1995 2000 2005 2010 2015 2020 2025 Efficient testing Personalised learning
Learning Analytics
Computer-Based Assessment (CBA) Technology-enhanced learning
Current use of ICT in assessment in schools
Central recommendations on the use of ICT in pupil assessment in compulsory education in primary and general
secondary education (ISCED 1, 2, 3), 2009/10
Source: Eurydice (2011). Key Data on Learning and Innovation through ICT at School in Europe 2011, Education, Audiovisual and Culture
Executive Agency.
There is scope for improvement
1st Generation CBA: Problem-Solving
http://www.worldclassarena.org/v5/flash/09_y
ear_old/backtofront.swf
World Class Arena Sample Test: Problem
Solving (9 years)
Examples
http://www.worldclassarena.org/v5/flash/13
_year_old/pyramids.swf
Problem Solving 13 years
Generation Re-Invention: Problem-Solving
Examples
1st Generation CBA: Situational Judgement
EPSO sample situational judgement test questions; http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm
Examples
Generation Re-Invention: e-tray exercise
EP
SO
sam
ple
ass
essm
ent
centr
e e-t
ray e
xer
cise
htt
p:/
/euro
pa.
eu/e
pso
/dis
cover
/pre
pa_
test
/sam
ple
_te
st/i
nd
ex_
en.h
tm
Examples
Future eAssessment Strategies
Multiple Choice
Short Answer
Generation 1
Automated administration
and/or scoring of
conventional tests
Adaptive Testing
Generation 2
Generation 4
Intelligent Tutors
Automated Feedback Personalised
feedback and
tutoring
Learning 2.0
Generation 3
Continuous
integrated
assessment
Data Mining & Analysis
Behavioural tracking
Generation Re-invention Transformative strategy: use technology to
change test formats
Computer-Based Assessment (CBA) Technology-enhanced learning
Peer assessment
Collaborative multimedia learning
environments
Online collaboration Virtual laboratories
Games
Simulations
Virtual Worlds Replication of complex
real life situations within a
test
Bu
nd
erso
n, In
ou
ye
& O
lsen
(1
98
9).
Th
e fo
ur
gen
erat
ion
s o
f co
mp
ute
rize
d e
du
cati
on
al
mea
sure
men
t. I
n R
. L
. L
inn
(E
d.)
, E
du
cati
on
al m
easu
rem
ent
(Th
ird
ed
., p
p. 3
67
-40
7).
New
Yo
rk:
Mac
mil
lan
Ben
net
t, R
. E
. (2
01
0).
Tec
hn
olo
gy f
or
Lar
ge-S
cale
Ass
essm
ent.
In
P. P
eter
son
, E
. B
aker
& B
.
McG
aw (
Ed
s.),
In
tern
ati
on
al E
ncy
clo
ped
ia o
f E
du
cati
on
(3
rd e
d.,
Vo
l. 8
, p
p. 4
8-5
5).
fo
rd:
Els
evie
r.
Learning Analytics
1990 1995 2000 2005 2010 2015 2020 2025 Efficient testing Personalised learning
Towards Generation 3: Datamining
Signals http://www.itap.purdue.edu/tlt/sig
nals/signals_final/index.htm
Similar: GPS at Northern Arizona
University
http://resourceconnect.nau.edu/gr
ade-performance-status.asp
Examples
Towards Generation 3: Behaviour Tracking
Example: SNAPP
SNAPP is a software tool that
allows users to visualize the
network of interactions
resulting from discussion
forum posts and replies.
Teachers can rapidly identify
patterns of user behaviour.
SNAPP can extract all user
interactions from various
commercial and open source
learning management
systems, such as BlackBoard
and Moodle.
Examples
Towards Gen 4: Intelligent Tutoring Systems
GnuTutor (http://gnututor.com/)
OpenSource version of AutoTutor https://www.youtube.com/watch?v=k6HDEEnYuIA&feature=player_embedded
AutoTutor https://www.youtube.com/watch?v=aPcoZPjL2G8
Examples
Generation 1 Generation 2
Generation 4
Personalised
feedback and
tutoring
Learning 2.0
Generation 3
Continuous
integrated
assessment
Generation Re-invention
Computer-Based Assessment (CBA) Technology-enhanced learning
Collaborative multimedia learning
environments
Bu
nd
erso
n, In
ou
ye
& O
lsen
(1
98
9).
Th
e fo
ur
gen
erat
ion
s o
f co
mp
ute
rize
d e
du
cati
on
al
mea
sure
men
t. I
n R
. L
. L
inn
(E
d.)
, E
du
cati
on
al m
easu
rem
ent
(Th
ird
ed
., p
p. 3
67
-40
7).
New
Yo
rk:
Mac
mil
lan
Ben
net
t, R
. E
. (2
01
0).
Tec
hn
olo
gy f
or
Lar
ge-S
cale
Ass
essm
ent.
In
P. P
eter
son
, E
. B
aker
& B
.
McG
aw (
Ed
s.),
In
tern
ati
on
al E
ncy
clo
ped
ia o
f E
du
cati
on
(3
rd e
d.,
Vo
l. 8
, p
p. 4
8-5
5).
fo
rd:
Els
evie
r.
- mainstream use for high stakes testing
- further development of innovative test formats
- increased ease of use at school level
- trend towards automated scoring
→ cultural change:
increased acceptance of CBA
Automated administration
and/or scoring of
conventional tests
Adaptive Testing
Replication of complex real life
situations within a test
- Multimedia collaborative learning
environments support key competence
development.
- Peer and portfolio assessment evolve
naturally.
-Data mining and analysis will make
integrated assessment and tutoring possible
in the future.
We are in a transitional period: Moving from the “Testing
Paradigm” to the (future) “Tutoring Paradigm” in which explicit testing could become obsolete
Technological trends
1990 1995 2000 2005 2010 2015 2020 2025 Efficient testing Personalised learning
Key Questions
1. Which ICT tools, formats, environments, devices etc. are most suitable for the assessment of which Key Competences?
2. How can ICT be used to support competence-based assessment strategies that adequately capture the skills and attitudes dimensions of all eight Key Competences and their transversal nature?
Assessing Key Competences
F = Formative Assessment
D = Diagnostic Assessment
S = Summative Assessment
1 Communication in the mother tongue DS FD FS
2 Communication in foreign languages DS FS
3a Mathematical competence DS FD FDS FDS
3b Basic competences in science and technology FS FDS
4 Digital competence FDS
5 Learning to learn FS FDS
6 Social and civic competences FS
7 Sense of initiative and entrepreneurship FS
8 Cultural awareness and expression FS FS
CB
A
ePo
rtfo
lio
s
Pee
r
Ass
essm
ent
Sel
f
Ass
essm
ent
Qu
izze
s /
sim
ple
g
am
es
Vir
tua
l w
orl
ds
ga
mes
Sim
ula
tion
s
Inte
llig
ent
Tu
tors
etc
.
Focus Areas
Mother Tongue Literacy
1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)
Matching spoken and written language
http://www.familylearning.org.uk/sight_word_games.html
Foreign language learning
2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)
htt
p:/
/ww
w.n
ickjr
.de/
gam
es/1
07
-sw
iper
s-bu
chst
abie
r-buch
Mathematical games 3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)
Mathematical games using simulation for secondary school education
Source: http://hotmath.com/games.html.
5 Learning to Learn CBA (F) Peer Assessment (FS) Self Assessment (FS)
http://spark.uts.edu.au/
Example:
SparkPlus
Peer Assessment for Collaborative Work
• http://www.youtube.com/watch?v=Tp8v2UBbmN4
www.moodle.org
Moodle is a free web
application that educators can
use to create effective online
learning sites.
Moodle Statistics Registered sites: 77,213 Countries: 229 Courses: 7,116,466 Users: 66,349,750 Teachers: 1,292,731 Enrolments: 56,306,274 Forum posts: 118,413,272 Resources: 63,891,942 Quiz questions: 147,057,434
Peer Assessment using moodle
ePortfolios
1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)
2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)
8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS) http
://sites.google.co
m/site/ep
ortfo
lioap
ps/o
verv
iew/m
od
el-k-1
2
ePortfolios
ePortfolios
1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)
2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)
8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)
http
://sites.google.co
m/site/ep
ortfo
lioap
ps/o
verv
iew/lev
els
Level 1 - ePortfolio as Storage
Collection:
Creating the Digital Archive (regularly – weekly/monthly)
A Focus on Contents & Digital Conversion
Level 2 - ePortfolio as Workspace
Collection + Reflection:
Immediate Reflection on Learning & Artifacts in Collection (regularly)
A Focus on Process & Documentation of Learning
Level 3 - ePortfolio as Showcase
Selection/Reflection/Direction (each semester? End of year?)
A Focus on Product & Documentation of Achievement
→ A (formative/summative) assessment approach on the basis of artefacts
→ Focus on production and reflection
A game approach to arts
8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)
http://pheon.org; http://pheon.org/intro.html;
http://americanart.si.edu/multimedia/games/pheon.cfm.
Pheon is a multimedia scavenger hunt that tests players’ ingenuity through
exploring the collection of the Smithsonian American Art Museum, making
objects, and texting through the galleries
ePortfolios meet Virtual reality
Science: Molecular Workbench
3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)
http://mw2.concord.org/public/part2/diffat/page3.cml
3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)
Simulations for mathematics
Sample task from US future New State Tests
(2014/15)
http://www.k12center.org/rsc/pdf/a-sea-change-in-assessment-letter-size.pdf
Cognitive Tutor (Carnegie)
3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)
http://www.youtube.com/watch?v=RR4HPrGlUz
8&feature=related
Science: Quest Atlantis
3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)
Example: Quest Alantis Quest Atlantis (QA) is an international learning and teaching project that uses a 3D multi-user environment to immerse children, ages 9-16, in educational tasks. QA combines strategies used in the commercial gaming environment with lessons from educational research on learning and motivation.
Citizenship 6 Social and civic competences CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)
Educational game, designed to help teach concepts in citizenship, geography, and media.
Developed by Serious Games International, it has detailed lesson plans and assignments for students.
Example: Child labour in
Bangladesh:
http://www.youtube.com/wat
ch?v=mf9Ju86IVYw
Global Conflicts http://www.globalconflicts.eu/
Digital Competence
4 Digital competence CBA (S) Simulations (FDS)
Internet and Communication Core Certification (IC3) exam sample question
http://www.certiport.com/Portal/common/htmllibrary/Sales-Video-Demos/demo_ic3.html
→ Focus on knowledge
CBA for Digital Competence
Digital Competence: IC3
4 Digital competence CBA (S) Simulations (FDS)
Inte
rnet
and C
om
munic
atio
n C
ore
Cer
tifi
cati
on (
IC3)
exam
sam
ple
ques
tion
htt
p:/
/ww
w.c
erti
port
.com
/Port
al/c
om
mon/h
tmll
ibra
ry/S
ales
-
Vid
eo-D
emos/
dem
o_ic
3.h
tml
→ Focus on operational skills
CBA for Digital Competence
Entrepreneurship Questionnaire
7 Sense of initiative and entrepreneurship CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)
? https://www.careeringahead.
com.au/survey/Entrepreneurs
hip/Generic/questions.asp
→ Focus on personality/disposition as a static fact
Entrepreneurship Questionnaire
Conclusions
ICT offers powerful, engaging assessment formats, in particular:
CBA ePortfolios Self / Peer
Assessment Quizzes /
simple games Virtual worlds/ games Simulations
Basic literacy
(foreign/mother tongue)
Basic mathematical
competence
Learning
to Learn
Creative writing
(foreign/mother tongue)
Cultural awareness and
expression
Scientific competence
Digital
competence
Social and civic
competences
Educational software
Mathematical
competence
CBA lends itself to simplified, knowledge-based assessment
→ Pedagogy comes first! Careful selection of question formats
ICT Assessment approaches are underdeveloped for:
Learning to Learn Sense of initiative and entrepreneurship Social and civic competences
Caveat:
The Paradigm Shift
Technology
1990 1995 2000 2005 2010 2015 2020 2025
Focus on: Efficient testing Focus on: Personalised learning
Era of Computer-Based Assessment (CBA) Era of Technology-enhanced learning
Pedagogy Efficient
Testing
Validity of
Scores
Quick
Feedback
Authentic
Contexts
CBA
(Gen.1)
Adaptive
Tests
(Gen.2)
Automatic
Scoring
Gen. Re-
Invention
Learning
2.0
Immersive
Learning
Learning
Analytics
Intelligent
Tutors
? ? ? ?
→ Primacy of Pedagogy:
Pedagogical guidelines for intergrated assessment
• Thank you very much for your attention!
• Christine Redecker
• Yves Punie
• [email protected] • [email protected]
http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=5719
further information ……
http://is.jrc.ec.europa.eu
Newsletters: http://is.jrc.es/pages/newsletter.html
Publications: http://is.jrc.ec.europa.eu/pages/Publications.html
Contact: [email protected]