Date post: | 15-Apr-2017 |
Category: |
Education |
Upload: | piet-desmet |
View: | 364 times |
Download: | 1 times |
Bridging the gap between closed and open items
or how to make CALL more intelligent
Piet Desmet & Bert Wylin
Fleat VI Harvard University August 11-15, 2015
1. Item-based learning & testing environments (ILTE): definition
2. CALL, SLA & LT: different views on a “classical” ILTE
3. Beyond the closed & open items in an ILTE
4. Half-closed items
5. Half-open items
6. Supported open items
7. Challenges for ILTEs
8. Conclusion
1. Item-based learning & testing environments (ILTE): definition
1.1. Definition of an item
“A digital item asks the learner to react to a given input, leading to an output that is treated by the system”.
Typically, items are• part of a series (or stand on themselves)• structured (organized),• (minimally) metadated,• reusable,• multimedial,• stored in an item bank
1.2. “Classical” items: closed or openCLOSED OPEN
Learner output
level of freedom limited totally free
# correct answers limited to 1 or a few many
predicatibility answers maximal very limited
Output treatment
correction type automated manual
reliability high
Examples
closed: multiple choice, multiple answer, drag & drop, order, fill gaps, etc.
open: upload text file, audio or video-recording (without correction)
2. CALL, SLA & LT: Different views on “classical” ILTEs
2.1. Within CALL: tutor vs tool
Computer as a tutor (tutorial CALL):ILTEs still crucial today although need for improvement
“Many programs being produced today feature little more than visually stimulating variations on the same gap-filling exercises used 40 years ago”
(Beatty 2003: 11)
vs
Computer as a tool (multimedia, CMC, web 2.0, etc.): ILTEs less important since main focus is on CMC, social media, immersive virtual worlds, etc. allowing for communicative activities and tasks
Tutorial CAL not even on the Hype cycle for education (Gartner, 2013)
2.2. Within SLA: cognitive vs socio-cultural
Different perspectives on SLA:
cognitive perspective: cognitive processing by the learner(noticing, motivation, etc.)
socio-cultural perspective: impact of social environment of the learner (collaboration between learners, scaffolding by interlocutor, etc.)
-> ILTEs are more crucial within a cognitive framework
2.3. Within language teaching: behavioral vs communicative/task-based
° Different methods:grammar-translationdirect methodscommunicative approachtask-based language teaching (TBLT)etc.-> ILTEs are considered to be less crucial in TBLT than before (cf. “drill & kill”)
° Different focus:focus on form vs focus on meaningrule-based vs usage-basedknowledge-oriented vs skills-orientedteacher-centered vs learner-centered-> ILTs are mainly associated with the left focuses
3. Beyond the closed & open items in an ILTE
3.1. Limitations of “classical” closed items
(a) too limited freedom at the level of the learner output
(b) too limited cognitive complexity
(c) limited number of item types
(d) less suited for advanced learners
-> need for more “intelligent” CALL
3.2. Old wine in new bottles…
Till recently only technological innovation
floppy disk (DOS only)
cd-rom (Windows)
website
platforms
CMS LMS learning platform testing platform
SPOC MOOC
3.3. “Our” solution: bridging the gap between closed and open items
= pedagogical innovation
still automated correction with high reliability
BUT:
Learner output: more freedommore correct answersless predictability
www.edumatic.com
http://www.delta-associates.com/what-about-the-old-advice-dont-reinvent-the-wheel-is-it-stupid-or-smart/
4. Half-closed items
CLOSED HALF-CLOSED
Learner output
level of freedom limited more free
# correct answers limited to 1 or a few limited
predicatibility answers maximal maximal
Output treatment
correction type automated automated
reliability high high
Examples
(1) select text(2) dictation
4.1. Definition
4.2. Select text
Learner output: selection of relevant passage in a text
The locus of the points of interest is not given beforehand
-> more freedom at the level of the learner output
Mechanism behind these items:
° mark the keyword(s) in a given text (sentence or paragraph) & link/group these keywords
° define ranges for selection (ranges as such don’t influence the score)
° prepare feedback for correct and wrong keywords
Bert: TE VERVANGEN DOOR VB VOOR TAAL!
4.3. Dictation
Learner output: transcription of a (bookmarked) audio file
Learner doesn’t know what are the possible points of interestLearner can decide not to transcribe certain parts (without impacton the correction mechanism)
-> more freedom at the level of the learner output
Mechanism behind these items:
Approximate string matching
Approximate String Matching @ Edumatic
• Normalization of input (or not)• caps• interpunction• accents
• algorithm based on best match with input
I inform you to XXX the (…) tomorrow (XXX).
• 3 codes: delete, insert, substitute (error)
• Attempts model:attempt – feedback – attempt – (…) – solution model
Approximate String Matching @ Edumatic
• “Brackets” model
[[In the/Every] morning, Mary listens to the radio./Mary listens to the radio [in the/every] morning.]
• not only feedback,also show solutions based on best match with student’s input
showing non matching solutions is an option
Bert: VB van gecorrigeerd dictee toevoegen
5. Half-open items
HALF-CLOSED HALF-OPEN
Learner output
level of freedom more free more free
# correct answers limited to 1 or a few many
predicatibility answers maximal limited(but feasable and progressive build up)
Output treatment
correction type automated automated
reliability high average to high
Examples(1) translate(2) reformulate(3) correct
5.1. Definition
5.2. Translate
xxx = substitute
(…) = insert
(xxx) = delete
5.2. Translate 2.0
Correction on the letter level
BE-ODL 21 maart 2006
5.3. Reformulate/correct
6. Open supported
HALF-OPEN OPEN SUPPORTEDLearner output
level of freedom more free free
# correct answers many many
predicatibility answers limited even more limited
Output treatment
correction type automated automated
reliability average to high average to high
6.1. Definition
6.2. Mechanism
• open question with free learner input
• with due date
• generation of feedback on the basis of:
model answerkeyword matching
• white list (+ score)• and• if• if then
• black list (0 or – score)• negations (and range)
4 functions of supported open item type:
1) Creation of open questionwith model answer, black list, white list, elaborated feedback, etc.
2) Publication of this item fix due date, select student groups, follow-up received
answers, etc.3) Half-automated correction of the answers
correction proposal on the basis of the available infomanual correction of scores and adaptation ofblack list & white list (-> update of automatic scores)
4) Generation of feedback report individualised feedback, fix scores, add personal commentsnotify all users by automatically generated mail
Item Input: create New item
Item Input: create New item
Add original text in “logical units”
(paragraph or
sentences)
Add instruction
Students make translations•Use quick codes to have alternative correct solutions
• Eg. [on passe/on passera/on fera/on effectuera/sera passée/sera prise/l'infirmière glissera]
•Decide about keyphrases•Add scores per keyphrase•Add feedback per keyphrase
• including error specific feedback
Student/candidate response
With or without Correction
button (practice vs. exam)
Student/candidate types
answer
Students make translations•While correcting student input,
• Add more options• Update all existing corrections
constantly
•See the effect of the updates in new student input:
• less and less corrections to make• more and more keyphrases
recognized (both correct and wrong answers)
Item Input: create New itemAdd
translation keywords
and keyphrases
Item Input: create New item
Option: set options for
spell checker
Option: provide model answer(for feedback)
Update translations/scores
Update, add, delete
translations
System asks to apply changes in translations to all students
Final reportingBased on updated
translations and scores
See individual and group
results
Bert: TE VERVANGEN DOOR VB VOOR TAAL!
• Use of supported open exercises in three steps
• Step 1 : try outas a marking and feedbacktool (aid) used by teaching staff-> human verification and improvement of the black & white list is necessary
• Step 2 : learning result of scenario 1 can be used as an exercise with full automatic corrective and elaborated feedback (with human intervention!)
-> human verificationand e-mail feedback
• Step 3 : exam simulation results of scenario 2 can be used as an exercise with full immediate automatic corrective and elaborated feedback (without human intervention!)
•!Supported open exercises are not limited to languages
•Excellent experiences in•Law faculty•Medical faculty
7.1. Adaptivity-> frontend: e.g. adaptive item sequencing
adaptive feedback
7.2. Gamification-> frontend: e.g. Badges & rankings
Collaboration & competition7.3. Flexible delivery mode
-> frontend e.g. Integration in App or digital textbookIntegration in skills oriented learning environment
7.4. Output correction through NLP-> from backend to frontend: e.g. parsing half-open input
7.5. Analysis of tracking & logging data-> from backend to frontend: e.g. reporting
7. Challenges for ILTEs
http://ingvihrannar.com/wp-content/uploads/2014/02/testing_cartoon.jpg
7.1. Adaptivity
4D-model of adaptive instruction
Vandewaetere, Desmet & Clarebout 2011 / Vandewaetere & Clarebout, 2012
Cognition (e.g. prior knowledge)
Affect (e.g. motivation)
Behavior (e.g. need for help)
What elements in the environment to adapt?
Adapt during interaction, between interactions, prior to interaction?
Who’s in control?Learner vs. instructor decides what/when/how to adapt?Or both?
http://www.slideshare.net/piet_desmet/2015-0522-presentatiecalicodesmet-vandewaetere-def
7.2. Gamification
MindSnacks ‘Swell’
Using gameplay mechanics for non-game applications
- Challenges embedded in a compelling story
- Various layers or levels & character upgrades
- Rewards (scores & badges)
- Social interacton & peer motivation through competition
http://www.playwarestudios.com/wp-content/uploads/2013/07/gbl-cartoon.jpg
7.3. Flexible delivery mode
“Classical” delivery mode
Items(in Activities)
from: Horton, William, E-Learning by Design, Wiley, 2011
(a) From a technological point of view
ILTE as a
- smartphone app
- daily small interactive e-mail or sms
- micro-series of items, embedded in a digital textbook
- etc.
More flexibility
(b) From a pedagogical point of view
“Skinning” of item types to be integrated in a skills oriented environment
e.g. multimedia learning environment focusing on audio-visual comprehension
e.g. situational judgment test / inbox exercises
www.franel.eu Nedbox
7.4. Output correction through NLP or statistical methods
NLP ASM
- by definition language dependent- high R&D effort
+ by definition language independent+ lower R&D effort
- unequal availability and quality of existing algorithms and tools- technologies not easily transferable to new tools/environments- slow
+ high availability of existing ASM algorithms+ easily reusable algorithms+ higher speed
+ better granularity (fineness with which input can be analyzed)
- highly depending on teacher’s input (number of correct answers predicted by teacher)
+ language specific intelligent feedback generation by the algorithm (cf. E-Tutor T. Heift)
- no automatic language specific feedback generation
NLP: lemmatisation -> tagging -> parsing (-> semantic analysis?)
Statistical methods: combine advantages of ASM & NLP!
Statistical error detection:training a classifier based on a corpus of corrected utterances with feedback
(cf. PhD Ruben Lagatie)
7.5. Analysis of tracking & logging data
From manually entering data to online massive storageFrom self-reporting data to behavioral dataFrom single measurements to longitudinal measurementsFrom inaccessible to everywhere
From big data to rich data…
Not the data, but the views on the data make it interesting…
For the user: - detailed reporting (from generic to specific!)
- advice on next steps
For the teacher: - reporting at individual and group level
- item analysis
For the user: detailed reporting (from generic to specific reports)
For the user: advise on next steps
For the teacher: reporting at group level
Bert: illustratie invoegen!
For the teacher/content author: item analysis
http://ayende.com/blog/2421/when-does-it-make-sense-to-reinvent-the-wheel
8. Conclusion
CLOSED HALF-CLOSED HALF-OPEN OPEN SUPPORTED
OPEN
Learner output
level of freedom
limited more free more free free totally free
# correct answers
limited to 1 or a few
limited many many many
predicatibility answers
maximal maximal limited very limited
very limited
Output treatment
correction type
automated automated automated automated manual
reliability high high average to high
average to high
More info
Piet Desmet Bert Wylin [email protected] [email protected]
www.linkedin.com/in/pietdesmet www.linkedin.com/in/bertwylin
@PietDesmet
ITEC www.kuleuven.be/itec