Intelligent systems and learning centred design

transcript

LEARNER CENTRED DESIGN

INTELLIGENT SYSTEMSand

Rosella Gennarihttp://www.inf.unibz.it/~gennari

Distinguished Speakers Oxford Women in CS

Settingtechnology enhanced learning

ClimaxTERENCE case study

Resolutionreflections

Story outline

TECHNOLOGY ENHANCED LEARNING

Technology Enhanced Learning (TEL) is the usage of technology for supporting a learning experience

Herby we take a narrow view: intelligent TEL =

Artificial Intelligence (AI) technology based products for supporting a learning experience

TEL 4 LEARNING EXPERIENCE

How can we design technological products that supports their users’

learning experience?

LET'S SEE EDUCATORS' VIEWPOINT...

Maria Montessori (1870-1952)

Paraphrasing her words, adequate tasks that come in a prepared environment, designed on top of the learner characteristics can effectively support the learner’s learning

was the first Italian woman physician and educator, best known for Montessori pedagogy

ousability of technology learning products

opedagogical effectiveness of technology learning products

TEL 4 LEARNING EXPERIENCE

Adequate tasks that come in a prepared environment designed on top of the learner characteristics can effectively support learning

HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL

Based on UCD process diagram (© Tom Wellings)

requirement specification

designevaluation

models + prototypes

intermediate product

final product

HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL

USABILITY + P. EFFECTIVENESS

Story outline

TERENCE DESIGN

Based on UCD process diagram (© Tom Wellings)

requirement specification

designevaluation

models + prototypes

intermediate products

final products

TERENCE was an FP7 TEL project blending user centred and evidence based design

USABILITY + P. EFFECTIVENESS

THE PROBLEM

‣ TERENCE developed an adaptive learning system (ALS) that, via a learner GUI, recommends poor comprehenders

- its learning material, i.e., books of stories and games

- its learning tasks, i.e., reading and playing

‣ so as to stimulate their reading comprehension

‣More than 10% of primary school children, older than 8, are diagnosed with deep text comprehension problems

‣ They are referred to as poor comprehenders

THE TERENCE WORLD

a d e q u a t e b o o k o f s t o r i e s

s i g n i n

a d e q u a t e s m a r t g a m e s

r e w a r d

TERENCE INTELLIGENT TEL PRODUCTS

ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF

VisualisationManagerillustrations

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule

game generation

adaptation to learners

Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.

TERENCE DESIGN

DATA GATHERING METHODS

‣Data for designing the learning material and tasks were from

- contextual inquiries with

‣ IT & UK diagnosis

‣ as well as IT & USA evidence-based medicine therapy experts

- field studies with educators and primary school children

CHARACTERI

Persona Name: Carol

Age: 8

Classroom: year 4

RC levels: low reading levels

Rural/Urban: urban

Deaf/hearing: deaf

First Language: Italian Sign Language

Cochlear Implantation (if deaf): yes

Degree of hearing loss (if deaf): profound

Motor skills (if deaf): average

Summary of the class represented by this persona

A younger deaf girl who is very enthusiastic about using new technology (such as iPhone and IPad) and who adores her Nintendo DS. Her reading RC levels are very low, but she reads together with her parents to learn new words and spelling. She also likes to do many other things such as drawing, taking care of her pets and going to the park.

Quote “I really love Mario and Luigi. And I would love to have an iPhone and an IPad, like my dad.”

Personality Open

Role in classroom Active

Role out of the classroom

Active

Console/Technology Carol and her sister watch TV after school. They like Tom & Jerry, Ben 10, Hello Kitty and Mickey Mouse Clubhouse.

Carol sometimes uses the computer, but only to play minilab games. Her computer is in her bedroom, but her parents don’t allow her to use it all of the time. She can use it only one hour per day. Carol’s dad has an iPhone and an IPad, and Carol would really like to use those as well, but her dad tells her she is a bit too young. Carol likes watching him with his IPad and iPhone though.

Carol doesn’t use a mobile phone.

Carol plays games on the computer and on her Nintendo DS. She plays by herself. She likes the mini-clip games on the computer, and Mario Kart and brain training games on her DS.

She likes games with non-photorealistic human avatars, and prefers fantasy avatars to animal avatars.

Socio-Cultural Level of his/her own family

Medium

School performance Carol has sever reading problems. In her class she is below average in all activities but drawing, where she feels she can truly express her intimate feelings.

Homework After school, Carol does her homework together with her mum.

Outdoors Activities Carol often goes to the park with her mum.

Indoors Activities Games on the DS Carol reads sometimes. Her mum and dad help her reading in the evening. She likes some of the stories they read together, but mostly, she wants to read because she has to learn new words and spelling. Carol likes drawing and taking care of her pets. Her mum often plays with her.

Home activities Carol also likes to help her mum in the kitchen or in the garden.

Sport activities Carol practices no specific sport.

SMART GAME REQUIREMENTSWhat for Description

Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.

Scheduling of reading and playing

1st silent reading; 2nd playing smart games; 3rd playing relaxing games

Constraints on actions Learners should get faster, hence a game has a maximal resolution time

Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)

Representation Production can be impaired hence promote resolution via visual representation and reasoning

TERENCE DESIGN

who is the actor of … ? what does (a main) character do?

when does … happen in relation to a central event?

why does the central event happen?

points for each smart

coins for all smartunlocked if read+play

visual feedback

Instructions Questions Motivational Interaction

Choices Choices for learner Fixed event

Solutions Choices that are either correct (c) or wrong (w)

Feedback Interaction Consistency Explanatory Solution

Smart points Proportional to the learner’s ability in the game level

Relaxing points

Constant

Avatar Happy/sad states

Time solution constant interaction constant

Rules States of the system, actions of the learner, constraints

What for Description

Constraints on actions

Learners should get faster, hence a game has a maximal resolution time

Progress and feedback

Monitor and give learners (1) idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)

TERENCE DESIGN

EXAMPLE METHODS IN TERENCE

APPROACH WITH WHOM EXAMPLE METHODS WHEN

analyticalHMI experts or domain experts

heuristic evaluationformative, summativeexpert evaluation

cognitive walk-through

small-scale learnersobservations

formativethink aloud

large-scale learners field studies summative

APPROACH WITH WHOM EXAMPLE METHODS HOW

G1. interfaces follow general design guidelines

G2. interfaces support the user’s next step to achieve a task

G3. interfaces provide users with timely feedback

Instructions are not under focus and cannot be easily read

Game question and possible resolutions should be proximally close

Evaluation of interfaces

TERENCE DESIGN

problem: 256 stories,

each with ~12 games

Smart game design

how can we automatise the

development of smart games via AI (and, hopefully, be efficient)?

Smart game design

enriched annotations

annotations

ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule

Semi-automated generation

annotations

Semi-automated generationALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule

image image image

annotations

template visual

image image image

annotations

Semi-automated generationALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule

text + visual

AUTOM. MANUAL AUTOM.

TERENCE DESIGN

LARGE-SCALE STUDY DESIGN

Common design of the intervention with TERENCE:

‣ how: pretest/posttest design, with experimental and control groups

‣ hypothesis: TERENCE improves reading comprehension measured with standardized text comprehension tests

ControlExperimental

A 3-PHASE INTERVENTION

‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE

‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs

-and requires (1) reading

-and requires (1) reading (2) playing smart

-and requires (1) reading (3) playing relaxing(2) playing smart

‣ Post-test (pedagogical only) for re-assessing text comprehension

-and requires (1) reading (3) playing relaxing(2) playing smart

The experimental group is of 344 learners:

‣Avezzano: 270 learners:

- 7-9 years old: 118

- 9-11 years old: 152

‣ Pescina: 74 learners:

- 7-9 years old: 37

- 9-11 years old: 37

‣ They were tested (January-February), stimulated (March-May), and re-tested (may-June)

EXPERIMENTAL GROUP IN IT

Pre-post performances for text comprehension (dependent variable) were as follows:

‣ Pescina:

- pre: 14 poor comprehenders (20.59%)

- post: 6 poor comprehenders (8.82%)

‣ Avezzano:

- pre: 15 poor comprehenders (5.95%)

- post: 2 poor comprehenders (0.79%)

MAIN RESULTS IN IT

PrePescina Avezzano

20,59%

‣ Wilcoxon signed-rank test supports that differences are statistically significant

- Pescina: z=-4.904, p<0.0001

- Avezzano: z=-2.266, p=0.0234

APPROACH WITH WHOM EXAMPLE METHODS HOW

EXPERT EVALUATION

Experts of pedagogy: 1 coordinator; 9 evaluators

Sophie'comes'down'the'steps

He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers.

revise selection of solutions

revise selection of central event

How-to:

1. each pair of evaluators read a story, and edited its games

2. the coordinator revised their work

3. a pair of evaluators was blindly assigned revised games, and another the manually created games

Main edit tasks:

(1) creation of missing games (~recall)

(2) revision of games (~precision)

From D4.2 and D4.3 technical annex

overall assessment of generation

text + visual

revision of Automated Reasoning (AR) selection of central events and solutions

revision of Natural Language Processing (NLP) of text

text text text

Edit tasks in details

TERENCE DESIGN

overall generation

text + visual

AR selection of central events and solutions

revision of NLP text

text text text

Analyses of evaluation results

AR selection of central events for games:

>Results: only in 15 out of 250 cases (6%), it was necessary to select a different central event than the automatically generated one

>Implications for AR: none picked up

Automated part evaluation-based re-design

AR selection of plausible solutions:

>Results: out of 140 changes of selection of solutions, the majority was for wrong solutions

- generate a wrong solution from correct one by changing participants, e.g.,

<correct_sentence id="2">The man ran and fell on the ground.

</correct_sentence>

<wrong_sentence id="2wh1">Peter ran and fell on the ground.

</wrong_sentence>

>Implications for WP4: new heuristics for wrong plausible solutions in the last part of Y3,

Overall generation: development times:

>Results for revision time:

- 12’6” per game instance:

↑ 12’8” for time games

↓ 10’6” for who games

>Results for creation time:

- avg. 23” per game instance

storytext + visual

>Implications for AR: the semi-automated development process seems to be promising for optimising development times

Game over1st 2nd 3

Sep. 2011 December 2012 September 2013

Sophie'comes'down'the'steps

He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers.

revise selection of solutions

revise selection of central event

Requirements+for Description

Dif$iculty*levels Macro*levels*for*learners:4*entry:*character*games;4*intermediate:*time*games;4*top:*causality*games.*

Scheduling*of*reading*and*playing

1st*silent*reading;* 2nd* playing* smart* games;*3rd*playing*relaxing*games

Constraints*on*actions

Learners* should* get* faster,* hence* a* game* has* a* maximal*resolution+time

Progress*and*feedback

Monitor* and* give* learners* (1)* idea* of* progress,* (2)*explanatory*feedback,*(3)*recall*their*attention*and*solicit+them*to*give*a*resolution*(in*time)

Representation Production*can*be* impaired* hence*promote* resolution*via*visual*representation+and+reasoning

Instruc(ons Ques%onsQues%ons Mo%va%onalMo%va%onalMo%va%onalMo%va%onal Interac%onInterac%on

Choices Choices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learner 3Fixed3event3Fixed3event3Fixed3event

Solu(ons Choices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%ons

Feedback Interac%on Consistency3(c/w)Consistency3(c/w)Consistency3(c/w) ExplanatoryExplanatoryExplanatory Solu%on

Smart6points Propor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3level

Relaxing6points ConstantConstantConstantConstantConstantConstantConstantConstant

Avatar Happy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3states

Time solu%on3constantsolu%on3constantsolu%on3constant interac%on3constantinterac%on3constantinterac%on3constantinterac%on3constantinterac%on3constant

Rules States3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraints

data structures

NLP+ AR1 for stories

AR1 for txt games

frameworkAR2 + NLP1 for txt games

AR2 for stories AR3 + NLP2 for txt games

requirements

Story outline

Till 2007

0 4 8 12 16

Response: work areas Amsterdam U. and CWI

FBK-irstFree U. of Bolzano

From 2007

0 4 8 12 16

Response: work areas

Free U. of Bolzano

Possible explanation?

co-designgamification cooperative learningHOW

WHY engagement design together inclusion

childrendesigners teachersWHO

GACOCO

treestree puzzle

Gamification of protocol (tasks, subtasks and types of feedback)

Gamification (competition for cooperation)MISSIONS

CHALLENGES

REWARDSwell

Acknowledgments to

TERENCE colleagues and schools

Current colleagues and schools

DIARY FOR PRESENTTHE TERENCE BOOK

Story outline