Post on 23-Feb-2016
description
transcript
chai::Computer human adapted interaction research group
Interfaces for Learning Data Visualizations
Judy KayCHAI: Computer Human Adapted Interaction Research Group
School of Information Technologies, University of SydneyPresident of the AIED Society (2009-11), Programme co-Chair ITS2010, General Chair AIED2011
Advisory Board User Modeling Programme co-Chair Pervasive 2012, Chair of the Joint Ubicomp and Pervasive Steering Committee
chai::Computer human adapted interaction research group
About me
• Inventing future technology to tackle important problems, notably in learning
• Personalisation• Personal data and its management• Putting people in control• Open Learner Models (OLMs)• Metacognition and OLMs• Interactive surfaces… walls, tables…
chai::Computer human adapted interaction research group
Learning analytics as a form of Learner/User Modelling
with interfaces
chai::Computer human adapted interaction research group
How to create interfaces for LA?
• User-centred approaches– Stake-holders– Mental models– The problem?
• Core tools and principles• Case studies
– Institution– Class– Individual learner
chai::Computer human adapted interaction research group
Interfaces… visualisations
chai::Computer human adapted interaction research group
Why visualisations?
Fekete, J. D., Van Wijk, J. J., Stasko, J. T., & North, C. (2008). The value of information visualization. In Information Visualization (pp. 1-18).
Springer Berlin Heidelberg.
“…easy and fast to see that there is no red circle, or to evaluate the relative quantity of red and blue circles. Color is one type of feature that can be processed preattentively, but only for some tasks and within some limits. [eg] if there were more than seven colors …, answering the question could not be done with preattentive processingand would require sequential scanning, a much longer process.
chai::Computer human adapted interaction research group
But how to create the right visualisations?
Are there simple rules?Simple principles?
Simple and constant solutions?
chai::Computer human adapted interaction research group
Principle: individual data takes on more meaning….
When comparisons are supported:• Others• Temporal• Contextual
chai::Computer human adapted interaction research group
Patina: Dynamic Heatmaps for Visualizing Application Usage (CHI2013) Justin Matejka, Tovi Grossman, and George Fitzmaurice
This user’s footprints
Overall population footprints
chai::Computer human adapted interaction research group
Extended Case Study
Concrete example of my work to underpin the activities
chai::Computer human adapted interaction research group
Defining features
• The problem:– Group work is hard but it is important– Group work in learning context has many
problems that cause great anguish, inefficiency• Target stakeholders:
– Learner as individual– Team leaders (manager, tracker)– Facilitators (tutor, lecturer)
chai::Computer human adapted interaction research group
Trac: Tool supporting long term group work
Used by team members, facilitators, teachers, some clients
chai::Computer human adapted interaction research group
TRAC• open source tool for supporting software development projects
Wiki page editorTicket ManagerSVN source repository
• Not a learning system but used in a learning context.
chai::Computer human adapted interaction research group
Huge amounts of data about the group members and their
interactions
chai::Computer human adapted interaction research group
Narcissus
Upton, K., and J. Kay. (2009) Narcissus: interactive activity mirror for small groups. In UMAP09, User Modeling, Adaptation and Personalisation, Springer-Verlag, 54-65
chai::Computer human adapted interaction research group19
Integrated of mirror tool Narcissus
tab
chai::Computer human adapted interaction research group
ITS2008Lifelong learning, learner
models and sugmented cognition
Lifelong modelling – mirrors and mining
chai::Computer human adapted interaction research group21
Header –Group view Display for
one user
Time – activity on that day is shown for each user, on each medium
chai::Computer human adapted interaction research group22
Wiki contribitions
svn contribitions
ticket contribitions
chai::Computer human adapted interaction research group23
Click on cell …
…to see details
chai::Computer human adapted interaction research group24
Explainsscoring
chai::Computer human adapted interaction research group25
Individual summary
Group average
26
Click on ticket activity for a day
Associated details
Click on ticket label
27
Details of that ticket
ITS2008Lifelong learning, learner models and sugmented cognition
Sequence mining
Managers Developers Loafers Others
Group 1 *1 3 1 1
Group 2 *1 0 1 3
Group 3 0 1 2 **3
Group 4 *1 3 2 0
Group 5 3 *1 0 3
Group 6 *1 1 3 1
Group 7 *1 0 2 4
Group 1 – 1 person had sequences characteristic of managers.
* That person had the manager role
Group 1 – 3 members had developer activity sequences
Group 3 – dysfunctionaland here we might see why
Group 5 – another way to be dysfunctional
chai::Computer human adapted interaction research group
Activity 1
• Your Stakeholders?
chai::Computer human adapted interaction research group
Activity 1
• Stakeholders?– Learners– Parents, Mentors, Facilitators– Teachers– Supervisors– Institutions– Quality assessors– Researchers
chai::Computer human adapted interaction research group
Activity 2
• Problems you would like to tackle?
chai::Computer human adapted interaction research group
Activity 2
• Current problems we aim to tackle?– Teacher: Early identification of at-risk individuals– Learner: Decision support
• Am I doing well enough?• Am I doing what is expected of me?
– Institution: Effectiveness of teaching and learning?
chai::Computer human adapted interaction research group
Building from SMILI
Bull, S., & Kay, J. (2007). Student Models that Invite the Learner In:
The SMILI:() Open Learner Modelling Framework. International Journal of Artificial Intelligence in Education, 17(2), 89-120.
chai::Computer human adapted interaction research group
What is an Open Learner Model?
• Any interface to data that a system keeps about the learner
• Came from AI + personalisation where learner model drives personalisation
• OLM has become a first-class citizen!• Link to Learning Analytics….
chai::Computer human adapted interaction research group
SMILI questions
• How does the open learner model fit into the overall interaction? – What problem does it aim to address?
• WHAT is open? • HOW is it presented? • WHO controls access?
chai::Computer human adapted interaction research group
The purposes for opening the learner model are:
• Improving accuracy • Promoting learner • Helping learners to plan and/or monitor their • Facilitating collaboration and/or competition • Facilitating navigation of the learning system • Assessment
• Complex of issues of managing personal data:– right of access to data about themselves– Right of control over their learner model – increasing trust
chai::Computer human adapted interaction research group
Scrutable user models and personalised systems
Research systems only, so farBut hints of their being ready to emerge in mainstream software
chai::Computer human adapted interaction research group
Interfaces to substantial learner models
Core concepts in a whole semester long subject
chai::Computer human adapted interaction research group
HCI subject with online lectures
• Exploit data from:– logs of interaction with lecture “slides”– class assessments
• Lightweight ontology for tagging– automatic analysis of online dictionary– augmented with class-specific concepts (as class
glossary) – enabling combination of multiple data sources about
each concept– and inference up/down ontology
chai::Computer human adapted interaction research group
ITS2008Lifelong learning, learner
models and sugmented cognition
SIVLots of green means
learner doing well
Weak aspects visible as red
Overviewvisualisation
chai::Computer human adapted interaction research group
ITS2008Lifelong learning, learner
models and sugmented cognition
SIV
Kay, J and A Lum. "Exploiting readily available web data for scrutable student models.” Proceedings of the conference on Artificial Intelligence in Education 2005.
Little detail
chai::Computer human adapted interaction research group
Mental models
chai::Computer human adapted interaction research group
Mental models
A set of beliefs that the user holds
chai::Computer human adapted interaction research group
Mental models
A set of beliefs that the user holdseg. A whale is a fish
The subject requires rote learningI expect to perform at about the median in this class
chai::Computer human adapted interaction research group
Mental models come from:• Formal education• And so much else
– Experience– Cultural expectations– Context– Emotional state – ….
• Determining what the user– Believes to be true– Trusts– Feels permitted to consider and do– Feeling of competence
chai::Computer human adapted interaction research group
Why do mental models matter for interface designers?
chai::Computer human adapted interaction research group
Why do mental models matter for interface designers?
They define • what a user can “see” and “hear”• How they interpret informationClashes between user, programmer, expert MMs
chai::Computer human adapted interaction research group
Activity
• Mental models• What are key elements for your LA needs?
chai::Computer human adapted interaction research group
Pervasive technologies Case study
Lots of embedded interaction devices, ready for interaction
Where things may be headed….
chai::Computer human adapted interaction research group
User models in real classrooms
For orchestrationFor in-class monitoring to inform teacher actions
For post-hoc reflection by the teacher
The collaborative task (concept mapping and problem solving)
[6] Novak, J. and A. Cañas, The Theory Underlying Concept Maps and How to Construct and Use Them T.R.I.C. 2006-01, Editor. 2006, Florida Institute for Human and Machine Cognition.
• Concept mapping is: – A tool for externalising knowledge
– Applied in different domains
– Promotes meaningful learning
– Has been used by organisations such asNASA, Navy, and universities around the world.
chai::Computer human adapted interaction research group
chai::Computer human adapted interaction research group
chai::Computer human adapted interaction research group
The big pictureCSCL
Computer Supported Collaborative learning
HCIHuman Computer
Interactions
EDMEducational Data-Mining Interactive tabletops in the
classroom
Interactive Tabletops and Surfaces 2010, 2011Work In Progress , CHI 2012
Int. Conf. in Learning Sciences, ICLS 2012Intelligent Tutoring Systems, ITS 2012
Computer Supported Collaborative Learning CSCL 2011Educational Data Mining 2011
Interactive Tabletops and Surfaces 2012Workshop on Orchestration , ICLS 2012
orchestration
chai::Computer human adapted interaction research group
Architecture
chai::Computer human adapted interaction research group
Collaid
Our gear
Learner’s physical differentiation
chai::Computer human adapted interaction research group
Our enriched interactive tabletop
Kinect sensor
Multi-touch tabletop
R. Martinez, A. Collins, J. Kay, and K. Yacef. Who did what? who said that? Collaid: an environment for capturing traces of collaborative learning at the tabletop. In ACM International Conference on Interactive Tabletops and Surfaces, ITS 2011, pages 172-181, 2011.
Logs:
Differentiated tabletop actions
Snapshots of the artefact
chai::Computer human adapted interaction research group
Our gear 2
CmateConcept Mapping at the Tabletop
chai::Computer human adapted interaction research group
Learning outcomes for activities
Concentric layout
Significant correlated with higher levels of equity of participation (>0.4).
Concentric
Oriented towards a Learner
“next time I would ask students to use a circular layout”
Teacher:
chai::Computer human adapted interaction research group
From Design to Enactment and Reflection
Collaboration and
equality
Adherence to the class
script
Learning outcomes for
activities
chai::Computer human adapted interaction research group
Classroom activity design
chai::Computer human adapted interaction research group
Adherence to the class script
(14 tutorials)
Implications
This was the most important activity from the learning perspective
It forced the teacher to use more time than the 50 minutes
There was not enough time for activity 2 as planned
chai::Computer human adapted interaction research group
Adherence to the class script
“This is a very good reminder... maybe the structure for the next tutorials should be changed to give more
time for Activity 2”
Teacher:
Effectiveness of the script
Post-hoc teacher’s reflection
Standardisation if multiple tutors.
Activity re-design
chai::Computer human adapted interaction research group
Learning outcomes for activities
* high achieving groups had more than 50% of these crucial propositions
This analysis suggests the low achieving groups took longer to get started.
“It would be more valuable to get this information per each group during the tutorials”.Teacher:
chai::Computer human adapted interaction research group
Can tabletops automatically alert teacher which group
may need attention?
Adding real-time learner model What’s the impact of
showing information to the teacher?
chai::Computer human adapted interaction research group
The Orchestration Dashboard
To help teachers to control multiple classroom tutorial
sessions
chai::Computer human adapted interaction research group
Awareness and Control
chai::Computer human adapted interaction research group
The Awareness Dashboard
To help teachers to determine whether groups or individual
learners need attention
Multiplatform!
chai::Computer human adapted interaction research group
Group 1
Group 2
Group 3
Class level Dashboard
Martinez Maldonado, R., Kay, J., Yacef, K. and Schwendimann, B. An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Environment. Intelligent Tutoring Systems (2012), 482-492.
chai::Computer human adapted interaction research group
Group 1
Group 2
Group 3
• A Best-First tree model trained in another dataset classifies each block 30 seconds of activity
Features:• # of active participants in verbal discussions, • amount of speech, • number of touches • symmetry of activity (Gini coefficient).
• Labels: Collaborative, Non-collaborative, or Average.
• The visualisation shows the accumulation of these.
Martinez R, Wallace J, Kay J, Yacef K Modelling and identifying collaborative situations in a collocated multi-display groupware setting. In: AIED 2011. pp. 196-204 (2011)
Class level: Indicator of detected collaboration.
chai::Computer human adapted interaction research group
Class level: Graph of interaction with others’ objects
Group 1
Group 2
Group 3
• The Circles indicate the number of touches
• The Lines represent the number of actions that each learner performed on others’ links and concepts
Alice Bob
Carl
chai::Computer human adapted interaction research group
But how to create the right visualisations?
Are there simple rules?Simple principles?
Simple and constant solutions?
chai::Computer human adapted interaction research group
State of the art
For learners….
Kahn Academy, what a student sees after the pre-test
Model of learner
Gamification element
chai::Computer human adapted interaction research group
State of the art
Skill metersGame elements
Good match to mental models
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 30-32.
chai::Computer human adapted interaction research group
State of the art
Teachers
chai::Computer human adapted interaction research group
Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter for awareness and self-reflection. In CHI'12 Extended Abstracts on Human Factors in Computing Systems (pp. 869-884). ACM.
Data about many students in an online learning environment.Current focus is red student
chai::Computer human adapted interaction research group
Acknowledgements
chai::Computer human adapted interaction research group
Interactive surfaces
Software infrastructure user control, scrutability
Interfaces to user model
AcknowledgementsData mining
Teacher assessment of usefulness (20 participants, most Computer Science)
“Visualization is much more effective at showing the differences between these datasets than statistics. Although the datasets are synthetic, Anscombe’s Quartet demonstrates that looking at the shape of the data is sometimes better than relying on statistical characterizations alone.
“Spence and Garrison …describe a simple plot called the Hertzsprung Russell … [shows] the temperature of stars on the X axis and their magnitude on the Y axis. … It turns out that no automatic analysis method has been able to find the same summarization,[as graphs at right] due to the noise and artifacts on the data such as the vertical bands.
chai::Computer human adapted interaction research group
The eye...the window of the soul,is the principal meansby which the central sense can most completely and abundantly appreciatethe infinite works of nature. Leonardo da Vinci (1452 - 1519)