Date post: | 01-Dec-2014 |
Category: |
Technology |
Upload: | games-for-change |
View: | 300 times |
Download: | 1 times |
Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational
Games
Ryan S.J.d. BakerAssistant Professor of Psychology, Learning Science, and Computer Science
Worcester Polytechnic Institute
2
Many ways to assess engagement and affect
• I’ll discuss two methods our lab uses
3
Quantitative Field Observations (Expert Judgments)
• Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting– Conducted using peripheral vision/side glances– Good inter-rater reliability: k 0.6-0.8– Include engaged behaviors (collaboration with other students) and
disengaged behaviors (off-task behavior)– Include positive affect (delight, engaged concentration) and negative
affect (boredom, frustration)
• Ecologically valid assessments of how much and when – Students are disengaged– Students experience specific affect
4
Automated Detectors• Models that assess student engagement and affect in real-time or
retrospectively from behavior within software• In our approach, no sensors used
– Improves scalability – lots of data being automatically collected these days– Reduces predictive power for some affective states, relative to sensor-based
detectors• Successful at detecting disengaged behaviors such as off-task
behavior, carelessness, gaming the system• Successful at detecting engaged concentration and boredom in two
learning systems– Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008)
• Used in interventions that improve learning outcomes(Baker et al., 2006)
5
Ongoing Project (NSF PSLC)
Quantitative Field
Observation
Affect Basic Research
Comparative Analysis
Completed for intelligent tutors; in process for serious games
Detector Development
6
Use in Research
7
How does student affectdiffer between games and ITS?
(Rodrigo & Baker, 2011)
8
Aplusix .vs. MathBlaster
Matched mathematical content between systems
Student affect assessed using quantitative field observationswith real students in real classrooms
9
Interesting differences in affect
Condition Engaged Concentration Delight
Aplusix 76% 6%
MathBlaster 63% 12%
Proportions of each affective state shown
10
How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review)
11
Students compete to be first to identify a substance chosen by their opponent
Chemistry Game (Yaron et al., 2010)
Student affect assessed using quantitative field observationswith real students in real classrooms
12
Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010)
Bored
Confused
Gaming the
System
13
With Social Behavior(Baker, Moore, et al., under review)
Off-Task Behavior Bored
Confused
Gaming the
System
On-Task Conversation
14
Bottom-Line
• Field observations and detectors are powerful tools
• For assessing and understanding student engagement and affect during learning
• Including in serious games