Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational

Games

Ryan S.J.d. BakerAssistant Professor of Psychology, Learning Science, and Computer Science

Worcester Polytechnic Institute

2

Many ways to assess engagement and affect

• I’ll discuss two methods our lab uses

3

Quantitative Field Observations (Expert Judgments)

• Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting– Conducted using peripheral vision/side glances– Good inter-rater reliability: k 0.6-0.8– Include engaged behaviors (collaboration with other students) and

disengaged behaviors (off-task behavior)– Include positive affect (delight, engaged concentration) and negative

affect (boredom, frustration)

• Ecologically valid assessments of how much and when – Students are disengaged– Students experience specific affect

4

Automated Detectors• Models that assess student engagement and affect in real-time or

retrospectively from behavior within software• In our approach, no sensors used

– Improves scalability – lots of data being automatically collected these days– Reduces predictive power for some affective states, relative to sensor-based

detectors• Successful at detecting disengaged behaviors such as off-task

behavior, carelessness, gaming the system• Successful at detecting engaged concentration and boredom in two

learning systems– Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008)

• Used in interventions that improve learning outcomes(Baker et al., 2006)

5

Ongoing Project (NSF PSLC)

Quantitative Field

Observation

Affect Basic Research

Comparative Analysis

Completed for intelligent tutors; in process for serious games

Detector Development

6

Use in Research

7

How does student affectdiffer between games and ITS?

(Rodrigo & Baker, 2011)

8

Aplusix .vs. MathBlaster

Matched mathematical content between systems

Student affect assessed using quantitative field observationswith real students in real classrooms

9

Interesting differences in affect

Condition Engaged Concentration Delight

Aplusix 76% 6%

MathBlaster 63% 12%

Proportions of each affective state shown

10

How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review)

11

Students compete to be first to identify a substance chosen by their opponent

Chemistry Game (Yaron et al., 2010)

Student affect assessed using quantitative field observationswith real students in real classrooms

12

Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010)

Bored

Confused

Gaming the

System

13

With Social Behavior(Baker, Moore, et al., under review)

Off-Task Behavior Bored

Confused

Gaming the

System

On-Task Conversation

14

Bottom-Line

• Field observations and detectors are powerful tools

• For assessing and understanding student engagement and affect during learning

• Including in serious games