Using Transaction-Level Data to Diagnose

Knowledge Gaps and Misconceptions

Randy Davies, Rob Nyland, John Chapman, Gove Allen

Brigham Young University

LAK15, Marist College, Poughkeepsie, NY

@robnyland @chapmjs

Introduction

• Does assessment data give us an

accurate picture of student knowledge?

• Do assessments leave room for possible

student misconceptions?

• What could be the possible problems with

these student misconceptions?

Types of Data

System Level Data

• Macro-level data: Data for groups of students, universities. Typically the realm of academic analytics.

Individual Level Data

• Assessment Data: Data about student performance in class activities

Transaction Level Data

• The individual transactions that create an assessment. Step level data.

Research Questions

1. How can we identify misconceptions from

student log data?

2. Does student log data tell a different story

than final answer data?

Progress Diagram

Individualize

Remediation

Process

Automate

Process

Actionable

Information

Manual Analysis

Phase 1 Phase 2 Phase 3

Data Collection

• Data collected from online Intro to Excel

Class

– www.myeducator.com

• Assessments are situated and task-based

• Step-level data for each task is captured

Example Student Log

Knowledge Components

• Syntax

• Cell Referencing

• Calculation

• Absolute

References ($)

Types of Errors

Optimal

SolutionTEXT TEXT TEXT

Used $ when

not needed

=$C11*$C$8

=C11*$C$8

Major Issue

Failed to use

$ when it was

needed

=C11*C8

=$C8*$C11

Used $

incorrectly

=C$11*C8

=C8*C$11

Type in value

to avoid $ use

=C11*0.0675

0.50.05

Error Weighting

Optimal Solution: =C11*C$8

0.5 0.6

Minor Issue

Transaction Level Data Example

D11Error

ratingD20

Error

ratin

g

Optimal Solution =C11*C$8 =F19*C$13/12

Step 1 =D11*C8 .5 =(F19*C13)/12 .5

Step 2 =C11*C8 .5=($F$19*$C$13)/1

2.55

Step 3 =C11*$C$8 .05 =(F19*C15)/12 .5

Step 4 =(F19*C13)/12 .5

Step 5 =F19*($C$13/12) .05

Final Solution =C11*$C$8 .05 =F19*($C$13/12) .05

Knowledge Gap Analysis Results

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Solution Process Final Answer

First Attempt

2nd Attempt

Amount of Error Detected

Major Errors Only

Knowledge Gap Analysis Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Persisted Resolved Emerged No Error

Solution Process

Final Answer

Only Major Errors

Knowledge Gap Analysis Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

Persisted Resolved Emerged No Error

Solution Process

Final Answer

Including Minor Errors

Future Work

• Automating the process of discovering

patterns in student answers

• Give feedback to the student based on

their responses

Questions?

Randy.Davies@byu.edu

robnyland@byu.edu

John.Chapman@byu.net