1

Measuring the Effectiveness of Error Messages Designed for Novice Programmers

Guillaume MarceauKathi Fisler (WPI)Shriram Krishnamurthi (Brown)Danny Yoo (WPI/Brown, Utah)

2

define: expected only one expression for the function body, but found at least one extra part

How well do error messages support learning(or fail to?)

When errors fail to teach,in which ways do they fail?

What makes a good error message?What is a valid metric of quality?

Can we make recommendations to the creators of pedagogical IDEs/compilers/ languages?

Prior Approaches to Message Effectiveness

“Were the messages helpful?”

No alternative

Did grades improve?

Too coarse

5

6

= execution

= receives an error message

The Student’s Response to the Error Message

= keystroke

[DEL] Deletes the problematic code wholesale.

[UNR]Unrelated to the error message, anddoes not help.

[DIFF]Unrelated to the error message, but it correctly addresses a different error or makes progress in some other way

[PART] Evidence that the student has understood the error message (though perhaps not wholly,) and is trying to take an appropriate action (though perhaps not well.)

[FIX] Fixes the proximate error (though other cringing errors might remain.)

8

Rater #1

goodbad

goodgoodbad

Rater #2goodbad

goodgoodgood

9

Rater #2Rater #1

Inter-rater agreement : κ

0.80 0.58

0.580.38

Expected Agreement = 0.702 + 0.302

= 0.58

0.0 1.0

slight fair moderate substantial excellent

0.20

0.40

0.60

0.80

0.84

0.38

eq. to chance

perfect agreement

worse than chance

10

Lab number #1 #2 #3# of errors a studentreceived during this lab(on average)

8.5 16.3 14.4

% error % bad # bad % error % bad # bad % error % bad # bad

arg. count 5% 48% 0.22 17% 27% 0.74 14% 17% 0.33parens matching 28% 24% 0.58 12% 14% 0.27 17% 0% 0.00runtime cond 3% 0% 0.00 3% 100% 0.49 4% 20% 0.12runtime type 2% 100% 0.15 8% 73% 0.91 16% 40% 0.93syntax cond 14% 51% 0.59 4% 50% 0.31 6% 26% 0.24syntax defi ne 16% 50% 0.68 14% 50% 1.14 6% 15% 0.14syntax func. ca l l 14% 64% 0.74 14% 17% 0.37 12% 14% 0.26syntax s truct 0% 0% 0.00 8% 32% 0.43 5% 92% 0.73unbound id. 16% 16% 0.21 13% 40% 0.85 16% 14% 0.32Total: 3.16 5.51 3.07

#error × %error × %bad = #bad

11

Lab number #1 #2 #3# of errors a studentreceived during this lab(on average)

8.5 16.3 14.4

% error % bad # bad % error % bad # bad % error % bad # bad

arg. count 5% 48% 0.22 17% 27% 0.74 14% 17% 0.33parens matching 28% 24% 0.58 12% 14% 0.27 17% 0% 0.00runtime cond 3% 0% 0.00 3% 100% 0.49 4% 20% 0.12runtime type 2% 100% 0.15 8% 73% 0.91 16% 40% 0.93syntax cond 14% 51% 0.59 4% 50% 0.31 6% 26% 0.24syntax defi ne 16% 50% 0.68 14% 50% 1.14 6% 15% 0.14syntax func. ca l l 14% 64% 0.74 14% 17% 0.37 12% 14% 0.26syntax s truct 0% 0% 0.00 8% 32% 0.43 5% 92% 0.73unbound id. 16% 16% 0.21 13% 40% 0.85 16% 14% 0.32Total: 3.16 5.51 3.07

Lab number #4 #5 #6# of errors a studentreceived during this lab(on average)

9.0 9.8 9.8

% error % bad # bad % error % bad # bad % error % bad # bad

arg. count 13% 20% 0.24 35% 21% 0.74 12% 31% 0.36parens matching 14% 0% 0.00 13% 0% 0.00 10% 15% 0.15runtime cond 6% 72% 0.40 8% 78% 0.62 1% 100% 0.06runtime type 8% 22% 0.17 6% 44% 0.26 3% 38% 0.13syntax cond 10% 28% 0.25 9% 20% 0.17 11% 11% 0.12syntax defi ne 7% 24% 0.14 2% 17% 0.03 3% 38% 0.10syntax func. ca l l 23% 27% 0.55 4% 29% 0.12 13% 38% 0.48syntax s truct 0% 0% 0.00 1% 0% 0.00 0% 0% 0.00unbound id. 16% 0% 0.00 20% 7% 0.14 34% 13% 0.44Total: 1.75 2.07 1.84

Vocabulary difficulties

n-ary tree

Difficult structure accessor names

Reflection

Rubric identifies pain points

Identifying fix needs deeper evaluation

Curriculum Error Messages

12

13

For the Community

Look at fine-grained edits

Gather around a common rubric

14

Interviews

Four interviews

One hour long each

Done around the midterm

Average-to-good students

15

Observation From Interviews

1 of 2

16

Interviewer: The error message says “the function body.”Do you know what “function body” means?

 Student: Nah, the input, everything that serves

as a piece of input? Interviewer: Actually, it's this. When DrScheme

says “function body” it means this part. Student: Oh man! I didn't… [The student proceeds to fix the error successfully]

17

What DrRacket Says:

define: expected only one expression for the function body, but found at least one extra part.

18

What the Student Sees:

define: expected only one thingamabob for the geewhiz’s whosiewhatsit, but found one extra whatchamacallit.

19

Circle an identifier in this program

(define (celsius->fahrenheit temp) (+ (* temp 9/5) 32))

RIGHT WRONG

20

Circle one instance of each vocabulary term in the code below

Vocabulary term Sample usage

Q1. Argument >: expects at least 2 arguments, given 1Q2. Selector this selector expects 1 argument, here it is provided 0 argumentsQ3. Procedure this procedure expects 2 arguments, here it is provided 0 argumentsQ4. Expression expected at least two expressions after `and', but found only one expressionQ5. Predicate this predicate expects 1 argument, here it is provided 2 arguments

;; (make-book number string string number number bst bst) (define-struct book (isbn title author year copies left right))

;; this-edition?: bst number number -> boolean;; Consumes a binary search tree, an ISBN number, and a year, and produces true ;; if the book with the given ISBN number was published in the given year(define (this-edition? a-bst isbn-num year) (cond [(symbol? a-bst) false] [(book? a-bst) (cond [(= isbn-num (book-isbn a-bst)) (= year (book-year a-bst))] [(< isbn-num (book-isbn a-bst)) (this-edition? (book-left a-bst) isbn-num year)] [else (this-edition? (book-right a-bst) isbn-num year)])]))

21

< 50%

22

Serendipitous Controlled Trials

BrownNEU WPI

Primitive name Procedure Primitive operator Field name Procedure application Predicate Defined name Type name Identifier Function body Function header

Argument Clause Expression Selector

= USED IN CLASS

Number of times the word was presented to a student in an error message (WPI)

Quiz score residual(after factoring out word difficulty and university strength)

Wordsused in class

Words NOTused in class

Wordsused in class

Words NOTused in class

p = 0.034

(which is statistically significant at the

0.05 level)

For words used in class:more exposure in error messages better quiz scores

For words not used in class:more exposure in error messages worse quiz scores

25

Old term New termProcedurePrimitive namePrimitive operatorPredicateSelectorConstructor

Function

NameIdentifier ArgumentDefined name

Variable, argument(“argument” is reserved for actualarguments in function calls)

Sequence At least oneStructure type name Structure nameQuestion—answer clause A clause is expected to have a question and an answer

Function header KeywordType< > ‘ ’

These words and notations are removed entirely and reworded in terms of other vocabulary words.

Function bodyExpressionField nameType nameTop levelBindingClausePart

These words stay unchanged

26

Observation From Interviews

2 of 2

27

Interviewer: When you get these highlights, what do they mean to you?

Student #1: The problem is between here and here, fix the problem between these two bars.

28

1. This expression contains the error

2. The parser didn’t expect to find this

3. The parser expected to see something after this, but nothing is there

4. This parenthesis is unmatched

5. This expression is inconsistent with another part of the code

DrRacket’s Highlight Semantics

29

Summary of Findings

1. Error messages need to explicate the meaning of the highlight.

2. Students need an avenue through which they will learn the vocabulary.

3. Error messages are hard to get right;user studies are important.

30

Read Understand Formulate

goodbad

goodgoodbad

Lab number #4 #5 #69.0 9.8 9.8

% error % bad # bad % error % bad # bad % error % bad # bad

arg. count 13% 20% 0.24 35% 21% 0.74 12% 31% 0.36parens matching 14% 0% 0.00 13% 0% 0.00 10% 15% 0.15runti me cond 6% 72% 0.40 8% 78% 0.62 1% 100% 0.06runti me type 8% 22% 0.17 6% 44% 0.26 3% 38% 0.13s yntax cond 10% 28% 0.25 9% 20% 0.17 11% 11% 0.12s yntax defi ne 7% 24% 0.14 2% 17% 0.03 3% 38% 0.10s yntax func. ca l l 23% 27% 0.55 4% 29% 0.12 13% 38% 0.48s yntax s truct 0% 0% 0.00 1% 0% 0.00 0% 0% 0.00unbound id. 16% 0% 0.00 20% 7% 0.14 34% 13% 0.44Total: 1.75 2.07 1.84

n-ary tree

[FIX] Fixes the proximate error (though other cringing errors might remain.)

[PART] Evidence that the student has understood the error message (though perhaps not wholly,) and is trying to take an appropriate action (though perhaps not well.)

[UNR] Edit unrelated to the error message, and does not help.

[DEL] Deletes the problematic code wholesale.

[DIFF] Edit unrelated to the error message, but it correctly addresses a different error or makes progress in some other way.

Kathi Fisler (WPI)Shriram Krishnamurthi (Brown)

kfisler@cs.wpi.edusk@cs.brown.edu