1/∞CRESST/UCLA

Towards Individualized Instruction With Technology-Enabled Tools and Methods

Gregory K. W. K. Chung, Girlie C. Delacruz, Gary B. Dionne, Eva L. Baker, John Lee, Ellen Osmundson

American Educational Research AssociationAnnual MeetingChicago, IL - April 9-13, 2007

Symposium: Rebooting the past: Leveraging advances in assessment, instruction, and technology to individualize instruction and learning

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing

2/∞CRESST/UCLA

Structure of Talk

• Problem

• Pre-algebra/Algebra

• Classroom constraints

• Research

• Domain analysis, assessment design, instructional design, screen shots

• Results

• Did it work?

3/∞CRESST/UCLA

Problem

4/∞CRESST/UCLA

Study Context• Why pre-algebra?

• Pre-algebra provides students with the fundamental skills and knowledge that underlie algebra

• Pre-algebra Algebra STEM

• 2001-02 LAUSD 9th grade cohort:

• Only 65% of 9th graders (~28,000) progress to 10th grade (23% retained in 9th grade [~10,000]; 12% leave district)

• Algebra identified as gatekeeper

• In fall 2006, 38% of CSU first-time freshmen needed remediation in mathematics (17,300)

• CSU 5-year graduation rate (STEM): 34%

5/∞CRESST/UCLA

Proportion Wrong

8th(1st semester)

9th (entering)

.23 .23

.51 .10

.38 .08

.34 .33

.86 .70

6th/7th Grade Standards

6/∞CRESST/UCLA

Research

7/∞CRESST/UCLA

Research Questions

• To what extent can students learn from “instructional parcels”—brief slices of instruction and practice?

• To what extent can automated reasoning (i.e., Bayesian networks) be used for automated diagnosis of pre-algebra knowledge gaps?

• What is the architecture for a diagnosis and remediation system?

8/∞CRESST/UCLA

General Idea

Automated Reasoning(Bayes net)

Pre-algebra pretest adding

fractions distributiveproperty

multiplicative identity

Individualized instruction

and practice

Individualized posttest

9/∞CRESST/UCLA

Research Study Overview

• Domain analysis and assessment design

• Identify the key concepts that underlie pre-algebra and the relations among those concepts

• Instruction and practice

• Design based on best-of-breed (worked examples, schema-based instruction, multimedia learning, effective feedback)

10/∞CRESST/UCLA

Domain Analysis

11/∞CRESST/UCLA

Domain Analysis

12/∞CRESST/UCLA

Domain Analysis

13/∞CRESST/UCLA

Sample Assessment Items

DP (OPER)

CP-ADD (CE)

AP-ADD (CE)

DP (CE)

DP (CE)

DP (CE)

DP (OPER)

DP = distributive propertyCP-ADD = commutative property of additionAP-ADD = associative property of addition

CE = common errorOPER = operation

14/∞CRESST/UCLA

Instructional Design

15/∞CRESST/UCLA

Design Assumptions

• To maximize the chance of learning with only brief exposure to content, instruction should:

• Direct the learner’s attention to important content

• Highlight and explain the importance of the content

• Use lay language and 1st/2nd person voice

• Use worked examples with visual annotations, coordinated and complementary narration

• Provide varied examples with different surface features but same underlying concept

• Provide practice on applying the concept

• Provide tailored and explanatory feedback

16/∞CRESST/UCLA

Multiplicative Identity - 4

17/∞CRESST/UCLA

Multiplicative Identity - 5

18/∞CRESST/UCLA

Multiplicative Identity - 6

19/∞CRESST/UCLA

Multiplicative Identity - 7

20/∞CRESST/UCLA

Multiplicative Identity - 8

21/∞CRESST/UCLA

Multiplicative Identity - 9

22/∞CRESST/UCLA

Multiplicative Identity - 10

23/∞CRESST/UCLA

Multiplicative Identity - 11

24/∞CRESST/UCLA

Multiplicative Identity – 12/12

25/∞CRESST/UCLA

Multiple Examples

Stage 1: What is the next step in solving the problem?

Stage 2: What is the result of carrying out the step?

Stage 3: What is the underlying math concept?

26/∞CRESST/UCLA

Feedback is tailored to the specific option selected

Feedback is tailored to the specific option selected

• Knowledge of results (correct/incorrect)

• Explanation—why correct or incorrect

• Knowledge of results (correct/incorrect)

• Explanation—why correct or incorrectWhat to think about to solve problemWhat to think about to solve problem

Animation/video—goal, why, how, andcommon errors

Animation/video—goal, why, how, andcommon errors

27/∞CRESST/UCLA

Method

• 2 group pretest, posttest design

• 113 middle school students

• Instruction vs. no instruction (stratified by concept and high, medium, low knowledge)

• Procedure

• Pretest (84 items, = .89)

• Instruction on concepts

• 4-6 concepts per student (out of 10)

• Individualized posttest

28/∞CRESST/UCLA

Method

• Analysis

• Examined 6 scales (pretest = .61 - .75)

• Adding fractions, distributive property, transformations, multiplicative identity

• Multiplying and adding fractions, rational number equivalence

• Participants dropped an analysis if

• Diagnosed as high knowledge

• Non-compliant (20-50%, depending on scale)

29/∞CRESST/UCLA

Results

Adding fractions

Distributive property

Transfor-mations

Multiplicative identity

Multiply, add fractions

Rationale number

equivalence

max = 8n = 9/22p = .50d = --

max = 8n = 14/18p = .04d = .76

max = 6n = 13/21p = .04d = .77

max = 7n = 7/21p = .06d = .91

max = 6n = 13/12p = .03d = .91

max = 6n = 13/12p = .01d = .50

Instruction

No instruction

30/∞CRESST/UCLA

Student Perceptions

• 46% reported tool the was very useful, 53% reported they were very willing to use

• Tool not for everyone

• “I really understood the way they took step by step to show the problem.”

• “I thought that the video and the practice problems were very easy to understand, but if they were a little more ‘exciting’ it would help make the process more fun.”

• “No, it just made me confused. I like seeing everything on a board, not computer.”

31/∞CRESST/UCLA

Closing Remarks

• Preliminary results promising

• Instruction appears effective, even if brief

• Technical aspects of individualization of instruction and assessment tractable

• DITSy – Dumb Intelligent Tutoring System

• Limitations and next steps

• One instructional day between pretest and posttest; unknown retention effect; low on some scales; novel assessment format

• Refine measures, replicate, validate diagnosis, examine more complex outcomes, examine instructional variables

32/∞CRESST/UCLA

Backup

33/∞CRESST/UCLA

Content

• Ten concepts tested

• Properties (e.g., distributive property)

• Problem solving (e.g., multiplying fractions)

• Time per parcel

• Mean time (mm:ss): 3:45

• Range (mm:ss): 2:05 to 5:19, 10:58

34/∞CRESST/UCLA

Task Sequence

Instruction(worked examples, narration,

visual annotations)

Practice (stage 1)Identify thenext step

Practice (stage 2)Identify the result

Feedback(tailored andexplanatory)

Practice (stage 3)Identify the

math concept

Feedback(tailored and explanatory)

Feedback(tailored and explanatory)

35/∞CRESST/UCLA

Incorrect response feedback: (a) confirmation that the response is incorrect, and (b) a brief explanation of why the response is wrong

Hint on what to think about to solve problem

Feedback is tailored to the specific option selected.

36/∞CRESST/UCLA

Correct response feedback: (a) confirmation that the response is correct, and (b) a brief explanation of why the response is correct

37/∞CRESST/UCLA

Don’t know response feedback: guidance on what the student should be considering

38/∞CRESST/UCLA

animation/video feedback: goal, why, how, and common errors

39/∞CRESST/UCLA

Timing

Concept Default Time

10:58 Adding fractions

5:19 Reducing fractions

4:12 Associative property of multiplication

4:09 Distributive property

3:27 Multiplicative inverse

3:19 Multiplying fractions

3:04 Multiplicative identity

2:27 Associative property of addition

2:14 Commutative property of addition

2:05 Commutative property of multiplication

40/∞CRESST/UCLA

Some Observations

• Classroom / group instruction inefficient

• A lot of time spent on non-instructional activities

• Teacher telling jokes

• Students settling down (pulling out notebook from backpack, opening textbook, getting up to sharpen pencil)

• Teacher writing the equation on the whiteboard

41/∞CRESST/UCLA

Pretest-Posttest (Instruction)

Adding fractions

Distributive property

Transfor-mations

Multiplicative identity

Multiply, add fractions

Rational number

equivalence

max = 8n = 22p = .02d = .50

max = 8n = 18p = .09d = .54

max = 6n = 21p < .001d = .61

max = 7n = 21p < .001d =1.36

max = 6n = 11p = .14d = .37

max = 6n = 12p = .10d = .43

42/∞CRESST/UCLA

Pre-algebra Bayes Net

definitions

definitions

operations

operations

transformations

transformations

common errorscommon errors

43/∞CRESST/UCLA

Diagnosis and Remediation

• Model the domain of pre-algebra with a Bayesian network

• Treats test items as evidence of understanding

• Computes the probability of a student understanding a concept

• Short slices of instruction that are focused on a single concept

• NOT intended to replace classroom teaching

• Intended to support homework, review, wrap-around activities

44/∞CRESST/UCLA

45/∞CRESST/UCLA

Multiplicative Identity - 1

46/∞CRESST/UCLA

Multiplicative Identity - 2

47/∞CRESST/UCLA

Multiplicative Identity - 3