Nuts and Bolts of Progress Monitoring

Laura Boynton Hauerwas

Kristen Matthes

May 31, 2006

Topics for the morning1. Progress Monitoring Assessment:

• What?• Why?

2. Sample PM tools3. Graphing Data4. Data-Base Decisions

• Setting Goals• Performance Level• Rate of Learning

5. Linking cases to RI RTI processes

Assessment in a RTI model

BenchmarkingTo screen and identify students who are at-risk and in need of interventions

All students

Three times a year

All areas

At grade-level

Progress MonitoringTo monitor progress of individual students and determine rate of improvement and need for adaptation of intervention

Students who are not achieving benchmarks (PLP, IEP)

Weekly, biweekly, monthly assessments

In area of need

At instructional level

What assessments do you use?

Reading Math Social-Emotional

Behavior

What measures do you use?

Who do you assess?

When do you assess?

How is the information used? Benchmark/

Progress Monitoring

Progress Monitoring

Benefits of Progress Monitoring

Parents and students know what is expected

Teachers know what is working or not working with their instruction based on data

Easy to understand way to show parents progress

Teams have comprehensive data on student performance for decision making

Progress Monitoring

CBMs• Are assessments to monitor progress

• Are designed to serve as ““indicatorsindicators”” of general reading achievement. R-CBM doesn’t measure everything, but measures the importantimportant things.

• Are Standardized testsStandardized tests to be given, scored, and interpreted in a standard way. standard way.

• Are researchedresearched with respect to psychometric properties to ensure accurate measures of learning.

• Are SensitiveSensitive to improvement in Short PeriodsShort Periods of time.

• Designed to be as short as possibleas short as possible to ensure its ““do abilitydo ability.”.”

• Are linked to decision makinglinked to decision making for promoting positive achievement and Problem-Solving

Progress Monitoring

National Center on Student Progress Monitoring www.studentprogress.org

Samples of CBMs

• Reading

• Math

• Writing

• Spelling

MAZE - CBM

AIMSweb Reading Comprehension Measure

www.aimsweb.com

DIBELS Phoneme Segmentation Fluency

https://dibels.uoregon.edu

79x

41 644

52 85 264 70 8+

13 07x

B C D E

G H I J

L M N O

Q R S T

V W X Y

A

F

K

P

U

92 )

23

13 =

15 0414 41

Sheet #2

Password: AIR

Computation 4

Name: Date

12 =64 +

47 =13 -

511

311 =+

249 )

10 73x

62x

86x

724 )

655 )

105 )

3574x

90x

306 )

819 )

82 8543 04

+ 90

3223x

45x

306 )

Taken from Fuchs, L. S., Hamlett, C. A., & Fuchs, D. (1998). Monitoring Basic Skills Progress: Basic Math Computation (2nd ed.). [computer program]. Austin, TX: ProEd.

Available: from

http://www.proedinc.com

MATH COMPUTATION

Concepts and Applications

Sample page from a three-page test for Grade 2 Math Concepts and Applications– From Monitoring

Basic Skills Progress

CBM - Writing

www.interventioncentral.org

Total Words WrittenCorrect Word SequencesWords Correctly Spelled

Spelling

AIMSweb Spelling Probes

Date: 1/25/ 04 Time: 2:15 pm Setting: Classroo m/ Ma thStudent: Den n i s Teacher: Miss S Observer: S Freli ngTeacher Behavior: A = Atten tio n : prai se, r ed i rectin g, providin g a c omman d

I = No a tten tio n : provi d ing n o a ttenti on o f an y kin dStudent Behavior: - = Acti ve engag ement : wr iting , read i ng, a nswering

tea cher qu estion+ = O ff ta sk beha vio r : screa min g, r epetitiv e a udi ble noi ses,ou t of seat, l a ug hing , pound i ng fist on ta ble

Interval length: 10 secon ds

Teacher A A A I I I I A A A

Student + + + + - - - - + +

Teacher I A A I I A I I I A

Student - - + - - + - - - -

Teacher A I I I I A A A I I

Student + + - - - - + + - -

Graphing

Graphing is an essential part of PM

Without graphic displays, the decision making process is difficult

Teacher graphing vs. Student graphing

How to Develop Graphs

Graphing Data: Beetle, SUV, Race Car

Hand Graphing

Excel and Chart Dog

Web-based data systems

Hand Graphing1. Establish Baseline (Median score)

2. Set up graph

3. Set Goal

4. Draw Aimline

5. Measure Student Progress

6. Plot Student Performance

7. Connect Indicators of Student Performance

8. Analyze Student Performance

9. Make Instructional Changes

10. Continue to Measure and Monitor Student Performance

Hand Graphing

Goal 44

Hand Graphing

Testing SessionsBaseline Session 1 Session 3Session 2 Session 4 Session 6Session 5 Session 7 Session 8

Nu

mb

er

of

Wo

rds

Re

ad

Co

rre

ctly

30

35

40

45

50

Hand GraphingAdvantages

• Easy to do• No technology

required• Students can easily

maintain their own graphs

• Can be done immediately

• Free

Disadvantages• Added paper• Organization required• No long-term storage• Not automatic

Excel• Excel Template

Website to access Excel Templates

http://www.oswego.edu/~mcdougal/web_site_4_11_2005/index.html

To add trend line:1. Select Data Points2. Right Click …Add Trend Line3. Choose Linear, 4. To find slope –Options Add Equation to the chart (y= slopex+ intercept)

Chart Dog• Tool at www.interventioncentral.org

Excel and Chart Dog

Advantage• Automatic• Storage capability

built-in• Easy to do• Clear displays of data• Free if you have

EXCEL, Chart Dog is free

Disadvantage• Requires technology• Time to enter data• Students may not be

able to do data entry themselves

• Requires some understanding of EXCEL or Chart Dog

Web Based System

Web-Based SystemAdvantage

• Web based data entry from anywhere

• Storage capability built-in

• Trend line drawn automatically

• Can annotate graphs interventions/goals

• Norms –benchmarks and Rate of Improvement

• Lots of flexibility

• Email graphs

Disadvantage

• Requires technology

• Cost

• Students may not be able to do data entry themselves

• Requires some training

To do this will take newlearning for everyone

BREAK

Data-Based Decisions

1. Performance Level • Gaps in Performance

• PLP Not at Grade Level• Special Education Significant Discrepancy

2. Rate of Learning• Trend in performance (slope)

• Response to Instruction• General Direction, Rate of Change

1. Performance Level: Gap/ Discrepancy

• Be objective. Does it refer to an observable/measurable characteristic of behavior?

• Use numbers to define the discrepancy.– Percentile rank– Discrepancy Ratios– Cut scores

• Norms

Norms… What to use? Local, National

• Local norms can be helpful to determine local performance levels and rates of progress

• Time consuming and costly to develop• National norms and research norms are

available.

BUT….

Percentile Ranks

Requires a Larger Normative Data Base, Preferably Benchmark Data

< 25th At Risk, Consider Problem-Solving at the Group Level

<10th Potential Severe Problem, Consider Individual Problem Solving

1. Performance Level

Discrepancy Ratios

1. Sample 5-7 Students or Whole Class, Grade

2. Figure Median and Graph

3. Divide by 2 and Graph

4. Students Who Performance Below the Line May Need Problem Solving

Performance Level

Can Compute…

Peer MedianTarget Student Median

145 40

= Discrepancy of 3.6x

Cut Scores• A number which represents the point at which scores

can be divided into different groups (for example does not meet, meets, and exceeds expectations) for decision-making purposes.

• May be based on research (e.g., a correlation between scoring at or above a certain level on a CBM or DIBELS task and future academic success) or expectation (e.g., grades at C or above, no more than 3 office referrals).

Performance Level

Generally speaking… - A student who is 1.5x discrepant from his/her peers may benefit from intensive group interventions.– A student who is 2-2.5x discrepant from his/her

peers is appropriate for individualized problem-solving and intensive intervention resources may be appropriate.

Example: Jessica is 2.1x discrepant from peerson the Math CBM and may benefit from

intensive interventions in math.

Data-base decisions on performance level enables team to make decisions about levels of support and resource from the start.

Rebecca 2nd grader• List all areas of concern:

– Off-task behavior– Reading difficulties– Poor handwriting

• Identify primary area of concern and define it in observable and measurable terms:– Reading – Definition: number words read correctly when reading a grade level

passage orally

• Collect baseline data on primary area of concern and state discrepancy statement:– Baseline data collected in the area of test from CBM reading probes– Discrepancy Statement: Rebecca reads 41 WRC per minute

in Fall of 2nd grade while her peers read ____ WRC per minute __________________________________________________________________

Rob 7th grader• List all areas of concern:

– Calling out

– Lack of homework completion

– Poor handwriting

• Identify primary area of concern and define it in observable and measurable terms:– Work Completion

– Definition: Turning in teacher assigned work at beginning of class period on the day that it is due.

• Collect baseline data on primary area of concern and state discrepancy statement:– Baseline data collected in the area of review from teacher grade books

– Discrepancy Statement: Rob currently turns in homework 54% of the time while his peers turn in homework 86% of the time (_____ discrepant)

Data-Based Decisions

1. Performance Level • Gaps in Performance

• PLP Not at Grade Level• Special Education Significant Discrepancy

2. Rate of Learning• Trend in performance (slope)

• Response to Instruction• General Direction, Rate of Change

2. Rate of Learning

• Why?– Determine when what we are doing isn’t

working and intervene early – Better able to predict student success at

meeting goals– Better able to identify who needs more

intensive instruction

Name Grade Area Initial Performance Discrepancy

Follow Up Performance Discrepancy

Outcome Decision

Bill 3 Reading 10th percentile 20th percentile

Satisfactory; Maintain Intervention

Susie 2 Math 1.2x NA No Severe Problem

Rob 4 Homework

65% completion 64% completion

No Progress, Problem solve and Modify intervention plan

Rate of LearningTracking Student Outcomes Using Initial Performance Discrepancies

Rate of Learning

• Analyzing Rate using PM Data

• Rules:– Setting Goals– Data Point Rules– Trend Line Rules– Slope

Setting Goals

1. End of the Year Benchmarks• GLEs for Reading Fluency (2nd grade 80-100 WPM, 5th grade 125-150)

• AIMSweb Math Computation Norms (1st grade 17 DPM, 5th grade 52 DPM)

2. National Norms for Improvement• Math Calculations (>.3 DPM 2nd and 3rd grade, >.5 DPM 4-6th grade)

(Fuchs, 2006)• Reading Fluency (Deno, 2005)

Grade Modest Reasonable Ambitious

1-2 1 Word Per Week 1.5 Word Per Week 2.0 Word Per Week

3-6 .5 Word Per Week 1.0 Word Per Week 1.5 Word Per Week

Setting Goals3. Individual ROI

• Weekly rate of improvement in “baseline slope” calculated from 8 data points (Slope: Difference of highest and lowest/#weeks)

• Baseline multiplied by 1.5• Product multiplied by number of weeks until end of year• Add to student’s final baseline score to produce end of year

goal.

Baseline Reading scores: 52, 54, 52, 53, 55, 58, 55, 56

Difference: 58-52 =5Divide by number of weeks: 5/8 =.625 (SLOPE)

Baseline multiplied by 1.5: .625 × 1.5 = .9375Number of weeks left (6 weeks): .9375 ×6 =5.6

Add to final baseline score: 56+5.6 = 61.6

End of the year goal 62

Jim – 5th grader

Math Calculation

December-January Monitoring 36, 37, 36, 36, 37, 38, 39, 37

January Benchmark38

Base determination on 18 weeks left in the year

What goal would you set for Jim in math for the end of year?

Jim – 5th grader

End of Year Benchmarks 52 DPM (.77 ROI)

National Norms 47 DPM (.5 ROI)

(38+18*.5)

Individual ROI 48 DPM (.56 ROI)

3/8*1.5 =.56 ROI

38 + (18*.56)

What goal would you set for Jim in math for the end of year?

Decisions based on data-points

Decisions are based on at least 4 data points• If all 4 scores fall above goal-line, responding to

instruction (increase goal if continues for 4 more data points)

• If scores are hovering about the goal line, continue what you are doing.

• If all 4 scores are below goal-line, but parallel, decide to “wait” for 4 more points to see if student performance accelerates in level to reach original goal.

• If all 4 scores fall below goal-line, not responding to instruction, revise plan and implement different teaching strategy.

• Mark change on graph with vertical line.

Derived from: Fuchs and Fuchs (2006) and Shapiro (2006)

Maze Monitoring

0

2

4

6

8

10

12

14

16

Baseline Week 1-a

Week 1-b

Week 2-a

Week 2-b

Week 3-a

Week 3-b

Week 4-a

Week 4-b

Nu

mb

er

Co

rrect

Joel

Goal Line

Linear (Goal Line)

Decisions based on trend lines

Trend lines based on 6-8 data-points

• If trend line is steeper than goal line, increase the goal.

• If trend line is flatter than goal line, revise instruction

• If trend line equals goal line, make no change at this time.

Maze Monitoring

0

2

4

6

8

10

12

14

16

Baseline Week 1-a

Week 1-b

Week 2-a

Week 2-b

Week 3-a

Week 3-b

Week 4-a

Week 4-b

Nu

mb

er

Co

rrect

Joel

Goal Line

Linear (Goal Line)

Linear (Joel)

Data-base decision on slope

Rate of Improvement = slope or r(in statistics)

• Y = slopeX + intercept

• Consider discrepancy from ROI norms

Maze Monitoring

y = -0.2167x + 12.417

y = 0.25x + 11.75

0

2

4

6

8

10

12

14

16

Baseline Week 1-a

Week 1-b

Week 2-a

Week 2-b

Week 3-a

Week 3-b

Week 4-a

Week 4-b

Nu

mb

er

Co

rrect

Joel

Goal Line

Linear (Joel)

Linear (Goal Line)

Is Jim responding to the

intervention?

Math Calculations

y = 0.5789x + 37.421

y = 0.5612x + 35.987

30

32

34

36

38

40

42

44

46

48

50

Baselin

e

Week 1

Week 2

Week 3

Week 4

Week 5

Week 6

Week 7

Week 8

CheckP

oin

t

Week 9

Week 1

0

Week 1

1

Week 1

2

Week 1

3

Week 1

4

Week 1

5

Week 1

6

Week 1

7

Week 1

8

Corr

ect D

igits

in 4

min

ute

s

Jim

Goal

Linear (Goal)

Linear (Jim)

Checkpoint:•Is Jim Responding to the Math Intervention?•Are modifications necessary?

End of Year: What would you say about Jim’s Math?

How can you use the literacy, math and social-emotional/behavior assessments that you have? Do they provide information about– a student’s performance level – rate of learning – area of instructional need?

Your chart might include one of these literacy measures?

R-CBM Rigby DRA

Performance Level

YES YES? YES

Rate of Learning

YES NO NO

Instructional needs

YES? YES YES

What is the difference between data-driven and data informing?

Rebecca- 2nd grader

• Fall benchmark some concern• 41 WPM, <25th percentile, 1.96X Discrepant

• Instructional Plan– small group guided reading– book bag for home with leveled readers

Rebecca- 2nd grader

Rebecca Fall Monitoring

y = 0.4571x + 41.067

35

40

45

50

55

60

Monitoring Probes

Wo

rds R

ea

d C

orr

ectly

Rebecca

Linear (Rebecca)

What additional data is needed to modify interventions and

meet Rebecca’s needs?Possible

Hypotheses

Review Interview Observe Test

Rebecca reads 41 WRC per minute in the Fall of 2nd grade while her peers read ____WRC per minute and with small group guided reading limited rate of improvement because …

Case Example: Rebecca

Possible

Hypotheses

Review Interview Observe Test

She had insufficient opportunity to practice

•Interview Teacher/Parent

•Observe during SSR

•CBM

She lacks the phonological skills to read the text fluently.

•Review her Records

•R-CBM scores

•Interview Teacher/Parent

•Phonological Assessment

Instructional materials are above her reading level.

•Review her Records/CBM scores

•Interview Teacher/Parent

•CBM survey level assessment

Rebecca reads 41 WRC per minute in the Fall of 2nd grade while her peers read 80 WRC per minute and with small group guided reading limited rate of improvement because …

Case Example: RebeccaPredicted

Hypotheses

Review Interview Observe Test

She had insufficient opportunity to practice

•Instructional Planning Form revealed Rebecca had 15 minutes of independent reading practice per day

•Observation revealed that during SSR, Rebecca often played with items on desk.

•CBM probes indicated that she was not performing comparable to peers

She lacks the phonological skills to read the text fluently.

•Review of records indicated that she had done well in remedial phonics program the previous year and was graduated out of the program.

•IPF suggested that instructional time was given to phonological skills.

•CBM probes indicated that she was not performing comparable to peers; DIBELS assessments did not point to a phonological awareness or phonics problem.

Her reading level is not instructionally appropriate.

•Teacher reports showed that she was at the 25th percentile in the Spring of 1st grade

•Teacher felt that curriculum was appropriate

•Survey level assessment revealed that Rebecca had mastered 1st grade text

Case Example: Rebecca

Rebecca reads 41 WRC per minute in the Fall of 2nd grade while her peers read 80 WRC per minute and is improving at a limited rate (.5 ROI) because …

She had insufficient opportunity to practice.

What was the impact of progress monitoring assessments for Rebecca and her teachers?

Rebecca’s progress during Reader’s Theatre

3 - 2 - 1

3 things you learned today

2 things you still have questions about

1 statement about how you feel about PM now

We will have tough choices to make – we’ll decide based on what’s best for our kids

If you’re not hopelessly confused, you’re out of touch!

If you are hopelessly confused, then you only have one choice

— try stuff.

Tom Peters, Embracing Chaos, 1993

We will make mistakes – and fix them along the way

Questions:Laura Hauerwas lhauerwa@providence.eduKristen Matthes

matthesk@bpsmail.org

Thank You!

References

Deno, S., Lembke, E. and Anderson, A. (2005) Progress Monitoring Study Group Content Module available at www.studentprogress.org.

Fuchs and Fuchs (April, 2006) Progress Monitoring: Identifying LD and Improving Student Outcomes. Presentation at National SEA Conference on SLD Determination. Kansas City, MO.

Hawkins, and Tilly, D. (Sept. 2005). Response to Intervention On the Ground: Diagnosing the Learning Enabled. Pacific Northwest Institute on Special Education and the Law.

Shapiro, E. (2006). Advanced Issues in Monitoring Student Progress in Reading. Presentation for Pennsylvania Training and Technical Assistance Network on on RTI.

Tilly (Sept, 2005) Problem Identification and Analysis. Presentation at the Principal Lead Problem Solving Initiative, Highland Park, IL.