Using the Observation Survey as Your Assessment Tool: A Rationale Francisco X. Gómez-Bellengé The...

Using the Observation Survey as Your Assessment Tool:

A Rationale

Francisco X. Gómez-Bellengé

The Ohio State University

Using OS for Assessment-Francisco X. Gomez Bellenge

Assessment & RTI

• In a recent article, Richard Allington (2006) said he was skeptical of RTI if RTI meant merely the juxtaposition of commercial programs across the three tiers;

• This could lead to “curricular incoherence”• In an article titled “The Congruence of

Classroom and Remedial Reading Instruction” (Elementary School Journal, 1985), Richard Allington, Peter Johnston & Peter Afflerbach argued that any intervention had to build primarily around students’ instructional needs.


Allington cont’d

• For instance, if students struggled because they did not use effective comprehension strategies, then expanding comprehension strategy instruction in the intervention lessons would seem a good idea.

• If students struggled because they did not self-monitor, then that would become one focus of the intervention lessons.


Allington 3

• Designing effective intervention requires that a student’s specific instructional needs be identified.

• Because curricular coherence is needed across the three tiers, meeting students’ specific instructional needs can be achieved in part with a common assessment framework.


Assessment in K-2

In K-2, this assessment framework could be the Observation Survey

• It is a not for profit assessment

• Developed by Marie Clay, the founder of Reading Recovery

• It is comprised of six tasks that are real world tasks

• Administered individually by teachers


Observation Survey

• Requires teacher training and an interpretation of scores

• Recent U.S. grade 1 norms available (fall, mid-year, year-end)

• Recent New Zealand age norms available

• The Observation Survey is a criterion referenced assessment

• Measures are not normally distributed

The Six Measures of An Observation Survey of Early

Literacy Achievement


Text Reading Level

Scoring: text levels 00-02 = readiness; 3-8 = pre primer; 9-12 = primer; 14-16 = end of grade 1; 18-20 = grade 2; 22-24 = grade 3; 26-30 = grades 4-6.

Purpose: to determine an appropriate level of text difficulty and to record, using a running record, what the child does when reading continuous text

Task: to read texts representing a gradient of difficulty until the highest text level with 90% accuracy or better is determined with teacher recording text reading behaviors during the oral reading task; texts were drawn from established basal systems and have, over the years, proved to be a stable measure of reading performance.


Letter Identification

• Scoring: maximum score = 54

• Purpose: to find out what letters the child knows and the preferred mode of identification

• Task: to identify upper and lower case letters and conventional print forms of ‘a’ and ‘g’


Ohio Word Test


• Purpose: to find out whether the child is building up a personal resource of reading vocabulary

• Task: to read a list of 20 high-frequency words


Concepts About Print


• Purpose: to find out what the child has learned about the way spoken language is put into print

• Task: to perform a variety of tasks during book reading by the teacher


Writing Vocabulary

• Scoring: count of words in a 10 minute time limit

• Purpose: to find out whether the child is building a personal resource of words that are known and that can be written in every detail

• Task: to write all known words in 10 minutes


Hearing and Recording Sounds in Words


• Purpose: to assess phonemic awareness by determining how well the child represents the sounds of letters and clusters of letters in graphic form

• Task: to write a dictated sentence, with credit for every sound correctly represented


OS & CBM

• The Observation Survey is similar to Curriculum-Based Measurement:– Well-designed standardized assessments– Established reliability and validity– Designed to inform instruction– Not for profit– Not normally distributed– Grounded in substantial scientific peer-review

research


DIBELS

• Developed by a group at the University of Oregon

• Also teacher administered individually

• Also meant to inform instructional decision-making but not inform instruction itself, e.g.,

• nonsense word and speeded tasks are not related to real-world classroom tasks


DIBELS Measures


DIBELS 2

• Minimum training involved

• No interpretation of scores-decision criteria set in advance

• Developers of DIBELS + writers of Reading First legislation + advisers to writers of state grants + grant reviewers=

Conflict of interest, Inspector General Report


Decision criteria

Assessment and Identification of First-Grade Students at Risk:

Correlating the Dynamic Indicators of Basic Early Literacy Skills and An Observation Survey

of Early Literacy Achievement


Correlating the OS & DIBELS

• The Iowa Test of Basic Skills is one of the most respected standardized assessments

• DIBELS creators demonstrated worth of their assessment by comparing to ITBS

• CBM developers compare their assessments to ITBS

• OS researchers have also looked at ITBS


OS & ITBS

Study found that OS & ITBS

• Both identify same students as low readers

• Measures correlate moderately to highly

• Progress for Reading Recovery students evident on both the Observation Survey & the Iowa Test of Basic Skills


OS & DIBELS: Theory of Learning

• DIBELS and OS assessment are based on alternate theoretical constructs and ways of defining the processes of early literacy acquisition

• With DIBELS, fluency and accuracy are proxies for degree of learning

• Underlying theory is that skills acquisition equates with emergence of reading


OS & RR Theory of Learning

• The OS was designed to assess a range of items and behaviors accounting for the complexity of learning how to read in first grade

• During the earliest stages of literacy acquisition in particular, speed of response may mask important developmental markers of students’ progress.

• Emphasis is placed on strategic behavior used by beginning literacy learners


OS & DIBELS

• The measures correlate reasonably well, especially when you would expect them to:

OS Measure DIBELS Measure

Letter Identification Letter Naming Fluency

Text Reading Level Oral Reading Fluency

Hearing & Recording Sounds in Words

Nonsense Word Fluency & Phoneme Segmentation Fluency


OS & DIBELS/2

• For Reading Recovery students, both the OS & DIBELS show progress during the intervention

• Big difference: Only ½ of students identified as at-risk by OS & Reading Recovery are identified as at-risk using DIBELS criteria for “needs substantial intervention” classification.


Why the discrepancy

• Students who do well on speeded tasks of specific skills may not have necessarily developed the ability to process text and extract meaning from it.

• These students will do well on DIBELS & on some tasks of the OS

• If not identified as at-risk early, problems may develop later (e.g., grade 3)


Scott Paris, RRQ, 2005

Theories about reading have neglected basic differences in the developmental trajectories of skills related to reading. … that some reading skills, such as learning the letters of the alphabet, are constrained to small sets of knowledge that are mastered in relatively brief periods of development. In contrast, other skills, such as vocabulary, are unconstrained by the knowledge to be acquired or the duration of learning.


Early Literacy Development

• Scott Paris argues that when looking at a specific skill, (e.g. letter identification) early in a child’s life there will be no progress on that measure, followed by a period of rapid progress, after which no progress occurs because the skills has been acquired.


Birth age 5 age 7

100%

0%


Early Literacy Development/2

• Before and after the skill of task is being learned, students either know it or don’t know it; there is little variation and therefore, the distribution of scores would be non-normal; this is due to the nature of the task, not a flaw in its assessment.


Non-normal distributionsSpring Letter IdentificationFall Text Reading Level


DIBELS & Normal Distribution

• DIBELS forces a normal distribution of scores even when it should no be. This is done by either speeding the task or creating a task that is not the actual reading task (e.g., nonsense word fluency)

• Normal distribution of scores is sometimes confused with being standardized, reliable, or valid.


Normal Distribution

• Normal distribution is just that- it does not mean a task is reliable, valid or standardized

• Normal distribution is required under conventional statistics to measure effect of a treatment-more modern techniques obviate this need

Letter Identification Task for Adults:

What’s a reasonable time per letter?


L J I T


L J I T


And another thing …

• Scott Paris’s article, while brilliant, misses an important point: While the acquisition of specific skills does not occur evenly over time, the emergence of literacy, of learning to extract meaning from text, is largely unobserved and not directly depended on discrete skills acquisition


Assessment

• Any assessment system must accommodate the complexity of the process of literacy acquisition and the change that occurs over time.

RTI & Reading Recovery

Statistics &

Conceptual Framework




Reading Recovery National 05-06


Parting Thoughts

• The Observation Survey is a valid and reliable assessment tool

• It is well suited to be used within an RTI framework

• DIBELS is not mandated; selection has been severely criticized in studies

• You can’t buy a coherent RTI off the shelf• Literacy & Sped folks need to talk to each

other

Date post:	18-Jan-2016
Category:	Documents
Upload:	david-singleton
View:	218 times
Download:	0 times

Using the Observation Survey as Your Assessment Tool: A Rationale Francisco X. Gómez-Bellengé The...

Documents