READABILITY FORMULAS AND TEXT COMPLEXITY

Elfrieda H. Hiebert TextProject & University of California, Santa Cruz

Paper presented at the annual meeting of the Literacy Research Association, San Diego, CA, November 30, 2012.

Text Complexity Model

Reader and Task

1.  Quantitative measures – readability and other scores of text complexity often best measured by computer software.

2.  Qualitative measures – levels of meaning, structure, language conventionality and clarity, and knowledge demands often best measured by an attentive human reader.

3.  Reader and Task considerations – background knowledge of reader, motivation, interests, and complexity generated by tasks often best determined by educators employing their professional judgment.

2 Elfrieda H. Hiebert www.textproject.org

3 Elfrieda H. Hiebert www.textproject.org

While the prior two elements of the model focus on the inherent complexity of text, variables specific to particular readers (such as motivation, knowledge, and experiences) and to particular tasks (such as purpose and the complexity of the task assigned and the questions posed) must also be considered when determining whether a text is appropriate for a given student. Such assessments are best made by teachers employing their professional judgment, experience, and knowledge of their students and the subject. (CCSS Appendix A, page 4)

4 Elfrieda H. Hiebert www.textproject.org

Hiebert, E.H. (October, 2010). Anchoring Text Difficulty for the 21st Century: A Comparison of the Exemplars from the National Assessment of Educational Assessment and the Common Core State Standards (Reading Research Report 10.02). Santa Cruz, CA: TextProject, Inc.

The CCSS Staircase of Text Complexity

5 Elfrieda H. Hiebert www.textproject.org

Assumption: Guidelines for grade-level text requirements for instructional texts and for high-stakes tests can be based on readability formulas. • Evidence of the emphasis on readability formulas among

the CCSS writers is the new addendum to the Common Core State Standards: Nelson, Perfetti, Liben, & Liben (2012)

6 Elfrieda H. Hiebert www.textproject.org

Purposes of this Presentation • Comparison of 1st and 2nd generation readability systems • Overview of research on relationship of syntax and

vocabulary to comprehension • Summary of two analyses considering relationship of

syntax and vocabulary to Lexiles

7 Elfrieda H. Hiebert www.textproject.org

1st & 2nd Generation Readability Formulas

1st generation: Dale-Chall: Sarah, Plain & Tall

•  # of words not found on Dale-Chall Word List: 56

• Percent of words not found on Dale-Chall Word List: 9%

• Grade Level: Grades 5-6

2nd generation: Lexile: Sarah, Plain & Tall

•  Lexile: 510L (beg. of Gr. 2-3 band) •  9.04 Mean Sentence Length •  3.76 Mean Log Word Frequency

8 Elfrieda H. Hiebert www.textproject.org

Research on Relationship of Syntax & Vocabulary to Comprehension:

(a) Correlational •  Early studies: Davis, 1944, Botzum, 1951; Davis, 1968; Clark

1972: vocabulary factor loadings ranged from .41 to .93.

•  Pasquarella, Gottardo, & Grant (2012): For native English learners, only vocabulary predicted reading comprehension.

•  Guo, Roehrig, & Williams (2011); Vocabulary knowledge made an independent contribution to reading comprehension above and beyond those of morphologic and syntactic awareness; morphologic awareness affected reading comprehension directly; syntactic awareness predicted reading comprehension not only directly but also indirectly via vocabulary knowledge.

9 Elfrieda H. Hiebert www.textproject.org

•Nelson, Perfetti, Liben, & Liben (2012) Lexile (Pearson r)

CCSS Exemplars Informational* .61 CCSS Exemplars Narrative* .29 State Tests Grades 3-5 .30 State Tests Grades 6-8 .26 State Tests 9-11 .22 GM All Grade Levels (Student Performances)

.69

SAT9 Grades 3-5 (Student Performances)

.41

SAT9 Grades 6-8 (Student Performances

.24

SAT9 Grades 9-11 (Student Performances

.08

*To Grade Band (with 2 to 3 grade span) 10 Elfrieda H. Hiebert www.textproject.org

Research on Relationship of Syntax & Vocabulary to Comprehension: (b) Descriptive

Deane et al.(2006): When Lexile scores are used to assess readability levels and the Educators’ Word Frequency Index is used to measure word familiarity, third-and sixth-grade texts exhibit similar levels of vocabulary demand.

11 Elfrieda H. Hiebert www.textproject.org

Research on Relationship of Syntax & Vocabulary to Comprehension: (c) Experimental Manipulations

Arya, Hiebert, & Pearson (2012): 16 expository texts represented differences in syntactic and lexical complexity across four science-related topics. Lexical complexity had a significant impact on students’ comprehension on two of the four topics. Comprehension performance was not influenced by the syntactic complexity of texts, regardless of topic.

12 Elfrieda H. Hiebert www.textproject.org

Two analyses • Purpose was to determine the relationship of syntax and

vocabulary to Lexiles: correlational and also as a function of manipulations of semantic and syntactic features

13 Elfrieda H. Hiebert www.textproject.org

Analysis 1: Correlations • Database consisted of 1,518 texts.

•  Texts were distributed across 4 Lexile groups, with genre (Narrative & Expository) distributed almost evenly (Nar: 732; Ex: 786) across the sample

Lexile Narrative Expository Total

Primary (1-400)

244 57 301

Elementary (401-800)

310 337 647

Secondary (801-1200)

146 311 457

College (1201+)

32 81 113

14 Elfrieda H. Hiebert www.textproject.org

By Length Length Narrative Expository Narrative +

Expository Very short (50-200 words)

152 125 277

Short (201-500 words)

268 208 476

Moderate (501-1,000 words)

167 235 402

Long (1,000-2,000 words)

144 219 363

15 Elfrieda H. Hiebert www.textproject.org

By Source Sources # Assessment 235 Beginning 240 Core Content 342 Core Reading 244 Exemplar 208 Literature 144 Magazine 105

16 Elfrieda H. Hiebert www.textproject.org

Correlations of Syntax & Vocabulary to Lexile

•  Syntax: .939

•  Vocabulary (Mean Log Word Frequency): -.529

17 Elfrieda H. Hiebert www.textproject.org

Analysis 2: Manipulations of Texts • A text representing elementary, secondary, and college

grade bands was chosen •  For each variable (i.e., sentence length, vocabulary

frequency): •  5 changes were made to a text (text analyzed) •  5 additional changes were made to a text (text analyzed).

18 Elfrieda H. Hiebert www.textproject.org

Illustrations of manipulations Sentence Length •  Original sentence: PEOPLE, Golly! I suppose if people ever

learn to fly like any common hedge-sparrow we shall never hear the end of it!

•  First set of changes: PEOPLE! Golly! I suppose if people ever learn to fly like any common hedge-sparrow we shall never hear the end of it!

•  Second set of changes: He said the old man didn't talk Greek right, . And He couldn't stand listening to him teach the language wrong.

19 Elfrieda H. Hiebert www.textproject.org

Illustrations of manipulations Vocabulary Frequency

•  Original Sentence: you had to wait for the widow to tuck down her head and grumble a little over the victuals, though there warn't really anything the matter with them, that is, nothing only everything was cooked by itself.

•  1st set of changes: You had to wait for the widow to tuck down her head and grumble a little over the victuals, though there warn't wasn’t really anything the matter with them, that is, nothing only everything was cooked by itself.

•  2nd set of Changes: You had to wait for the widow to tuck down her head and grumble a little over the victuals, food though there wasn't really anything the matter with them, that is, nothing only everything was cooked by itself.

20 Elfrieda H. Hiebert www.textproject.org

Analysis 2: Manipulations of Texts—Sentence Length

Original 5 sentence divisions

Change in Lexile

10 sentence divisions

Change in Lexile

Elementary (Dr. Dolittle)

600 (10.4) 500 (9.1) -100 460 (8.6) -10

Secondary (Adventures of Huck Finn)

980 (18.7) 860 (15.1) -120 730 (13.7) -130

College (Gulliver’s Travels)

2290 (55.38)

1970 (49.22)

-320 1680 (36.92)

-290

21 Elfrieda H. Hiebert www.textproject.org

Analysis 2: Manipulations of Texts—Vocabulary Frequency

Original 5 Vocabulary Frequency Changes

Change in Lexile

10 Vocabulary Frequency Changes

Change in Lexile

Elementary (Dr. Dolittle)

600 3.80 590 (3.78)

-10 590 (3.79)

0

Secondary (Adventures of Huck Finn)

980 3.92 970 (3.94)

-10 960 (3.96)

-10

College (Gulliver’s Travels)

2290 2.61 2270 (2.66)

-20 2260 (2.69

-10

22 Elfrieda H. Hiebert www.textproject.org

Conclusions •  The cautions of Becoming a Nation of Readers are still

applicable: •  “The Commission urges those who buy books and those who

write and edit them to supplement analyses using readabilty formulas with analyses of the deeper factors that are essential for quality.” (p. 65)

•  The 2nd generation of readability systems, of which the Lexile Framework is the most prominent, appear to represent syntax more than vocabulary differences.

23 Elfrieda H. Hiebert www.textproject.org

Conclusions (continued) •  The analyses in this project suggest that caution needs to

be paid in use of readability systems for the designation of texts, especially those for assessments.

• A focus needs to be placed on teachers’ ability to identify features of texts that “grow their students’ capacity in reading.”

24 Elfrieda H. Hiebert www.textproject.org

WWW.TEXTPROJECT.ORG