Assessing the Cognitive Demands of a Century of Reading ... · from text and integrating it with...

Assessing the Cognitive Demands ofa Century of Reading Curricula:An Analysis of Reading Text and

Comprehension Tasks From 1910 to 2000

Robert J. StevensXiaofei Lu

David P. BakerPennsylvania State University

Melissa N. RayNorthern Illinois University

Sarah A. EckertAgnes Irwin SchoolDavid A. Gamson

Pennsylvania State University

This research investigated the cognitive demands of reading curricula from1910 to 2000. We considered both the nature of the text used and the com-prehension tasks asked of students in determining the cognitive demands ofthe curricula. Contrary to the common assumption of a trend of simplifica-tion of the texts and comprehension tasks in third- and sixth-grade curric-ula, the results indicate that curricular complexity declined early in the cen-tury and leveled off over the middle decades but has notably increased sincethe 1970s, particularly for the third-grade curricula.

KEYWORDS: reading curricula, historical analyses, cognitive complexity

Learning to read and comprehend information presented in text is animportant skill that is the foundation for students’ future academic activ-

ities. Most of the instruction on reading comprehension occurs in elementaryschool, and reading instruction during the 20th century, and thus far in the21st century, has been dominated by published reading curricula, known asbasal reading series (Chall & Squires, 1991; Smith, 1965; Venezky, 1987). Thegoal of this project was to investigate the historical changes in the cognitivedemands made on students by the reading curriculum materials.

Theoretical developments in learning resulting from the cognitive revo-lution of the 1970s and the expansion of research on reading has led to

American Educational Research Journal

June 2015, Vol. 52, No. 3, pp. 582–617

DOI: 10.3102/0002831215573531

� 2015 AERA. http://aerj.aera.net

at Aarhus Universitets Biblioteker / Aarhus University Libraries on July 19, 2015http://aerj.aera.netDownloaded from

http://aerj.aera.net

a better understanding of the complexity of reading comprehension. Thisconceptualization traces its theoretical roots to Bartlett’s (1932) ideas of con-struction of meaning as an interaction of learner characteristics, learningactivity, learning materials, and criterion tasks. Since that time, theory andbasic research have helped develop our understanding of reading compre-hension as a relatively complex process of meaning construction. Readingcomprehension is currently conceived of as the interaction of the text, thereader (e.g., prior knowledge), the criterion task, and the broader sociocul-tural context (e.g., instructional context, sociocultural group) (Kintsch &Kintsch, 2005; Pearson & Fielding, 1991; Pressley, 2000; RAND ReadingStudy Group, 2002; Rapp, van den Broek, McMaster, Kendeou, & Espin,2007; Spiro, 1980). Perhaps Kintsch and Kintsch (2005) conceptualized itmost clearly as a ‘‘delicate interaction’’ of processes of taking meaningfrom text and integrating it with one’s prior knowledge under the influenceof a ‘‘multitude of contextual constraints’’ (p. 71).

Each of these four elements (learner, activities, materials, and criteriontasks) are multifaceted, adding to the complexity of the variables involvedin reading comprehension. For example, considerations of the readerinclude variables like the reader’s prior knowledge, cognitive abilities, skillsor strategies, motivation, and attitudes (e.g., RAND Reading Study Group,

ROBERT J. STEVENS, PhD, is a professor and chair of educational psychology in theDepartment of Educational Psychology, Counseling, and Special Education at thePennsylvania State University, 141 CEDAR Building, University Park, PA 16802;e-mail: [email protected]. He does research in reading comprehension, effective literacyinstruction, and teacher effectiveness, with particular interest in the schooling of at-risk students.

XIAOFEI LU, PhD, is Gil Watz Early Career Professor in Language and Linguistics andassociate professor of applied linguistics at the Pennsylvania State University. Hisresearch interests are primarily in corpus linguistics, computational linguistics, andintelligent computer-assisted language learning.

DAVID P. BAKER is professor of education and sociology in the Departments ofEducation Policy Studies and sociology at the Pennsylvania State University. Hedoes research on the impact of education on social institutions, including healthand demography, labor markets, cognitive abilities in populations, and politicalmobilization.

MELISSA N. RAY, PhD, is a research scientist at the Center for the Interdisciplinary Studyof Language and Literacy and Department of Psychology at Northern IllinoisUniversity. Her research is focused on individual differences in reading comprehen-sion, the relationship between text structure and comprehension, and the role of lit-eracy in college readiness.

SARAH A. ECKERT, PhD, teaches history at the Agnes Irwin School in Bryn Mawr,Pennsylvania.

DAVID A. GAMSON, PhD, is an assistant professor of education in the Department ofEducational Policy Studies at the Pennsylvania State University. His research focuseson the history of American education.

Cognitive Demands of a Century of Reading Curricula

583 at Aarhus Universitets Biblioteker / Aarhus University Libraries on July 19, 2015http://aerj.aera.netDownloaded from


2002; Rapp et al., 2007; Spiro, 1980). All of these variables are important andplay a role in a reader’s construction of meaning.

This project investigated the influence over time of two of these fourcomponents of reading comprehension, reading texts and reading compre-hension tasks, in the context of reading instruction. An investigation of read-ing curricula over the past century provides documentation of the evolutionof reading materials commonly used in American schools. Reading curricu-lum materials are, and have been, a driving influence in reading instruction.As such, these materials influence the development of children’s cognitiveabilities. For example, in the mid 1960s nearly 95% of all teachers used basalreading series in their classrooms (Chall & Squires, 1991). While teachersmay currently use a wider variety of reading materials (e.g., children’s liter-ature) in classroom instruction, the influence of commercial reading seriesremains strong. The National Assessment of Educational Progress (NAEP)reported in 1992 that 36% of fourth-grade teachers said they relied exclu-sively on basal readers, and 49% said they used basal readers supplementedwith trade books for their reading instruction (Wade & Moje, 2000). Morerecent analyses of classroom reading instruction found a similar relianceon basal reading series (Brenner & Hiebert, 2010). These responses suggestbasal reading curricula play a central role in reading instruction for a greatmajority of fourth-grade teachers.

Earlier historical research (e.g., Chall, 1967; Chall & Squires, 1991; Smith,1965; Venezky, 1991) has documented changes in the goals and implemen-tation of reading instruction as well as the evolution of reading curricula inresponse to the changing goals of reading instruction in schools. For exam-ple, early in the 20th century, reading instruction focused on elocution, andmuch of reading instruction involved reading orally (Smith, 1965; Venezky,1987). Reading research by Huey (1908), Thorndike (1917), and others ledto an important reconceptualization of reading as the process of translatingprinted words into meaning. As this conceptualization became more preva-lent, it resulted in important changes in how teachers taught reading, whatstudents read, and what activities students did in response to what theyhad read.

The earlier historical accounts document similar theory- and research-based influences. Changes in the curriculum materials included increasingthe amount of authentic text and informational text students read and askingcomprehension questions requiring deep rather than surface-level compre-hension of those texts. However, most of the historical research in readinghas been descriptive (e.g., Smith, 1965; Venezky, 1987). Only Chall,Conrad, and Harris (1977) and Chall and Squires (1991) report quantitativeresearch studying changes in vocabulary difficulty and readability in basalreading series over segments of the 20th century. This study will expandon and attempt to quantify relationships described in the previous researchby looking at text difficulty and the demands of comprehension tasks to get

Stevens et al.



a more complete understanding of the systematic changes in reading curric-ula. This research will also contextualize these curricular changes in the the-ory of cognition (e.g., cognitive complexity and cognitive load of readingmaterials and comprehension tasks) as a way to understand the cognitivecomplexity of the materials and the changing cognitive demands curricularchanges make on students.

Are Reading Texts Getting Simpler?

Some researchers have argued that ‘‘the materials teachers use for read-ing instruction today are considerably different from those that were used forreading instruction for nearly the entire twentieth century’’ (Martinez &McGee, 2000, p. 154). Some recent research suggests that there has beena decline in the difficulty of textbooks used in school that has resulted ina decline in verbal SAT scores (e.g., Hayes, Wolfer, & Wolfe, 1996) anda lack of preparation for later work and academic endeavors (Williamson,2006). Hayes et al. (1996) conducted a comprehensive analysis of 800 text-books (from a variety of content areas) published from 1919 to 1991 andused in elementary through high school. They grouped curricula into threelarge historical periods: 1919–1945, 1946–1962, and 1963–1991. Hayes et al.did not look at the readability of the text because ‘‘psycholinguists havemany strong reservations about these measures’’ (p. 494), suggesting thatreadability estimates lacked accuracy particularly when comparing textacross time. Instead, they looked at the lexical difficulty (i.e., the level of dif-ficulty of the vocabulary) of the text as their sole measure of text difficulty.The overall conclusion was that ‘‘a major simplification of schoolbooksoccurred after World War II and that, beyond third grade, current levelshave never been lower in American history’’ (p. 505). Specifically, Hayeset al. found that (a) the most difficult basal readers were generally publishedprior to 1918, (b) significant simplification of lexical difficulty occurred afterWorld War I, (c) after World War II readers again became simpler, and (d)today’s sixth-, seventh-, and eighth-grade reading texts are simpler thanfifth-grade basal readers prior to World War II. The authors decry this simpli-fication as a leading cause for declining SAT verbal scores.

Other authors (e.g., Adams, 2010–2011) have also commented on thedeclining vocabulary difficulty in school texts. Using the findings of Hayeset al. (1996), Adams (2010–2011) links the vocabulary simplification toresults of the National Assessment of Educational Progress. Noting thatHayes et al. found small increases in primary grade reading curricula vocab-ulary since 1963, Adams suggests that the vocabulary difficulty increaserelates to increases in NAEP reading performance for 9- and 13-year-oldssince 1971. However, she notes there is a simplification of vocabulary in cur-ricula used in fourth-grade since World War II. Further, Adams suggests that




this vocabulary simplification may be the cause of flat NAEP reading perfor-mance for 17-year-old students.

Recent Analyses of Text in Reading Curricula

Previously published research (Gamson, Lu, & Eckert, 2013) from thisproject has shown that the mean readability level and mean lexical difficultydropped through the early part of the century to a low point in the 1950sthrough 1970s. However, particularly for third-grade text, there have beenincreases in average readability and lexical difficulty over the past three dec-ades. Third-grade lexical difficulty measures (LEX and word frequency band)scores for texts published in the 2000s are higher than at any other time inthe century. Reading texts for sixth grade do not show an upturn in the latterhalf of the century, but the measures of lexical difficulty have increased inrecent decades. Similarly, this research indicates that readability estimateshave increased in recent decades for third-grade texts while remaining fairlystable for the past 50 years for sixth-grade texts.

A goal of this article is to expand on these text analyses for third- andsixth-grade reading materials by examining additional measures and dimen-sions of their linguistic complexity. We also will consider the cognitivedemands made of students by the comprehension tasks in the curricula.

Research on Reading Comprehension Tasks

As we discussed earlier, there is more to the complexity of reading thanthe text. What students are asked to do in comprehending and using theinformation in the text also affects the cognitive demands of reading instruc-tion. There is a rich research literature that has investigated the content anduse of reading curricula (e.g., Beck, McKeown, McCaslin, & Burkes, 1979;Brenner & Hiebert, 2010; Durkin, 1981; Smith, 1965). Our goal is to add tothis literature with a historical investigation describing the changes in read-ing curricula over the 20th century. We will categorize reading comprehen-sion tasks to track changes in the types of comprehension skills emphasized(e.g., literal vs. inferential comprehension).

Project Goals

The goal of this project was to analyze and describe the changes in thecomplexity of reading curricula text and the comprehension tasks asked ofstudents in those curricula. We looked at the curricula for each decade,rather than longer time periods used in previous research (e.g., Hayes etal., 1996), so our analyses would be more sensitive to changes that occurin reading materials that tend to have new copyrights, and hence contentrevisions, approximately every 5 years.

Stevens et al.



Method

Selection and Acquisition of the Reading Curricula

Our goal was to locate a sample of third- and sixth-grade basal readingtextbooks from 1900 to 2008 that represented the textbook market. In orderto collect the textbooks for this study, we began by casting a wide net inorder to identify as many third- and sixth-grade reading books from 1900to 2008 as possible. During the initial phase of collection we identified majorpublishers (e.g., Ginn & Co., Houghton Mifflin, MacMillan, and ScottForesman) and attempted to follow the textbook series issued by each pub-lisher throughout the century. After textbook publication was tracked formajor publishers, we began a more purposeful search, using state adoptionrecords from Texas and California as well as from a wide variety of profes-sional and academic publications. Finally, we used Bowker’s (1948–2010)Books in Print as a source to identify additional reading curricula and helpfill in any gaps in our textbook sample.

Textbook Sample for Text Difficulty Analysis

The sample used for the text difficulty analysis included texts (i.e., indi-vidual reading units) from 187 third-grade and 71 sixth-grade reading text-books published between 1905 and 2004. We selected third grade becauseit is a critical year in developing students’ comprehension skills throughreading instruction and is prior to the documented ‘‘fourth grade slump’’(Chall, Jacobs, & Baldwin, 1990), when increased demands in text complex-ity and comprehension tasks overwhelm some students. As such, third-gradereading instruction is critical to preparing students for the increased empha-sis on reading in content areas typical in middle and high school. Weselected sixth-grade reading curricula as text to represent the changingemphasis of the role of reading in the curricula in middle school. Duringmiddle school, students encounter a wider variety of types of text withincreasing complexity and increased demands on their comprehension.There is an important shift in reading with increased emphasis on students’learning the content presented in text, particularly in content areas like socialstudies and science. Investigating reading curricula used in sixth grade mayprovide evidence of the degree to which curricula provide instructionrelated to demands found in content area reading.

The textbooks were scanned and digitized using OmniPage, an opticalcharacter recognition program. Each text was manually checked for accuracyand labeled with relevant meta-information, such as publisher, print year,grade level, start page, end page, and genre. Table 1 summarizes the detailsof the data set for third- and sixth-grade texts. To ensure the reliability of thetext difficulty measures applied, only texts with at least 100 words wereincluded. Altogether, there were 5,259 texts with a total of over 5 million




words in the third-grade data set and 2,782 texts and over 4 million words inthe sixth-grade data set. Textbooks were grouped into 10 time periods, eachof which covers one decade, beginning 5 years prior to the beginning ofa decade and ending 4 years after the turn of the decade.

Textbook Sample for Analysis of Comprehension Tasks

From the aforementioned sample of texts, we selected a subsample touse in the analysis of comprehension tasks. For each decade we selectedthree widely used basal reading series at both third- and sixth-grade levels.These series spanned the decades from the 1920s to the 2000s. While wewere able to locate student readers for the decades prior to 1920 to use inthe linguistic analysis of the text students read, we were unable to locatea sufficient number of teacher’s manuals, which at that time contained thecomprehension questions for use in the comprehension task analyses.Hence, the comprehension task analyses included 54 textbooks in third

Table 1

Details of the Third- and Sixth-Grade Data Sets in Text Analyses

Grade Decade Years Books Texts Words Mean Length SD

3 1910 1905–1914 10 489 346,401 708.39 795.00

1920 1915–1924 7 298 216,670 727.08 673.34

1930 1925–1934 16 638 471,188 738.54 631.13

1940 1935–1944 10 308 322,083 1,045.72 615.41

1950 1945–1954 12 384 400,946 1,044.13 638.58

1960 1955–1964 13 362 368,677 1,018.44 683.29

1970 1965–1974 35 857 982,683 1,146.65 735.49

1980 1975–1984 33 803 831,931 1,036.03 711.29

1990 1985–1994 25 566 618,952 1,093.55 727.96

2000 1995–2004 26 554 489,526 883.62 685.59

All 1905–2004 187 5,259 5,049,057 960.08 716.38

6 1910 1905–1914 3 133 158,503 1,191.75 1,086.15

1920 1915–1924 5 252 444,822 1,765.17 2,755.04

1930 1925–1934 6 357 427,087 1,196.32 1,294.04

1940 1935–1944 6 272 486,467 1,788.48 1,491.61

1950 1945–1954 4 160 306,149 1,913.43 1,404.17

1960 1955–1964 8 301 537,425 1,785.47 981.39

1970 1965–1974 8 262 549,843 2,098.64 2,002.21

1980 1975–1984 14 456 798,805 1,751.77 2,340.00

1990 1985–1994 13 351 816,438 2,326.03 2,454.29

2000 1995–2004 4 238 413,919 1,739.16 1,664.93

All 1905–2004 71 2,782 4,939,458 1,775.51 1,940.12

Stevens et al.



grade (because there are two levels, 3.1 and 3.2 in third grade) and 27 text-books in sixth grade.

Sampling Stories for Analysis of Comprehension Tasks

For each of the curriculum and grades selected, we sampled eight storiesto use in the comprehension tasks analyses. In order to systematically sam-ple all of the curricula in a similar fashion, we selected the first four stories,and starting in the middle of the book we selected four more stories.

Only selections longer than one page were used in these analyses. Wefound that some curricula periodically had one-page selections like‘‘About the Author.’’ Beyond the one-page selections, curricula tended tohave longer, multiple page narratives or informational selections. Our ratio-nale for using selections that were longer than one page was that teacherswere likely to allocate much more instructional time to longer selections,and as a result, those selections would have more instructional impact.

Text Difficulty Analysis

Lexical Sophistication

Lexical sophistication measures the degree of sophistication of thewords in a text. In readability formulas, two approaches have been usedto define sophisticated words, one based on word length or syllable struc-ture, the other based on word frequency or some notion of word familiarity.Admittedly, what constitutes familiar words may vary among readers of eventhe same age. However, previous studies have reported experimental evi-dence that word frequency has a significant effect on reading comprehen-sion (Marks, Doctorow, & Wittrock, 1974; McGregor, 1989). Gamson et al.(2013) reported results of lexical sophistication using two measures, forexample, LEX (which was originally proposed by Hayes et al., 1996) andword frequency band scores. In this article, we examine lexical sophistica-tion using two additional frequency-based measures. We consider a wordas sophisticated if it is not among the 2,000 most frequent words in theAmerican National Corpus (ANC) (Reppen & Ide, 2004), a massive corpusof American English texts of all genres and transcribed speech data with22 million words in its current release. Specifically, we consider the follow-ing two lexical sophistication ratios: sophisticated word ratio (i.e., the pro-portion of words in the text that are sophisticated) and sophisticated wordtype ratio (i.e., the proportion of unique words in the text that are sophisti-cated). These ratios were computed for each text using the lexical complex-ity analyzer developed by Lu (2012).




Lexical Diversity

Lexical diversity measures the range of vocabulary used in a text.Malvern, Richards, Chipere, and Duran (2004) provided a thorough discus-sion of the many measures of lexical diversity that exist. A widely used mea-sure of lexical diversity is type-token ratio (TTR), for example, the ratio ofthe number of word types or unique words to the total number of wordsin a text (Templin, 1957). This measure has been shown to decrease asthe length of the text increases (Richards, 1987), and several transformationsof the ratio have been proposed to minimize the sample size effect. We con-sider both the original TTR and one of its transformation here, namely, RootTTR (RTTR; Guiraud, 1960). RTTR adjusts the denominator in the originalTTR to the square root of the number of words. These measures were againcomputed for each text in the data set using the lexical complexity analyzerdeveloped by Lu (2012).

Syntactic Complexity

Another aspect of text difficulty considered here is syntactic complexity.As discussed previously, most readability formulas measure syntactic com-plexity using mean length of sentence. While this measure is intuitively use-ful, it is insensitive to structural differences among sentences. Many othersyntactic complexity measures proposed for child language acquisitionresearch rank syntactic constructions based on patterns of syntactic develop-ment or the frequency of occurrence of these constructions in naturalspeech, such as Developmental Level (D-Level; Covington, He, Brown,Nacxi, & Brown, 2006; Rosenberg & Abbeduto, 1987), DevelopmentalSentence Scoring (DSS; Lee, 1974), and Index of Productive Syntax (IPSyn;Scarborough, 1990). There is also a family of metrics that quantify the pro-cessing demand of different types of sentence structures, such as Yngvedepth (Yngve, 1960).

We adopt the D-Level scale for measuring syntactic complexity here, fortwo reasons. First, as a measure based on child language acquisitionresearch, it has excellent validity (Covington et al., 2006). Second, it hasbeen shown by psycholinguistic experiments to be a more adequate indexof sentence comprehension and recall than many other metrics (Cheung &Kemper, 1992). Originally proposed by Rosenberg and Abbeduto (1987)and subsequently revised by Covington et al. (2006), this scale classifiesa sentence into one of eight increasingly complex developmental levelsbased on its structure, as summarized in Table 2. The D-Level scores forthe sentences in each text were computed using the Developmental LevelAnalyzer (Lu, 2009). We reported two indices based on the D-Level analysishere, namely, the proportion of complex sentences (i.e., sentences in Levels1 to 7 defined in Table 2) in each text and the average D-Level score for eachtext.

Stevens et al.



Cognitive Demands of Comprehension Tasks

Comprehension tasks (i.e., questions) may create greater or lesser cog-nitive demands on the reader. The questions may require readers to processvarying amounts of textual information; they may also ask readers to process

Table 2

Convington, He, Brown, Nacxi, and Brown (2006)

Revised Developmental Level Scale

Level Description Example

0 Simple sentences, including questions John cried.

Sentences with auxiliaries and semi-

auxiliaries

This has solved it.

Simple elliptical (incomplete) sentences John did.

1 Infinitive or -ing complement with

same subject as main clause

Try to smile.

2 Conjoined noun phrases in subject

position

John and Mary left.

Sentences conjoined with

a coordinating conjunction

I came early but Peter arrived late.

Conjoined verbal, adjectival, or

adverbial constructions

He sang and jumped.

3 Relative or appositional clause

modifying object of main verb

John scolded the boy who stole his

bicycle.

Nominalization in object position I understand his rejection of the offer.

Finite clause as object of main verb John knew that Mary was angry.

Subject extraposition It was hard for John to tell Mary.

Raising John seems to Mary to be happy.

4 Non-\finite complement with its own

understood subject

I saw him walking away.

Comparative with object of comparison John is older than Mary.

5 Sentences joined by a subordinating

conjunction

They won’t play if it rains.

Nonfinite clauses in adjunct position Having tried both, I prefer the second

one.

6 Relative or appositional clause

modifying subject of main verb

The man who cleans the room left

early.

Embedded clause serving as subject of

main verb

For John to have left Mary was

surprising.

Nominalization serving as subject of

main verb

His rejection of the offer was

unexpected.

7 More than one structure from Levels

1–6

John decided to leave Mary when he

heard that she was seeing Mark.




the information in either simpler, more literal ways or more complex ways,such as making inferences based on the text and their own prior knowledge.These two constructs, the amount of text and the type of processing of text,have an impact on the cognitive complexity of the reading instructionprovided.

Working Memory Load

One way to conceptualize the complexity of comprehension tasks is toconsider the amount of text a reader needs to process in order to respond tothe comprehension task (Perfetti, 1985). As described previously, readingcomprehension involves the process of integrating text information withone’s prior knowledge to construct meaning. Comprehension tasks putdemands on a reader’s working memory as they require the reader to retaininformation from the text in order to generate an answer. These tasks mayrequire the retention of only a sentence or as little as one idea unit or theretention of a paragraph or more containing many idea units. An idea unitcan be understood as the smallest number of words for expressing anidea or thought (Johnson, 1970; Meyer, 1975; Taylor, 1980). For example, lit-eral comprehension questions typically require information from one sen-tence or a part of a sentence, thus making fewer demands on the reader’sworking memory capacity. Other tasks may ask the reader to summarizetext, requiring an integration of information from one paragraph or manyparagraphs. The sheer number of ideas the reader must hold in workingmemory relates directly to the cognitive load of the task (Carlson,Chandler, & Sweller, 2003; Mayer, 2002; Perfetti, 1985; van den Broek,McMaster, Kendeou, & Espin, 2007). When the reader can process oneidea at a time, the interactivity is low, and the resulting cognitive demandis low. When the reader must process many ideas at the same time, suchas when asked to summarize a paragraph, the interactivity is higher, andthe resulting cognitive load of the task is also higher.

To assess the amount of text needed for each comprehension task, wecounted the number of content words in the portion of text that was relatedto the answer to the comprehension question. A content word provides theinformation in the text, and a count of them is a good proxy for the numberof idea units in a portion of text. For example, in the sentence ‘‘The blackbear climbed the tree and grabbed the food’’ the words black, bear, climbed,tree, grabbed, and food are the content words, for a count of 6. In an analysisof a subsample of third-grade texts selected across decades, we found thata count of the content words in text correlated highly with the number ofidea units in the same text (r = .92). While similar to idea units, we preferredthe use of content words in this study because they were simpler to count inreading texts and easier to train coders to count reliably.

Stevens et al.



Previous research on cognitive load has also suggested that extraneousinformation adds to one’s cognitive load just as additional necessary infor-mation adds to cognitive load (e.g., Carlson et al., 2003; van den Broeket al., 2007). For example: ‘‘Washington rode into battle hoping to lead histroops to victory while charging forward on his white horse.’’ The question‘‘What color horse did Washington ride?’’ requires only ‘‘Washingtonrode . . . white horse’’ or a count of four content words. However, the words‘‘into battle hoping to lead his troops to victory while charging forward onhis’’ intervene and add to the cognitive demand of the task by requiringthat the student comprehend that text and recognize that it is not relevantfor this particular comprehension task.

As a result of these considerations, to assess the working memory loadaspect of the cognitive demand of comprehension tasks, we measured (a)the information needed to answer the question and (b) the extraneous infor-mation in the portion of text readers were asked to process for the compre-hension task.

Processing Complexity

The complexity of the processing demands of a task also adds to thecognitive load required to respond to that task (Carlson et al., 2003;Mayer, 2002; van den Broek et al., 2007). The more the reader is requiredto make connections between different pieces of information, the greaterthe processing demands on working memory. This includes connectionsbetween information from the text with prior knowledge as well as from dif-ferent segments of text. Similarly, when a reader must search long-termmemory and identify and use specific prior knowledge in combinationwith textual information to solve the comprehension task, it increases theprocessing demands on working memory and entails a greater cognitiveload. One way to think of the type of processing of text would be literalcomprehension versus inferential comprehension. Literal comprehensionsimply requires students to recall the information presented in the text,whereas inferential comprehension requires students to integrate whatthey have read with relevant prior knowledge to understand the informationin ways not explicitly stated in the text. Typically, inferential comprehensionmakes greater cognitive demands on the reader (Perfetti, 1985; van denBroek et al., 2007).

The analyses of comprehension tasks looked similar to previousresearch on the types of comprehension questions asked in reading curricula(e.g., Armbruster, Stevens, & Rosenshine, 1977). Following the pattern ofprevious research, we categorized questions into the following types:explicit detail, paraphrased detail, inference, assessing prior knowledge,character trait, opinion, supporting opinion, prediction, author’s purpose,compare and contrast, cause and effect, following directions, figurative




language, fact and opinion, interpreting illustrations, mood and setting, mainidea, sequence of events, summarization, understanding genre, and words incontext.

Level of Ideas Assessed

Comprehension questions also vary in whether they ask students abouthigh-level or low-level ideas presented in the text. Low-level ideas are thosethat require understanding of a sentence or clause in the text. Higher-levelideas require the understanding of ideas presented across multiple sentences(Haenggi & Perfetti, 1994; Magliano & Mills, 2003). Higher-level ideas aremore thematic and promote deeper processing of the text information(Goldman & Rakestraw, 2000; Kintsch & van Dijk, 1978). Questions thatask about lower-level ideas typically promote more superficial processingof the text. For example, a question that asks students to summarize whathappened in a portion of text requires the reader to process ideas frommany sentences and about a higher-level or more thematic idea from thetext. Questions that asked about explicit details in the text are likely torequire processing of a sentence or clause and be about lower-level ideas.Raters coded the questions as asking for either higher-level or lower-levelideas from the text.

Interrater Reliability

Four raters coded the constructs pertaining to the cognitive demands ofcomprehension tasks described previously. Interrater agreement was deter-mined by using 10 samples of reading materials randomly selected from thecurricula used in these analyses. Each sample included the text and twoquestions about the text. Raters counts of (a) the information needed toanswer a question and (b) the amount of extraneous information were cor-related to determine the interrater reliability. For interrater agreement, wecalculated the correlation between each pair of raters across the 20 ratings(2 questions each for 10 passages). The eight correlations ranged from r =.76 to .89 with an average of .81 on measures of information needed.Similarly, for counts of extraneous information, the correlations rangedfrom r = .78 to .87, with an average of .84.

For the coding of comprehension question category and level of ideaasked, we computed the percentage agreement between raters based ontheir ratings of the same questions, as described previously. The interrateragreement for categorizing the type of comprehension question rangedfrom 72% to 93% with a mean of 81% agreement. On the coding of the levelof ideas (high-level ideas vs. low-level ideas), the interrater agreementranged from 88% to 95% agreement, with a mean of 92%.

Stevens et al.



Results

Text Difficulty

Lexical Sophistication

Figure 1 summarizes the mean sophisticated word ratio (SWR) andsophisticated word type ratio (SWTR) of third-grade texts. One-wayANOVAs revealed significant between-decade differences in both meanSWR, F(9, 5249) = 30.592, p = .000, and SWTR, F(9, 5249) = 361.023, p =.000. The significance level for all ANOVAs used a Bonferroni adjustmentof .05/6 = .008 for the six ANOVAs conducted within each grade-level sam-ple. For SWR, a falling trend was observed from the 1910s through the 1940s,with the 1940s showing a significantly lower ratio than all other decades. Arising trend surfaced from the 1950s onwards (with the exception of the1980s), with the 2000s showing a significantly higher ratio than all other dec-ades. Note that the ratios for the 1920s–1930s period and the 1950s–1980speriod were all comparable. For SWTR, a falling trend was also observedfrom the 1910s through the 1940s. This was followed by a rising trendfrom the 1950s onwards. Following two significant increases in the 1970sand the 1990s, the 2000s achieved a significantly higher ratio than all otherdecades.

Figure 2 summarizes the mean SWR and SWTR of sixth-grade texts.Significant between-decade differences were observed for both meanSWR, F(9, 2772) = 14.572, p = .000, and SWTR, F(9, 2772) = 14.110, p =.000, as compared to an adjusted p = .008. For SWR, the 1910s and 1920shad the highest ratios, with the 1920s showing a significantly higher ratiothan all other decades. The 1930s through the 2000s had largely comparableratios, with the exception of the 1980s, which had a significantly lower ratiothan most other decades in this period. The ratio of the 2000s was signifi-cantly lower than that of only the 1920s. For SWTR, the 1920s showed a sig-nificantly higher ratio than all other decades. The 1980s showed a signifi-cantly lower ratio than most other decades, with the exception of the1950s and 2000s. No significant differences were found among the otherdecades.

Lexical Diversity

Figure 3 summarizes the mean TTR and RTTR of third-grade texts. One-way ANOVAs showed significant between-decade differences in both meanTTR, F(9, 5249) = 96.991, p = .000, and mean RTTR, F(9, 5249) = 42.362, p =.000, as compared to an adjusted p = .008. For TTR, the 1910s had a signifi-cantly higher ratio than all other decades, and the 1920s and 1930s both hadsignificantly higher ratios than the 1940s–1990s. The ratio remained low andstable between the 1940s and 1970s and then increased steadily from the




Figure 1. Mean sophisticated word ratio and sophisticated word type ratio of

third-grade reading texts. Y error bars indicate the 95% confidence interval for

the mean.

Stevens et al.



Figure 2. Mean sophisticated word ratio and sophisticated word type ratio of

sixth-grade reading texts. Y error bars indicate the 95% confidence interval for

the mean.




1980s onward. The more reliable RTTR measure, however, showed a verydifferent pattern of change. RTTR stayed low and stable from the 1910sthrough the 1960s, with no significant changes in this period. The 1970s

Figure 3. Mean type-token ratio and root type-token ratio of third-grade reading

texts. Y error bars indicate the 95% confidence interval for the mean.

Stevens et al.



and 1980s saw significant increases, and the 1990s and 2000s both achievedsignificantly higher ratios than all earlier decades.

There is some variability in the trends indicated in third grade. This maybe an artifact of the measurement used and the variability of the text in thecurricula. The TTR has been shown to be subject to a sample size effect,namely, it tends to decrease as sample size increases, whereas RTTR hasbeen shown to significantly reduce this effect, namely, it remains more stableacross texts of different lengths. The sample size effect for TTR may havegiven rise to a different pattern for TTR than that for RTTR, as there is vari-ation in the length of the texts within each decade.

Figure 4 summarizes the mean TTR and RTTR of sixth-grade reading texts.Significant between-decade differences were found for both mean TTR, F(9,2772) = 28.180, p = .000, and RTTR, F(9, 2772) = 6.000, p = .000, as comparedto an adjusted p = .008. For TTR, the 1910s had a significantly higher ratio thanthe 1940s–1990s, while the 1920s and 1930s both had a significantly higherRTTR than the 1940s–2000s. Following a significant decline in the 1940s,TTR stayed low and stable between the 1950s and 1990s. However, the2000s saw a significant increase, reaching a level that is significantly lowerthan only the 1920s and 1930s. RTTR again showed a different pattern ofchange. In general, much less variation was observed for RTTR than forTTR across the decades. On average, the later decades (1960–2000) had higherratios than the early decades (1900–1950). Notably, the 1930s had a signifi-cantly lower RTTR than either the 1940s or the decades from 1960 to 2000.The only other significant difference was found between the 1990s and the1920s, the decades with the highest and second lowest RTTR, respectively.

Syntactic Complexity

Figure 5 summarizes the mean proportion of complex sentences (PCS)and mean developmental level (D-Level) of third-grade texts. One-wayANOVAs showed significant between-decade differences for both meanPCS, F(9, 5249) = 59.628, p = .000, and D-Level, F(9, 5249) = 71.049, p =.000, as compared to an adjusted p = .008. For PCS, the 1910s had a signifi-cantly higher ratio than all later decades, the 1920s and 1930s both had sig-nificantly higher ratios than the 1940s–2000s, and the 1940s had a signifi-cantly higher ratio than the 1980s and 1990s. PCS stayed low and stable inthe 1950s–2000s, with no significant changes within this period. ForD-Level, the 1910s–1930s had significantly higher scores than the 1940s–2000s, with no significant differences found within the 1940s–2000s period.

Figure 6 summarizes the mean PCS and D-Level of sixth-grade texts.Significant between-decade differences were observed for both mean PCS,F(9, 2772) = 35.526, p = .000, and D-Level, F(9, 2772) = 49.987, p = .000,as compared to an adjusted p = .008. For PCS, the 1910s had a significantlyhigher ratio than the 1950s–2000s, the 1920s and 1930s both had significantly




higher ratios than the 1940s–2000s, and the 1940s had a significantly higherratio than the 1960s, 1980s, and 2000s. PCS stayed low and stable betweenthe 1950s and 2000s, with no significant changes within this period. D-

Figure 4. Mean type-token ration and root type-token ratio of sixth-grade reading

texts. Y error bars indicate the 95% confidence interval for the mean.

Stevens et al.



Level was again significantly higher in the 1910s–1930s than in the 1940s–2000s. The 1940s also had a significantly higher D-Level than most later

Figure 5. Mean proportion of complex sentences and developmental level of

third-grade texts. Y error bars indicate the 95% confidence interval for the mean.




decades (except the 1950s and 1970s). Within the 1950s–2000s period, the1980s showed a significantly lower D-Level than the 1970s and the 1990s.

Figure 6. Mean proportion of complex sentences and developmental level of

sixth-grade reading texts. Y error bars indicate the 95% confidence interval for

the mean.

Stevens et al.



Comprehension Tasks

Number of Comprehension Questions Asked

In considering the number of comprehension tasks, or questions stu-dents are asked, it is evident that there has been a fairly continuous increasein the number of questions throughout the century (see Table 3). In the1920s, third-grade curricula asked an average of 16 questions per story.That steadily increased up to the 2000 editions that asked an average of48 questions per story. Similarly, sixth-grade curricula in 1920 asked an aver-age of 14 questions per story, and by the 2000s, that increased to 36 ques-tions per story. These increases do not necessarily mean that teachers askedall these questions to their students, but it does show a change in emphasisby the textbook publishers that may have an influence on teachers’ behavior.

Information Needed to Answer Questions

Analyses of the average amount of information needed to answer a com-prehension question (as measured by the number of content words) showedan initial dip from 1920 to 1930 curricula, then essentially no change in the1940s and 1950s for both third- and sixth-grade materials (see Table 4). Inthe 1970s, third-grade curricula began asking questions that required theprocessing of more information (content words), and there was a gradualincrease in those demands through the 2000 curricula. In the 1930s, 1940s,and 1950s, on average, students were required to process only around 25content words to answer a typical comprehension question. By the 1990and 2000 editions, on average, students needed to process nearly twice asmuch text per question, averaging 47.1 content words per question in1990 and 49.7 in 2000.

A similar trend is observed in the sixth-grade curricula, with the gradualincrease in the cognitive demands per question beginning in the 1960 curric-ula. From a low of 32.4 content words in 1930 to a high of 68.9 content

Table 3

Number of Questions per Story: Mean and Standard Deviation, by Decade

1920 1930 1940 1950 1960 1970 1980 1990 2000

Third grade

Mean 16.2 7.1 23.0 27.8 31.0 31.3 27.8 31.8 48.5

Standard deviation 4.4 1.1 21.4 7.0 6.8 4.9 11.8 11.2 12.0

Sixth grade

Mean 14.0 9.8 29.4 16.3 20.0 26.3 37.3 31.8 36.2

Standard deviation 1.2 6.5 26.6 5.7 8.4 19.7 35.3 13.7 11.3




words in 2000 curricula, the cognitive demands of questions in the sixth-grade curricula based on the number of content words processed hasdoubled.

Extraneous Information in Text

It is thought that extraneous information embedded in text that studentsmust process to respond to a comprehension task would put additional cog-nitive demands on them as they attempt to answer the question. Our analy-ses indicated no strong pattern in the amount of extraneous informationinvolved in students’ processing of text to respond to comprehension tasks(see Table 4). The number of extraneous content words averaged from 3 to10 per comprehension question across the years and curricula we surveyed.

Types of Comprehension Questions

Different types of comprehension questions put different cognitivedemands on readers. In Table 5, we present the relative emphasis (measuredby percentage of the total number of questions) to the most frequent types ofcomprehension. In Table 6, we translate that percentage into the averagenumber of questions per story of that type of question.

Explicit Detail Questions

By inspection it is evident that in third grade, explicit detail questions arefrequently asked throughout all the curricula analyzed and across all deca-des. Early curricula, from the 1920s through 1950s, emphasized askingexplicit detail questions, averaging around one-third of all questions acrossthose decades (see Table 5). In the latter half of the century, there is a gradualdecline in the emphasis on explicit detail questions, down to an average ofonly 10% of all questions in the 2000 curricula. In the 1920s, that emphasis

Table 4

Information Needed to Answer a Question and Extraneous

Information in Text: Means, by Decade

1920 1930 1940 1950 1960 1970 1980 1990 2000

Third grade

Information needed 35.6 26.7 24.2 26.1 28.3 33.1 39.2 47.1 49.7

Extraneous information 3.5 9.4 5.4 7.0 8.6 9.3 7.4 10.1 7.8

Sixth grade

Information needed 39.7 32.4 36.4 36.3 46.9 47.5 34.3 64.2 68.9

Extraneous information 8.8 7.0 9.7 6.9 4.3 4.3 7.7 6.1 5.3

Stevens et al.



on explicit details amounted to about 5 questions (of an average 16) perstory. By 2000, while the number of questions per story has increased dra-matically, still about 5 questions per story (out of an average of 48) wereabout explicitly stated details.

The pattern is similar and in some ways more prevalent in sixth-gradecurricula. In the 1920 editions, 48.5% of all questions were explicit details,by far the most prevalent type of comprehension question in curricula.The emphasis diminished in more recent curricula, with 1990 and 2000 edi-tions asking for explicitly stated details in less than 10% of all comprehensionquestions. In 1920, that emphasis on explicit details meant that an average of6 out of 14 questions about the story were about explicitly stated details. Bythe 2000 curricula, sixth-grade curricula asked an average of only 2.9 ques-tions about explicit details.

Other Types of Comprehension Questions

Reading curricula ask a variety of kinds of comprehension questionsother than explicit details (as noted in Tables 5 and 6). However, there areno readily apparent trends in the relative emphasis on them over time.Inferential questions do seem to be more frequently used in sixth-gradereading curricula since about the 1960s. Prior to the 1960s, reading curriculaaveraged one inferential question per story (around 5% of all questions).

Table 5

Relative Emphasis on Types of Comprehension Questions (Averaging 5% of

Total or More): Mean Percentage of All Questions Asked, by Decade

1920 1930 1940 1950 1960 1970 1980 1990 2000

Third grade

Explicit detail 33.5 31.8 37.8 36.1 23.4 25.9 19.3 13.0 10.3

Paraphrased detail 8.8 14.1 26.3 8.1 11.4 11.5 16.8 13.1 13.7

Inferential 10.1 14.7 4.2 7.9 11.9 6.9 13.0 8.4 11.5

Assess prior knowledge 3.6 12.9 4.9 7.8 7.1 6.1 2.4 4.6 5.4

Character trait 1.5 2.9 1.8 8.8 5.9 5.5 6.1 10.2 6.2

Opinion 14.2 3.5 1.8 9.9 9.5 11.3 6.9 10.7 12.1

Sixth grade

Explicit detail 45.8 39.6 24.1 30.3 14.8 22.1 30.1 7.8 8.0


Inferential 5.6 5.5 5.9 5.5 13.1 9.7 10.8 10.1 9.2


Character trait 1.9 5.4 2.4 4.9 4.5 5.9 5.8 6.5 5.3

Opinion 7.9 14.5 6.2 15.9 10.9 12.8 7.8 13.5 6.3




Since 1960, inferential questions have been asked about 10% of the time insixth grade, ranging from an average of 2.6 to 4 questions per story.

Percentage of Questions About Higher-Level Ideas in Text

In both third- and sixth-grade curricula, there is an increase in the per-centage of questions about higher-level (more thematic) ideas in text (seeTable 7). In third grade, one-third or fewer of the comprehension questionsasked students about higher-level ideas. In more recent curricula, 1990 and2000 editions, nearly 50% of the questions asked were about higher-levelideas in the text.

Sixth-grade curricula provided evidence of an even stronger pattern ofincreased emphasis on questions about higher-level ideas in text. Froma low of only 11.1% of questions in the 1930s, questions about higher-levelideas increased to 49% in each of the 1990s and 2000s curricula.

Conclusions

In looking for patterns in the data it seems most appropriate to considerthe data by grade as there are different and clearer patterns in the third-gradecurricula than for the sixth-grade curricula.

Table 6

Types of Comprehension Questions: Average Number per Story, by Decade

1920 1930 1940 1950 1960 1970 1980 1990 2000

Third grade

Question per story 16.2 7.1 23.0 27.8 31.0 31.3 27.8 31.8 48.5

Explicit detail 5.4 2.3 8.7 10.0 7.3 8.1 9.4 4.1 5.0


Inferential 0.6 1.0 1.0 2.2 3.7 2.2 3.6 2.7 5.6


Character trait 1.5 0.9 0.4 2.4 1.8 1.7 1.7 3.2 3.0

Opinion 2.3 0.2 0.4 2.8 2.9 3.5 1.9 3.4 5.8

Sixth grade

Questions per story 14.0 9.8 29.4 16.3 20.0 26.3 37.3 31.8 36.2

Explicit detail 6.4 3.9 7.1 4.9 3.0 5.8 11.2 2.5 2.9


Inferential 0.7 0.5 1.7 0.9 2.6 2.6 4.0 3.2 3.3


Character trait 0.3 0.5 0.6 0.8 0.9 1.6 2.2 2.1 1.9

Opinion 1.1 1.4 1.8 2.6 2.2 3.4 2.9 4.3 1.1

Stevens et al.



Cognitive Demands of Third-Grade Curricula

Linguistic Analyses of the Text

The following patterns of change were observed for the three dimen-sions of text complexity, as summarized in Table 8. Lexical sophisticationand diversity both exhibited a different pattern of change than that of syntac-tic complexity. For lexical sophistication, while a falling trend was observedbetween the 1910s and the 1940s, a rising trend surfaced in the 1950s andcontinued through the 2000s, with the 2000s reaching a significantly higherlevel than all other decades. Lexical diversity as measured using RTTR (amore reliable measure than TTR) remained fairly stable from the 1910sthrough the 1960s and then increased from the 1970s onwards, with the1990s and 2000s reaching markedly higher levels than all earlier decades.Syntactic complexity was higher in the 1910s–1940s than the 1950s–2000s;there was a falling trend from the 1910s through the 1940s, but no significantchanges occurred in the 1950s–2000s period.

Analyses of Comprehension Tasks

The first interesting finding about reading comprehension tasks asked ofthird-grade students is the consistent increase in the quantity of questionsasked per story across the century (Table 3). The mean number of compre-hension questions increased from a low of 7 per story in 1930 to a high of48.5 questions per story in the 2000 editions. While this suggests anincreased emphasis on developing students’ comprehension of the story,one could question the upper limit of the efficacy of asking comprehensionin terms of facilitating versus interfering with students’ construction of mean-ing while reading. In the more recent editions the teacher’s manual has mul-tiple questions per page embedded in the story. While asking some ques-tions during reading may facilitate students’ understanding of what is read,it is unknown whether frequent questions during reading may interferewith the cohesion of the ideas in the story and hence interfere with the read-er’s construction of meaning.

Table 7

Percentage of High-Level Questions, by Decade

1920 1930 1940 1950 1960 1970 1980 1990 2000

Third grade 34.3 35.4 29.2 27.9 41.1 28.6 46.5 52.3 47.3

Sixth grade 14.6 11.1 26.2 18.5 20.6 32.5 34.4 49.7 49.1




In terms of the types of comprehension tasks asked of readers, thereseems to be a fairly consistent pattern of increasing cognitive demands inthe comprehension tasks during the second half of the century, as summa-rized in Table 8. After a slight drop from 1920 to 1930, the cognitive demandof questions, as measured by the mean number of content words needed toanswer each question (Table 4), stayed fairly stable though 1960, followedby gradual increases through 2000 to a mean of 49 that need to be processedto answer questions. This suggests that the comprehension questions askedof students in recent editions of reading curricula were more demandingcognitively, requiring students to process more textual information to con-struct an answer to the question.

Similar increased complexity can be seen in the types of questions askedof students, most notable in the marked decline in explicit detail questionsasked in curricula after 1950 (Tables 5 and 6). During the first half of the cen-tury, nearly one-third of comprehension questions in reading curricula wereexplicit details that required surface-level processing of the text. The empha-sis on literal comprehension questions declined to about one-quarter of allquestions in the 1960s and 1970s and continued to decline to the pointwhere curricula from 2000 asked explicit detail questions at a rate of around10% of all questions. This pattern of increasing complexity of questions issupported by the increasing percentage of higher-level questions (Table 7)in the more recent curricula. In the 1920s and 1930s, approximately one-

Table 8

Summary of Finding: Cognitive Demands of

Third-Grade Reading Texts and Tasks

Measure of Complexity Early 20th Century Mid 20th Century Late 20th Century

Text:

Lexical sophistication Decrease

1910–1940

Some increase

1950–1980

Marked increase

1990–2000

Lexical diversity Stable Stable

1910–1960

Increasing

1970–2000

Syntactic complexity Decease

1910–1940

Stable Stable

1950–2000

Tasks:

Information needed Decrease

1920–1940

Stable Increasing

1970–2000

Percentage explicit

detail questions

Stable Stable 1920–1950 Decreasing

1970–2000

Percentage questions of

high-level ideas

Stable Fairly stable Increasing

1980–1990

Stevens et al.



third of all questions were about higher-level ideas; this then dips to approx-imately one-quarter through the 1950s, indicating an emphasis on fairly lit-eral, detail-oriented comprehension. However, particularly in the past threedecades, comprehension questions in reading curricula have been fairly bal-anced in asking questions about higher-level ideas and lower-level ideas.This suggests that there is an important decline in conceptualizing compre-hension primarily as extracting ideas from text and an increase in valuingreaders’ interpretation and integration of text.

In general, these data on the nature of the text and tasks in third-gradereading curricula suggest a pattern of marked decrease in cognitive demandsearly in the century (1920–1930) and fairly stable cognitive demand in mid-century (1930–1950 or 1960). Starting in about 1970, the third-grade curriculaseem to gradually increase in both the complexity of the text and the cogni-tive demands of the comprehension tasks.

Cognitive Demands of Sixth-Grade Curricula

Linguistic Analyses of the Text

The following patterns of change were observed for the three dimen-sions of text complexity, as summarized in Table 9. For sixth grade, lexicalsophistication was significantly higher in the 1910s–1920s but remainedlargely stable in the 1930s–2000s (with a low point in the 1980s). Lexicaldiversity as measured using RTTR was generally higher in the 1950s–2000sthan in the 1910s–1940s, with the highest point in the 1990s and the lowestpoint in the 1930s. This suggests that the cognitive demands of sixth-gradereading texts have been fairly stable since about 1950. Syntactic complexitywas significantly higher in the 1910s–1940s than the 1950s–2000s; there wasa falling trend from the 1910s through the 1940s, but it remained largely sta-ble within the 1950s–2000s period.

Analyses of Comprehension Tasks

The analysis of comprehension tasks shows some similarities betweensixth-grade curricula and the trends previously reported for third-grade cur-ricula. First, the quantity of comprehension questions per story followed thesame gradual increase across the decades as found in the third-grade mate-rials. By 2000, reading curricula asked over 36 questions per story on aver-age, an increase 2.5 times the number of questions asked in the 1920s.

The cognitive demand of questions as measured by the amount of infor-mation students need to process to answer a question was fairly consistentfrom 1920 to 1950 (about 35 content words) but then increased in the1960s and 1970s and again in the 1990s and 2000s. By the latter part ofthe century, comprehension questions required students to process around




65 content words of information on average, nearly double the cognitivedemand of curricula tasks in the early half of the century.

The declining emphasis on explicit detail questions observed in thethird-grade curricula, described previously, is also evident in the analysesof comprehension questions in the sixth-grade curricula. In the 1920s, 45%of all questions asked were explicit detail questions, whereas by 1990 and2000 curricula, that had diminished to only 8% of the questions asked.The move away from more literal and lower-level processing of reading toprocessing more higher-level ideas is also seen in the increasing percentageof questions asked about higher-level ideas in stories. In the earlier half ofthe century, only 25% of the questions were asked about higher-level ideasin the stories, whereas by 2000 curricula, nearly half of all questions wereabout higher-level ideas. This trend away from literal comprehension ques-tions provides evidence that the tasks students are asked to complete havebecome more cognitively demanding by requiring the students to processmore information from the text to answer the questions or complete thetasks asked of them. Questions that ask students to interpret informationmay also increase the cognitive demand for readers to use their prior knowl-edge and integrate it with text information. Again, it also gives evidence toa change from conceptualizing comprehension as extracting informationfrom text to more cognitively demanding acts of interpreting and summariz-ing textual information.

Table 9

Summary of Findings: Cognitive Demands of

Sixth-Grade Reading Texts and Tasks

Measure of Complexity Early 20th Century Mid 20th Century Late 20th Century

Text:

Lexical sophistication Decrease

1910–1930

Fairly stable Fairly stable

1930–2000

Lexical diversity Decrease

1910–1930

Increase

1930–1970

Fairly stable

1990–2000

Syntactic complexity Decrease

1910–1940

Stable Stable

1950–2000

Tasks:

Information needed Decrease

1920–1930

Fairly stable

1940–1960

Increase

1970–2000

Percentage explicit

detail questions

Decrease

1920–1940

Varies

1950–1980

Marked decrease

1980–2000

Percentage questions of

high-level ideas

Stable

1920–1930

Fairly stable

1940–1960

Increase

1970–2000

Stevens et al.



Similar to the pattern found for third-grade curricula, the comprehensiontasks in sixth-grade curricula for the latter half of the century show a gradualincrease in the cognitive demands put on students. In general, there isa decline in the comprehension tasks asked about surface-level (explicitdetails) information in the text and more emphasis on asking questionsabout more thematic ideas (higher-level ideas) in the text, again requiringstudents to process more information from the text and increasing the cog-nitive demands of the comprehension tasks.

General Conclusions

These data offer a detailed look at the changes in cognitive demands ofcommonly used reading curricula. In general, there seems to be an initialand fairly dramatic decline in the cognitive demands and difficulty of curric-ula from 1920 to 1930. This may be due to the change in emphasis fromreading aloud and elocution to an emphasis on reading and comprehendingstories, as documented by N. B. Smith (1965). In part, this change seemsrelated to trends in reading research with William Gray (1924), E. B. Huey(1908), and Edward Thorndike (1917) advocating for more instructionalemphasis on developing students’ comprehension and using silent reading,rather than an emphasis on oral reading and elocution. The drop in the read-ability of text also may have been influenced by the use of then newly devel-oped readability measures (Lively & Pressey, 1923; Thorndike, 1921) thatmay have been used to ‘‘standardize’’ the complexity of the stories used inreading curricula.

From about 1930 through 1960 or 1970, the cognitive demands of read-ing curricula changed little. It is difficult to explain a lack of change.

In the period of 1970 through 2000 we observed a fairly consistentincrease in the difficulty of reading text and comprehension tasks, particu-larly at third grade. This may have been the result of criticism of schoolsand classroom instruction beginning in the 1950s. For example, WhyJohnny Can’t Read (Flesch, 1955) criticized typical reading instruction andthe growth of remedial instruction in American schools. The focus on read-ing instruction was heightened by the publication of A Nation at Risk(National Commission on Excellence in Education, 1983) and Becominga Nation of Readers (Anderson, Hiebert, Scott, & Wilkinson, 1985). Thesereports recommended that students be challenged with more rigorous andmeaningful curricula. The criticism may have led to increases in the demandsin reading curricula we observe beginning in 1970.

Similarly, there was a marked increase in reading comprehensionresearch from the 1970s through 2000s (Pearson & Fielding, 1991; Spiro,1980). The research investigated how students construct meaning whilereading and the role instruction plays in developing comprehension strate-gies. This research may have influenced the shift from explicit, literal




comprehension to more inferential and integrative comprehension tasks thatmake greater cognitive demands on readers. It is also likely that the researchwas influential in the move to more authentic text that was of greater rele-vance to the readers.

The texts for sixth-grade reading curricula, however, have been fairlystable over the past half of the twentieth century, at a relatively simpler levelthan what was observed during the 1910–1930 period. Comprehension taskshave to some degree become more cognitively demanding in recent decadesalong the lines of what is observed in the third-grade curricula. There hasbeen a move away from questions about explicitly stated ideas to more inter-pretation and integration of ideas. However, the change is not as profoundas that observed in the third-grade curricula.

Another difference over time that is notable is the types of reading mate-rial used in curricula. While the reading texts may not be as demanding asthose in 1910 or 1920, as evidenced by syntactic complexity, the text maybe more sensible and applicable to readers. Current reading texts use child-ren’s literature from authors like Beverly Cleary in the Scott Foresman curric-ula (Allington, Cortez, Cunningham, Sebesta, & Tierney, 1989) or PatriciaCompton in the Macmillan McGraw Hill curricula (Flood et al., 2003). Theaddition of children’s literature reflects the move toward using more authen-tic text begun in the 1980s (Chall & Squires, 1991). Children’s literature islikely more relevant and comprehensible for young readers rather than read-ing texts like Norse mythology selections found in the 1917 third-grade Ginnreader (Young & Field, 1917).

These findings also offer a more precise look at reading text and com-prehension task complexity that may inform the future work of curriculumdevelopers. The text analyses offer a method for analyzing text that ismore sophisticated than simply looking at vocabulary complexity or read-ability measures. Further, this study provides information about how ques-tions about the text can increase or decrease the overall difficulty of the read-ing curricula. Finally, these results raise an issue about how many questionsare too many. Inserted questions are an attempt to increase readers’ compre-hension; however, there may be an upper limit beyond which questionsbegin to detract from the cohesion of readers’ comprehension of whatthey’ve read.

These results may offer useful ways for teachers and curriculum adop-tion committees to think about reading curricula. Both the text and the typesof questions or tasks asked of students have an impact on the cognitivedemands of a curriculum. Reading curricula from different publishers varyin these areas. Making comparisons of the reading material and comprehen-sion tasks across publishers may help in assessing the cognitive demandswhen selecting curricula.

Stevens et al.



Limitations

This study is limited in that it investigated the cognitive demands ofreading curricula at only the third- and sixth-grade levels. The decisionwas made to look more deeply at a narrower range of curricula ratherthan less deeply and more broadly at reading curricula across multiplegrades. Our intent was to focus on third grade because more intense andfocused reading comprehension instruction tends to begin in third grade,whereas earlier instruction focuses more on decoding instruction. We alsoselected sixth grade because of the transition to more focus on reading tolearn in content areas found in early middle school. While this limits the con-clusions to some degree, we believe it allowed us to analyze a more variedsample of curricula at each grade level and to do finer-level analyses of thecognitive demands of the comprehension tasks asked of students.

Another limitation on the analyses of comprehension tasks is that wesampled only a subset of stories from a subset of the reading curricula.We attempted to sample reading curricula that were more commonly usedin given decades in an attempt to look at the potentially most influential cur-ricula. We also selected some curricula because various editions were usedacross all the decades of the study. However, as in any sampling method,there is the potential to miss data in the process.

This study is also limited in that it cannot analyze all the reading materi-als used in classrooms as there is great variability nationwide in (a) whetherteachers add children’s literature to their reading instruction and (b) whatthose literature selections are. While the majority of teachers use basal seriesin reading instruction (Wade & Moje, 2000), they are not the only source oftext used in reading instruction.

Future Research

Future research in the changes in cognitive demands of school curriculais needed to build on this study and the one conducted by Baker et al. (2010)on mathematics curricula. While these studies provide in-depth analyses ofelementary curricula, there is a similar need to investigate the cognitivedemands of secondary curricula. Our study suggests that the relativeincrease in cognitive demands in reading is greater in third grade than insixth grade. If there are declining cognitive demands in school curricula,as some authors’ data have suggested (e.g., Hayes et al., 1996; Williamson,2006), then these results suggest that more in-depth investigations of second-ary curricula are warranted.

Research should also consider the degree to which inserted questionsaid or hinder readers’ comprehension. This research found a steady increasein the number of questions asked per story, culminating in recent basal read-ers averaging 48 questions per story for third-grade editions. While teachersmay not use all the questions provided, research is needed to determine how




many questions are helpful in increasing comprehension and when ques-tions hinder or distract a reader’s comprehension.

Notes

This project was funded in part by a grant from the Spencer Foundation, Chicago, IL.The authors would like to thank the reviewers of the manuscript for their insightful

comments. The first author would also like to thank Charles Perfetti for his assistance dur-ing the planning phase of the project.

References

Adams, M. J. (2010–2011). Advancing our students’ language and literacy: The chal-lenge of complex texts. American Educator, 34, 3–11, 53.

Allington, R., Cortez, J., Cunningham, P., Sebesta, S., & Tierney, R. (1989). ScottForesman collections reading program. Glenview, IL: Scott, Foresman, andCompany.

Anderson, R. C., Hiebert, E. H., Scott, J. A., & Wilkinson, I. (1985). Becoming a nationof readers: The report of the Commission on Reading. Washington, DC: NationalInstitute of Education, U.S. Department of Education.

Armbruster, B. B., Stevens, R. J., & Rosenshine, B. V. (1977). Analyzing content cov-erage and emphasis: A study of three curricula and two tests. Paper presented atthe annual meeting of the American Educational Research Association, NewYork.

Baker, D., Knipe, H., Collins, J., Leon, J., Cummings, E., Blair, C., & Gamson, D.(2010). One hundred years of elementary school mathematics in the UnitedStates: A content analysis and cognitive assessment of textbooks from 1000 to2000. Journal for Research in Mathematics Education, 41, 383–424.

Bartlett, F. C. (1932). Remembering: A study in experimental and social psychology.London: Cambridge University Press.

Beck, I., McKeown, M., McCaslin, E., & Burkes, A. (1979). Instructional dimensionsthat may affect reading comprehension: Examples from two commercial readingprograms (Technical report). Pittsburgh, PA: Learning Research andDevelopment Center, University of Pittsburgh.

Bowker, R. R. (1948–2010). Books in print. New York, NY: R. R. Bowker Company.Brenner, D., & Hiebert, E. H. (2010). If I follow the teachers’ editions, isn’t that

enough? Analyzing reading volume in six core reading programs. ElementarySchool Journal, 110, 347–363.

Carlson, R., Chandler, P., & Sweller, J. (2003). Learning and understanding scienceinstructional materials. Journal of Educational Psychology, 95, 629–640.

Chall, J. S. (1967). Learning to read: The great debate. New York, NY: McGraw-Hill.Chall, J. S., Conrad, S., & Harris, S. (1977). An analysis of textbooks in relation to

declining SAT scores. Princeton, NJ: College Entrance Examination Board.Chall, J. S., Jacobs, V. A., & Baldwin, L. E. (1990). The reading crisis: Why poor chil-

dren fall behind. Cambridge, MA: Harvard University Press.Chall, J. S., & Squires, J. (1991). The publishing industry and textbooks. In R. Barr,

M. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook or reading research(Vol. 2, pp. 120–146). New York, NY: Macmillan.

Cheung, H., & Kemper, S. (1992). Competing complexity metrics and adults’ produc-tion of complex sentences. Applied Psycholinguistics, 13, 53–76.

Stevens et al.



Covington, M. A., He, C., Brown, C., Nacxi, L., & Brown, J. (2006). How complex is thatsentence? A proposed revision of the Rosenberg and Abbeduto D-Level scale(CASPR Research Report 2006-01). Athens, GA: Artificial Intelligence Center,University of Georgia.

Durkin, D. (1981). Reading comprehension instruction in five basal reader series.Reading Research Quarterly, 16, 515–544.

Flesch, R. (1955). Why Johnny can’t read. New York, NY: Harper.Flood, J., Hasbrouck, J, Hoffman, J., Lapp, D., Lubcker, D., Medearis, A., . . . Wood,

K. (2003). Macmillan McGraw Hill reading. New York, NY: Macmillan McGrawHill.

Gamson, D., Lu, X., & Eckert, S. (2013). Challenging the research base for theCommon Core State Standards: A historical reanalysis of text complexity.Educational Researcher, 42, 381–391.

Goldman, S., & Rakestraw, J. (2000). Structural aspects of constructing meaning fromtext. In M. Kamil, P. Mosenthal, P. D. Pearson, & R. Barr (Eds.) Handbook of read-ing research (Vol. 3, pp. 311–335). Mahwah, NJ: Lawrence Erlbaum Associates.

Gray, W. S. (1924). The importance of intelligent silent reading. Elementary SchoolJournal, 24, 348–356.

Guiraud, P. (1960). Problemes et methodes de la statistique linguistique [Problems andmethods of statistical linguistics]. Dordrecht, The Netherlands: D. Reidel.

Haenggi, D., & Perfetti, C. A. (1994). Processing components of college-level readingcomprehension. Discourse Processes, 17, 83–104.

Hayes, D. P., Wolfer, L. T., & Wolfe, M. F. (1996). Schoolbook simplification and itsrelation to the decline in SAT-verbal scores. American Educational ResearchJournal, 33, 489–508.

Huey, E. B. (1908). The psychology and pedagogy of reading. Cambridge, MA: MITPress.

Johnson, R. E. (1970). Recall of prose as a function of the structural importance of thelinguistic units. Journal of Verbal Learning and Verbal Behaviour, 9, 12–20.

Kintsch, W., & Kintsch, E. (2005). Comprehension. In S. G. Paris & S. A. Stahl (Eds.),Children’s reading comprehension and assessment (pp. 71–92). Mahwah, NJ:Lawrence Erlbaum Associates.

Kintsch, W., & van Dijk, T. (1978). Toward a model of text comprehension and pro-duction. Psychological Review, 85, 363–394.

Lee, L. (1974). Developmental sentence analysis. Evanston, IL: NorthwesternUniversity Press.

Lively, B. A., & Pressey, S. L. (1923). A method for measuring the vocabulary burdenof text books. Educational Administration and Supervision, 9, 389–398.

Lu, X. (2009). Automatic measurement of syntactic complexity in child languageacquisition. International Journal of Corpus Linguistics, 14, 3–28.

Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oralnarratives. The Modern Language Journal, 96, 190–208.

Magliano, J. P., & Mills, K. K. (2003). Assessing reading skill with a think-aloud pro-cedure and latent semantic analysis. Cognition and Instruction, 21, 251–283.

Malvern, D., Richards, B., Chipere, N., & Duran, P. (2004). Lexical diversity and lan-guage development: Quantification and assessment. Houndmills, UK: PalgraveMacmillan.

Marks, C. B., Doctorow, M. J., & Wittrock, M. C. (1974). Word frequency and readingcomprehension. Journal of Educational Research, 67, 259–262.

Martinez, M. G., & McGee, L. M. (2000). Children’s literature and reading instruction:Past, present, and future. Reading Research Quarterly, 35, 154–169.




Mayer, R. (2002). Cognitive theory and the design of multimedia instruction: Anexample of the two-way street between cognition and instruction. NewDirections in Teaching and Learning, 89, 55–71.

McGregor, A. K. (1989). The effect of word frequency and social class on children’scomprehension. Reading, 23, 105–115.

Meyer, B. J. F. (1975). The organization of prose and its effect on memory.Amsterdam: North Holland Publishing Company.

National Commission on Excellence in Education. (1983). A nation at risk: Theimperative for educational reform. Washington, DC: U. S. Department ofEducation.

Pearson, P. D., & Fielding, L (1991). Comprehension instruction. In R. Barr, M. Kamil,P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 3, pp.815–860). Mahwah, NJ: Lawrence Erlbaum Associates.

Perfetti, C. A. (1985). Reading ability. New York, NY: Oxford University Press.Pressley, M. (2000). What should comprehension instruction be the instruction of? In

R. Barr, M. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of readingresearch (Vol. 3, pp. 545–561). Mahwah, NJ: Lawrence Erlbaum Associates.

RAND Reading Study Group. (2002). Reading for understanding: Toward a researchand development program in reading comprehension. Santa Monica, CA: RAND.

Rapp, D. N., van den Broek, P., McMaster, K. L., Kendeou, P., & Espin, C. A. (2007).Higher-order comprehension processes in struggling readers: A perspective forresearch and intervention. Scientific Studies of Reading, 11, 289–312.

Reppen, R., & Ide, N. (2004). The American National Corpus: Overall goals and thefirst release. Journal of English Linguistics, 32, 105–113.

Richards, B. (1987). Type/token ratios: What do they really tell us? Journal of ChildLanguage, 14, 201–209.

Rosenberg, S., & Abbeduto, L. (1987). Indicators of linguistic competence in the peergroup conversational behavior of mildly retarded adults. Applied Psycholinguistics,8, 19–32.

Scarborough, H. S. (1990). Index of productive syntax. Applied Psycholinguistics, 11,1–22.

Smith, N. B. (1965). American reading instruction. Newark, DE: InternationalReading Association.

Spiro, R. J. (1980). Constructive processes in prose comprehension and recall. InR. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in reading com-prehension (pp. 245–278). Hillsdale, NJ: Lawrence Erlbaum Associates.

Taylor, B. M. (1980). Children’s memory for expository text after reading. ReadingResearch Quarterly, 15(3), 399–411.

Templin, M. (1957). Certain language skills in children: Their development and inter-relationships. Minneapolis, MN: The University of Minnesota Press.

Thorndike, E. L. (1917). Reading as reasoning: A study of mistakes in paragraph read-ing. Journal of Educational Psychology, 8, 323–332.

Thorndike, E. L. (1921). Teacher’s word book. New York, NY: Teachers College,Columbia University.

van den Broek, P., McMaster, K. L., Kendeou, P., & Espin, C. A. (2007). Higher ordercomprehension processes in struggling readers: A perspective for research andintervention. Scientific Studies of Reading, 11, 289–312.

Venezky, R. L. (1987). A history of the American reading textbook. The ElementarySchool Journal, 87, 246–265.

Wade, S. E., & Moje, E. B. (2000). The role of text in classroom learning. In M.L Kamil, P. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of readingresearch (Vol. 3, pp. 609–628). Mahwah, NJ: Lawrence Erlbaum Associates.

Stevens et al.



Williamson, G. L. (2006). Student readiness for postsecondary endeavors. Paper pre-sented at the annual meeting of the American Educational Research Association,San Francisco, CA.

Yngve, V. (1960). A model and a hypothesis for language structure. Proceedings ofthe American Philosophical Society, 104, 444–466.

Young, E. F., & Field, A. T. (1917). Young and Field literary readers. Boston, MA:Ginn and Company.

Manuscript received January 30, 2014Final revision received September 12, 2014

Accepted December 7, 2014




Date post:	23-Jul-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Assessing the Cognitive Demands of a Century of Reading ... · from text and integrating it with...

Documents