Common Core Standards, Content

Complexity, andDepth of Knowledge

(DOK) October 16, 2015Third Annual MENA ConferenceAmerican UniversityDubai, United Arab Emirates

Norman L. Webb

Wisconsin Center for Education Products and Services (WCEPS)

Common Core Standards, Content Complexity, and Depth of Knowledge (DOK)

Norman L. WebbSenior Research Scientist Emeritus

Wisconsin Center for Education ResearchUniversity of Wisconsin-Madison

andWisconsin Center for Education Products and

Services

Attainment To

Readiness

The Common Core Standards (CCS) add focus and coherence to the K-12 curriculum to help ensure students are college and career ready by the end of high school.

Outline

Part 1: History of Content Complexity

Part 2: Depth of Knowledge

Part 3: Analysis and Implications

Complexity of the Common Core

Standards

PART 1: History of Content Complexity

Higher performing countries on TIMSS

more effectively used cognitively

demanding tasks and had students engage in critical thinking

and reasoning.(Hiebert and Stigler, 2004)

What does“content complexity”

mean?

What factors make educational material simple vs. complex?

Content Complexit

y

Processing of

concepts and skills

Prior knowledge

and experience

Sophisticationof material

Number of parts

Application of

content structure

Content complexity is important because…

• Alignment

• Validity of assessments

• Clarity of communication

• Truth in advertising

Ralph Tyler’s Behavioral Aspect of the Objectives (course dependent)

1. Understanding of important facts and principles

2. Familiarity with dependable sources of information

3. Ability to interpret data

4. Ability to apply principles

5. Ability to study and report results of study

6. Broad and mature interests

7. Social attitudes

Bloom’s Taxonomy (1956)

Knowledge Recall of specifics and generalizations; of methods and processes; and of pattern, structure, or setting.

Comprehension

Knows what is being communicated and can use the material or idea without necessarily relating it.

Application Use of abstractions in particular and concrete situations.

Analysis Make clear the relative hierarchy of ideas in a body of material or to make explicit the relations among the ideas or both.

Synthesis Assemble parts into a whole.

Evaluation Judgments about the value of material and methods used for particular purposes.

Bloom’s Revised Taxonomy (2001)

OLD NEWKnowledge Remember

Comprehension Understand

Application Apply

Analysis Analyze

Synthesis Evaluate

Evaluation Create

Krathwol and Anderson, 2001

Knowledge Dimension of Revised Bloom’s Taxonomy

• Factual knowledge

• Conceptual knowledge

• Procedural knowledge

• Metacognitive knowledge

(Krathwohl, 2002)

Survey of Enacted Curriculum, ELA Cognitive

LevelsRecall Provide facts, terms, definitions,

conventions; describe, etc.

Demonstrate/Explain

Follow instructions; give examples; etc.

Analyze/Investigate

Categorize, schematize; distinguish fact from opinion; make inferences, draw conclusions; etc.

Evaluate Determine relevance, coherence, logical, internal consistency; test conclusions; etc.

Generate/Create Integrate, dramatize; predict probable consequences; etc.

PISA: Three Clusters Representing Understanding

of Mathematics

Reproduction cluster

Knowledge of facts and common problem representation

Connection cluster

Applying problem solving to non-routine situations in familiar contexts

Reflection cluster

Planning solution strategies and implementing them in problem settings with more elements and more unfamiliar contexts.

PISA: Three Science Competencies

• Identify scientific issues

• Explain phenomena scientifically

• Use scientific evidence

Why another system??

Standards-based educatio

n

Need for alignment

Need for appropriate system to describe content

complexity

Part 1: Summary

• Content complexity is worth considering

• Many different language systems and frameworks have been developed to describe content complexity

• DOK was developed to best fit the needs of describing content complexity as expressed in learning expectations (standards)

• DOK was developed to inform alignment studies (of standards and assessments)

Complexity of Common Core Standards

Part 2: Depth-of-Knowledge System

Depth of Knowledge (Webb, 1997)

Level 1:

Recallrecall of a fact, information, or procedure, etc.

Level 2:

Skills/Conceptsuse information or conceptual knowledge, connecting ideas, etc.

Level 3:

Strategic thinkingrequires reasoning, developing a plan; may not have a single correct answer or approach, etc.

Level 4:

Extended thinkingoften an investigation or project; involves extended time spent on complex problems, etc.

Depth of Knowledge (Webb, 1997)

ReadingLevel 1:

Recall of a fact, recognize text feature, verbatim recall, etc.

Level 2:

Draw meaning from text using structure and features, comprehend, connect ideas, etc.

Level 3:

Reason, conduct analysis to make inferences, read critically to attest to internal logic, implied values, identify abstract connections between texts, etc.

Level 4:

Often an investigation or project; involves extended time spent on complex problems, etc.

Why DOK?

Rationale for 4 levels

Based on contemporary learning models

Takes into account multiple factors that contribute to content complexity

Recognizes inherent differences in the structure of different disciplines

Provides a common language that is a good fit for today’s education world

Which of these means about the same as the word gauge?

a.balance

b.measure

c.select

d. warn Grade 5

Grade 8

A triangle has 0 diagonals, a quadrilateral has 2 diagonals, a pentagon has 5 diagonals, and a hexagon has 9 diagonals. If the pattern continues, how many diagonals will a octagon have?

Sides 3 4 5 6

Diagonals

0 2 5 9

A. 11B. 14C. 18D. 20

26

Which of these conclusions is best supported by information from the passage? a. If a candidate meets the personal and educational

qualifications and is in fair physical shape, his or her chances of becoming an agent are very good.

b. Compared with other law enforcement agencies in the country, the F.B.I. has a low success rate for tracking down and apprehending suspected offenders.

c. The job of an agent is not for everyone; it takes someone with special training who is not afraid of danger and doesn’t mind being socially isolated at times.

d. The life of a federal investigator is not as interesting as most people think; agents spend most of their time working at desks.

121 1) 190 13 2) 200 32 3) 290 + 34 4) N

Grade 4

None of the above

a)

b)

c)

d)

It’s still a level 1

(AP Photo/Matt Rourke)

As of 2007, Marc Umile had recited (from memory) more digits of Pi (over 15,000) than any other North American.

    COMPLEXITY

    LOW HIGH

DIFFICULTY

LOW    

HIGH    

Levels of content complexity

NOT Levels of importance

NOT Levels of progression/sequence

Hess’ Cognitive Rigor Matrix

Bloom’s Revised Taxonomy of Cognitive ProcessDimensions

Webb’s Depth-of-Knowledge (DOK) Levels

Level 1Recall &

Reproduction

Level 2Skills &

Concepts

Level 3Strategic Thinking/ Reasoning

Level 4Extended Thinking

Remember

Understand

Apply

Analyze

Evaluate

Create

This isn’t quite how we

see it.

IMPORTANT: It’s more than just

verbs! (We didn’t m

ake this.)

Content complexity is

continuous and generally

decided by group analyses.

Part 2: Summary

• DOK provides a common language to describe content complexity in the context of today’s education world

• Difficulty and complexity are related but different

• There are several common misunderstandings and misrepresentations of DOK

• With practice, there’s an increase in consistency when applying DOK

Complexity of the Common Core Standards

Part 3: Analysis and Implications

Common Core Standards (CCS)

and DOK

What distribution of DOK do we see in the CCS?

Standards for Mathematical Practice

1. Make sense of problems and persevere in solving them.

2. Reason abstractly and quantitatively.3. Construct viable arguments and critique the

reasoning of others.4. Model with mathematics.5. Use appropriate tools strategically.6. Attend to precision.7. Look for and make use of structure.8. Look for and express regularity in repeated

reasoning.

1. Thinking and reasoning

2. Argumentation

3. Communication

4. Modeling

5. Problem posing and solving

6. Representation

7. Using symbolic, formal and technical language and operations

8. Use of aids and tools

Mathematical Competency Used for Pisa

Critical Areas

Domains

Clusters

Standards

Structure of CCS Mathematics

Grade 7 CCS Mathematics StandardsLevel Description DOK

7.RP Ratios and Proportional Relationships7.RP.1 Compute unit rates associated with ratios of fractions, including

ratios of lengths, areas and other quantities measured in like or different units. For example, if a person walks 1/2 mile in each 1/4 hour, compute the unit rate as the complex fraction 1/2/1/4 miles per hour, equivalently 2 miles per hour.

7.RP.2 Recognize and represent proportional relationships between quantities. a. Decide whether two quantities are in a proportional relationship, e.g., by testing for equivalent ratios in a table or graphing on a coordinate plane and observing whether the graph is a straight line through the origin. b. Identify the constant of proportionality (unit rate) in tables, graphs, equations, diagrams, and verbal descriptions of proportional relationships. c. Represent proportional relationships by equations. d. Explain what a point (x, y) on the graph of a proportional relationship means in terms of the situation, with special attention to the points (0, 0) and (1, r) where r is the unit rate.

7.RP.3 Use proportional relationships to solve multistep ratio and percent problems. Examples: simple interest, tax, markups and markdowns, gratuities and commissions, fees, percent increase and decrease, percent error.

41

Mathematics 7.RP.2

Recognize and represent proportional relationships

a. Decide whether two quantities are in a proportional relationship

b. Identify the constant of proportionality (unit rate)

c. Represent proportional relationships by equations.

d. Explain the meaning of a point (x, y) on the graph of a proportional relationship

Depth of Knowledge of All Mathematics Common Core State Standards

Grade Total DOK Level

1 2 3 4 3 25 24 24 3 0 4 28 28 16 5 0 5 26 26 18 3 0 6 29 29 20 2 0 7 24 18 22 8 0 8 28 26 25 9 0

Number and Quantity 27 27 15 0 0 Algebra 27 26 21 7 0 Functions 28 27 24 4 0 Geometry 43 24 36 19 1 Statistics and Probability 31 27 29 7 0 TOTAL 316 282 250 67 1 Percentage of Total Standards at DOK Level (Standards may cover a range of DOK levels)

89% 79% 21% < 1%

Sato, E, Lagunoff, R, and Worth, P. (March 4, 2011) SMARTER Balanced Assessment Consortium Common Core State Standards analysis: Eligible content for the summative assessment. Final report submitted to the SMARTER Balanced Assessment Consortium. San Francisco, CA: West Ed. http://www.k12.wa.us/SMARTER/pubdocs/SBAC_CCSS_Eligible_Content_Final_Report_030411.pdf downloaded October 17, 2011.

43

44

Florida Analysis of DOK Levels of CCSS

Mathematics for High School Total #

Standards

DOK Level

1 2 3 4

Number and Quantity 27 13 14    

Functions 28 2 23 3  Algebra 27 7 14 6  Geometry 43 5 24 14  >1Statistics and Probability

31 3 27 1  

Total 156 30 102 24  Percent   19% 65% 15%  Smarter Balanced %   84% 80% 24% 1%

Reading Standards for Literature K–5

Reading Standards for Informational Text K–5

Reading Standards: Foundational Skills (K–5)

Writing Standards K–5

Speaking and Listening Standards K–5

Language Standards K–5

 

K-5 Language Arts Standards

45

CCS Grade 7 Reading (Inf) & Language Standards

(p. 1)

46

Level Description DOKRI Reading Standards for Informational TextRI.7.KID Key Ideas and DetailsRI.7.KID.1

Cite several pieces of textual evidence to support analysis of what the text says explicitly as well as inferences drawn from the text.

RI.7.KID.2

Determine two or more central ideas in a text and analyze their development over the course of the text; provide an objective summary of the text.

RI.7.KID.3

Analyze the interactions between individuals, events, and ideas in a text (e.g., how ideas influence individuals or events, or how individuals influence ideas or events).

RI.7.KID.1

Cite several pieces of evidence

to support analysis of what the text says explicitly

as well as inferences drawn from the

text.47

Depth of Knowledge of Eligible ELA Common Core State Standards

Grade Total DOK Level

1 2 3 4 3 35 20 27 24 7 4 35 18 26 28 9 5 35 16 30 29 12 6 36 13 25 31 15 7 36 14 25 31 18 8 36 13 25 31 18

9–10 36 13 25 31 18 11–12 36 13 25 31 21

TOTAL 285 120 208 236 118 Percentage of Total Standards at DOK Level (Standards may cover a range of DOK levels) 42% 73% 83% 41%

Sato, E, Lagunoff, R, and Worth, P. (March 4, 2011) SMARTER Balanced Assessment Consortium Common Core State Standards analysis: Eligible content for the summative assessment. Final report submitted to the SMARTER Balanced Assessment Consortium. San Francisco, CA: West Ed. http://www.k12.wa.us/SMARTER/pubdocs/SBAC_CCSS_Eligible_Content_Final_Report_030411.pdf downloaded October 17, 2011.

48

Florida Analysis of CCS DOK LevelsEnglish Language Arts Grades K-2

K-2Areas

Total #Standards

DOK Level

1 2 3 4

Read Lit 27 4 19 4 0

Read Inform

30 3 20 7 0

Read Found

10 7 3 0 0

Writing 21 0 9 9 3

Spk List 18 3 11 4 0

Language 16 6 7 3 0

Total 122 23 69 27 3

Percent   19% 56% 22% 2%49

Florida Analysis of CCS DOK LevelsEnglish Language Arts Grades 3-5

Grades 3-5Areas

Total #Standards

DOK Level

1 2 3 4

Read Lit 27 0 15 12 0Read Inform 30 0 15 14 1

Read Found 6 3 3 0 0Writing 29 0 3 23 3Spk List 18 1 7 10 0Language 18 6 7 5 0Total 128 10 50 64 4Percent   8% 39% 50% 3%

50

Florida Analysis of CCS DOK LevelsEnglish Language Arts Grades 6-8

Grades 6-8Areas

Total #Standards

DOK Level

1 2 3 4

Read Lit 27 0 8 19 0Read Inform 30 0 11 19 0

Writing 30 0 4 20 6Spk List 18 0 5 13 0Language 18 0 14 4 0Total 123 0 42 75 6Percent    0% 34% 61%  5%

51

Florida Analysis of CCS DOK LevelsEnglish Language Arts Grades 9-12

Grades 6-8Areas

Total #Standards

DOK Level

1 2 3 4

Read Lit 18 0 3 15 0Read Inform 20 0 5 15 0Writing 20 0 1 11 8Spk List 12 0 2 10 0Language 12 0 7 5 0Read History 20 0 5 15 0Read Sci Tech

200 8 12 0

Write HST 18 0 1 9 8Total 140 0 32 92 16Percent     23% 66% 11%

52

1. Clarify important learning outcomes and priorities

2. Plan and deliver effective instruction

3. Select and design appropriate assessments

4. Align standards, instruction, and assessments.

Strategies For Applying DOK To Help Improve Learning

53

Deconstruct standards and expectations

Outline learning trajectory

Map to given time

Anticipate learning struggles

Clarify Important Learning Outcomes and Priorities

54

2.OA.3. Determine whether a group of objects (up to 20) has an odd or even number of members, e.g., by pairing objects or counting them by 2s; write an equation to express an even number as a sum of two equal addends. (Grade 2 Operations and Algebraic Thinking)

DOK 1:

Define: odd group

even equation

number equal

addend sum

Count to 100

Count by 2

Deconstruction of a Standard

55

DOK 1 (continued):

Understand the number line

Place numbers on a number line

Use skip counting

Pair objects

  Compare numbers

Identify a number as odd or even

Identify an equation

DOK 2:

Write an equation

Deconstruction (continued)

56

Assessment Items: Factors that Can Influence DOK

• Question• DOK 1 What, where, find, compute• DOK 2 Why or why not, compare, organize• DOK 3 Imply, infer, project, generalize

• Context• DOK 1 None• DOK 2 Familiar and relevant• DOK 3 Unfamiliar, multiple

• Application• DOK 1 Routine or none• DOK 2 Typical, routine, basic• DOK 3-4 Fix, verify, and justify; hypothetical

But REMEMBER: The DOK of an

item depends on the main

challenge for a typical student.

How Does Item Type Influence DOK?

• multiple choice

• constructed response

• cloze

• hot text

• fill-in-the-blank

• sequencing

• drag-and-drop

• multi-part itemBut REMEMBER: The DOK of an

item depends on the main

challenge for a typical student.

Practice in Using DOK

59

Look at the drawing. The numbers alongside each column and row are the total values of the symbols within each column and row. What should replace the question mark?

Grade 8:

A. 23

B. 25

C. 28

D. 30

E. 32 

Is there a match among

1. Complexity of outcome not process

2. Challenge of instructional activity

3. Necessary performance on assessment

Check on Alignment

61

Summary and Takeaway Themes

• DOK is a valuable common language, with ongoing training and discussion

• Complexity and difficulty are related but different

• It’s critical to consider alignment

• Consider central performance in objectives

• Consider item type (MC, CR, cloze, etc.)

• Consider appropriate distribution by DOK level

Website 

webbalign.org

Alignment Tool 

watv2.wceruw.org