Design frameworkItem developmentCognitive complexitySample items

MATHITEM DESIGN

4 BUILDING BLOCKS

Learning Progressio

n

Item Design

Outcomes/Scoring

Assessment Quality

BEAR Assessment System Step 2A match between what is taught and what is

assessedConstructive alignment aligning teaching and

assessment to the learning outcomes/standards (Biggs, 2003)

Proposed items are located along the LP map

ITEM DESIGN FRAMEWORK

Learning Outcomes/Standard

sAssessme

nt Task

Teaching &

Instruction

One Framework(Wilson & Sloane, 2000)

Item design framework used by Smarter-Balanced Assessment under the evidence-centered design approach (Mislevy, Steinberg, & Almond, 2003)

Defined as the degree to which expectations and assessments work together to improve and measure students learning

ALIGNMENT FRAMEWORK

1. Categorical concurrence• Commonality between the content categories of the

standards and those of the assessment items2. Range of knowledge correspondence

• Number of objectives within each standard covered by item(s)

3. Balance of representation• Relative coverage of content categories by items

in a test4. Depth of Knowledge consistency

• Match between the cognitive demand of items and the level of cognitive demand communicated by the wording of the objectives

4 CRITERIA TO DETERMINE THE DEGREE OF ALIGNMENT

Universal designDesign item that accurately assess the targeted competency for all students

Ensure item fairness – make sure that items are equally difficult for groups of equal ability (e.g. males and females; urban and rural)

Vocabulary & languageuse content-specific language appropriate to the assessed grade

For non-content-specific material, use vocabulary/language from previous grade levels

ITEM DEVELOPMENT

Grade appropriateness Design items that assess a primary content

domain/standard of the appropriate grade “For non-reading items, the reading level is

approximately one grade level below the grade level of the test, except for specifically assessed content terms or concepts” (SBAC, 2012)

Using items to link tests For pre-post test designs, include some items that

appeared on previous test(s) to measure student progress If the time between tests is relatively long (i.e. 2-3

months), the same test can be used If the time is short (i.e. 2-3 weeks), pick a few

items to reuse and include new ones

ITEM DEVELOPMENT

Use of Modified Bloom’s TaxonomyDefinition

An example of Cognitive Rigor Matrix (Hess, et al., 2009)

Demonstration on how to align standards and proposed item(s) on the LP map

COGNITIVE COMPLEXITY

Modified by Anderson & Krathwohl (2001)

MODIFIED BLOOM’S TAXONOMY

Old Bloom’s Taxonomy (Bloom, 1956)

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

COGNITIVE RIGOR MATRIX

SAMPLE ITEMS FOCUS ON APPLICATION PROCESS

Process Level 1: Recall & Reproduction

Level 2: Skills & Concepts

Level 3: Strategic Thinking/Reasoning

Level 4: Extended Thinking

Apply:Carry out or use a procedure in a given situations; carry out (apply to a familiar task) or use (apply) to an unfamiliar task.

Follow simple/routine procedures.Apply an algorithm or formula.

Solve routine problems applying multiple concepts or decision points.

Use or apply concepts to solve non-routine problems.

Select or devise an approach among many alternatives to solve a novel problem.

SAMPLE ITEM*

Intended level: Grade 8 Domain: Expressions and EquationsContent standard: Analyze and solve linear equations

and pairs of simultaneous linear equations.CCSS: Analyze and solve pairs of simultaneous linear

equations (8.EE.8). Intended claims: 1, 2, 3 and 4

Max is organizing a trip to the airport for a party of 75 people. He can use two types of taxi. A small taxi costs $40 for the trip and holds up to 4 people. A large taxi costs $63 for the trip and holds up to 7 people.

* Adapted from SBAC (2013, p. 134)

SAMPLE ITEM GRADE 8

LP level Specific Standard Cognitive Rigor

Analyze and solve linear equations and pairs of simultaneous linear equations.

Analyze and solve pairs of simultaneous linear equations (8.EE.8).

Apply, Level 3 (Strategic Thinking/reasoning)

How many taxis of each type should Max order, to keep the total cost as low as possible? Explain.

SAMPLE ITEM GRADE 7

LP level Specific Standard Cognitive Rigor

Solve real-life and mathematical problems using numerical and algebraic expressions and equations.

Use variables to represent quantities in a real-world or mathematical problem, and construct simple equations and inequalities to solve problems by reasoning about the quantities. (7.EE.4).

Apply, Level 2 (Skills & Concepts

Let L be the number of large taxis needed and S be the number of small taxis needed. a. Write an expression to show the number of taxis needed.b. If Max orders 6 large taxis, how many small taxis will he need? b. How much will the cost be?

SAMPLE ITEM GRADE 6

LP level Specific Standard Cognitive Rigor

Apply and extend previous understandings of arithmetic to algebraic expressions.

Write, read, and evaluate expressions in which letters stand for numbers. (6.EE.2).

Apply, Level 1 (Recall and Reproduction)

a. Let x be the number of large taxis needed. If Max wants to order large taxis only, evaluate x. b. If each person must equally share the cost of taking large taxi defined previously, write an expression for the cost of each person. Let y be the amount of money each person must pay. Calculate y.

Bloom, B. S . (1956) . Taxonomy of educat iona l ob ject ives . Handbook I : The Cogni t ive Domain . New York, NY: Dav id McKay Co.

Anderson, L . W. , & Krathwohl , D . (2001) . A Taxonomy for Learn ing, Teaching, and Assess ing: A Rev is ion o f B loom's Taxonomy of Educat iona l Object ives . New York, NY: Longman.

Hess , K . , Car loc , D . , Jones , B . , & Walkup, J . , (2009 ) . What exact ly do “ fewer, c learer, and h igher s tandards” rea l ly look l ike in the c lassroom? Us ing a cogni t ive r igor matr ix to ana lyze curr icu lum, p lan lessons , and implement assessments . Paper presented at CCSSO, Detro i t , Mich igan.

Nitko , A. J . , & Brookhart , S . (2007) . Educat iona l assessment o f s tudents . Upper Saddle R iver , N J : Pearson Educat ion, Inc .

McMi l lan , J . H. (2007) . Classroom assessment . Pr inc ip les and pract ice for eff ect ive s tandard-based inst ruct ion (4th ed. ) . Boston: Pearson - A l lyn & Bacon.

Oregon Department o f Educat ion. (2014, June) . Assessment gu idance . Webb, N . (2007) . Al ign ing assessments and s tandards . Retr ieved f rom

http: / /www.wcer.w isc .edu/news/coverStor ies /a l ign ing_assessments_and_standards .php Wilson, M. (2005) . Construct ing measures : An i tem response model ing approach . New

York, NY: Psycho logy Press , Tay lor & Franc is Group. Wilson, M. , & S loane, K . (2000) . From pr inc ip les to pract ice: An embedded assessment

system. Appl ied Measurement in Educat ion, 13 (2) , pp. 181-208. Smarter Ba lanced Assessment Consor t ium. (2012, Apr i l ) . Genera l i tem spec ifi cat ions . Smarter Ba lanced Assessment Consor t ium. (2013, June) . Content spec ifi cat ions for the

summat ive assessment o f the Common Core State Standards for Mathemat ics . Rev ised draft .

BIBLIOGRAPHY

I tem Design (Math) PPT by the Oregon Department of Educat ion and Berkeley Evaluat ion and Assessment Research Center is l icensed under a CC BY 4.0.

You are free to: Share — copy and red i s t r i bu te the mater ia l i n any med ium o r fo rmat Adapt — remix , t rans fo rm, and bu i l d upon the mater ia l

Under the fo l lowing terms: Attr ibut ion — You mus t g ive approp r ia te c red i t, p rov ide a l i nk to the l i cense , and

ind i ca te i f changes were made. You may do so in any reasonab le manner , bu t no t i n any way tha t sugges ts the l i censo r endo rses you o r you r use .

NonCommerc ia l — You may no t use the mater ia l fo r commerc ia l pu rposes. ShareAl ike — I f you remix , t rans fo rm, o r bu i l d upon the mater ia l , you mus t d i s t r i bu te you r

con t r ibu t i ons under the same l i cense as the o r ig ina l .

Oregon Department of Educat ion welcomes edit ing of these resources and would great ly appreciate being able to learn from the changes made. To share an edited version of this resource, please contact Cr isten McLean, cr isten.mclean@state.or.us.

CREATIVE COMMONS LICENSE