PARCC Bias and Sensitivity Review

AFT & NEA Item Review Boot CampFebruary 9, 2014

Points of Discussion

Purpose of Bias and Sensitivity Reviews

Identifying Bias and Stereotypes

Sensitivity Awareness and Detecting Passage Bias

Review for universal design and text complexity

Guiding Questions and Sample Items

Confidential and Proprietary.

Process

Step 1: Item Writing PARCC item developers were instructed to use the PARCC

Accessibility Guidelines and the Item Development ITN language on writing accessible and universally designed items.

Vendors each trained item writers on the guidelines and additional internal guidelines for content and bias & sensitivity.

Step 2: Item Reviewing Formal process is being conducted for reviewing items at the

Core Leadership, State Educator, and Bias & Sensitivity Committee levels

Purpose is to ensure items have been written for content, accessibility, and bias & sensitivity

3

Purpose of Bias and Sensitivity Review

Review test materials for potential sources of bias and stereotypes.

Apply professional test development standards to ensure materials are fair and not insensitive or offensive.

4Confidential and Proprietary.

Bias and Sensitivity Review Committee Charge

Your role is to provide expert review of passages for potential sources of bias and insensitivity

Use the guidelines and samples provided in training to apply bias and sensitivity criteria to review of passages/texts and ensure that passages are not unfair or insensitive

Remember: Content concerns are addressed by expert content review committee

Important to focus exclusively on PARCC’s bias and sensitivity review criteria during your review

Other concerns identified will be placed in “parking lot” for consideration by PARCC leadership

5

What Is Bias?

Language or content that prevents members of a group from demonstrating they possess the knowledge and skills being measured.

Language or content that advantages members of a group in demonstrating they possess the knowledge and skills being measured.

Confidential and Proprietary.

Common Forms of Bias and Stereotypes

Regional and geographic bias Gender and age stereotypes Ethnic, cultural, and religious stereotypes Socioeconomic and occupational

stereotypes

Confidential and Proprietary.

Guideline #1

Avoid Cognitive Sources of Construct-Irrelevant Variance

8Confidential and Proprietary.

9

Some Background

Construct = knowledge, skill, or other attribute (KSA) you are trying to testConstruct-relevant = related to KSA that you are trying to testConstruct-irrelevant = KSA that is not related to what you are trying to testVariance = differences in test scores

9Confidential and Proprietary.

Construct-irrelevant variance lowers validity.

Construct-irrelevant variance that has different

effects across groups lowers fairness.

Construct-irrelevant differences across groups

decrease validity & fairness.

Construct-relevant differences across groups are

valid and, therefore, fair.

Variance, Validity & Fairness

Confidential and Proprietary.

Avoid Affective Sources of Construct-Irrelevant Variance

Guideline #2

Confidential and Proprietary.

Sensitivity Awareness

Any reference or language in an item or passage that might cause a student to

have an emotional reaction during the test administration can prevent a

student from being able to accurately demonstrate ability.

Note: The emotional factor is not limited to negative emotions, but can also

include a “giggle factor” in items.

12Confidential and Proprietary.

Avoid Physical Sources of Construct-Irrelevant Variance

Guideline #3

Confidential and Proprietary.

Visual stimuli in the middle of paragraphs

Decorative rather than informative illustrations

Fonts that are hard to read

Letters that look alike (e.g., O, Q) used as labels for dif-ferent things in the same item/stimulus

Samples of Barriers

Confidential and Proprietary.

How Might Item Bias be Detected?

Judgmental Procedure• Bias and sensitivity review prior to field-testing

Statistical Procedure• DIF (Differential Item Functioning) analysis following field-

tests and operational administrations

Confidential and Proprietary.

Guiding Questions for Passages

Does the passage disadvantage any population (gender, race, ethnicity, language, religion, socioeconomic status, disability or geographic region) for non-educationally relevant

reasons?

Does the passage contain controversial or emotionally charged subject matter that is not supported by the Common Core State Standards?

Is the passage potentially offensive, demeaning, insensitive, or negative toward any population?

Does the passage depict any population in a stereotypical manner?

16Confidential and Proprietary.

What is Universal Design as Applied to Assessment?

Concept or philosophy that, when applied to assessments, provides all students with equal opportunities to demonstrate what they have learned.

Considers the full range of participating students when developing items, tasks, and prompts that measure a desired construct

Acknowledges differences among individuals and enables flexible adjustments for a broad range of students. Not one-size fits all.

Purpose is to provide access for the greatest number of students during assessment, and minimize the need for individualized design or accommodations.

17

Who is Intended to Benefit from Universally Designed Assessments?

ALL students benefit from assessments that are universally designed, including, but not limited to:

Students who are gifted and talented English learners Students with physical, cognitive, or sensory disabilities Students with emotional or language/learning disabilities Students with more than one of these characteristics Students with unique linguistic needs Underperforming students Students without disabilities

18

Finding Out What Students Know and Can Do

By increasing access at the beginning through item writing and review, the assessments will allow participation from the widest possible range of students

Testing results shouldn’t be affected by disability, gender, race, English language ability, etc.

19

RESULT? Valid inferences about performance for all

students who participate

• Core Leadership Group, Bias & Sensitivity Review Committees, and Accessibility Reviewers will all review for components that make up universally designed assessments.

Universal Design Item Review

1. Are the items and tasks amenable to accommodations?

2. Are the items and tasks designed for maximum readability, comprehensibility, and legibility?

3. Does the item or task material use a clear and accessible text format?

4. When appropriate, does the item/task material use clear and accessible visual elements?

Accessibility Universal Design Review

5. Have all accessibility features been considered that may increase access while preserving the targeted construct?

6. Have multiple means of presentation, expression, and engagement been considered with regard to the item/task?

7. Have changes to the format been considered that do not alter the item/task meaning or difficulty?

Accessibility Universal Design Review

1. Does the item take into consideration the diversity of the assessment population?

2. Are the items sensitive and free of bias?

3. Are instructions and procedures simple, clear, and intuitive?

Questions for Bias & Sensitivity Universal Design Review

1. Items that rely on color

• If you print out the item in black and white….Are you still able to response to the test item?

2. The use of language for specific devices(tablets, PCs, etc.)

• Use the word “Select” versus “Click”

• Use of the word “Enter” versus “Type”

• Use the word “Select or Choose” versus “Highlight”

Lessons Learned from Phase I Item Development for Accessibility

3. Labels on geometric figures

• Avoid using o, i and l

• Avoid using the following letter combinations

• u and v

• p and q

• m and n

Lessons Learned from Phase I Item Development for Accessibility

3. The use of idioms for ELA

– Items should not use idioms or words that obscure what is being asked of students. 

– Items that use idioms or words that obscure what is being asked mail fail to meet UDL guidelines and will need to be rewritten. 

– The only exception to this is for those items that are designed to measure comprehension of idioms or where an item refers to a specific line from a text that might use an idiom.  In these exceptions where the idiom is part of the construct being measured, the idiom must remain in the item

Lessons Learned from Phase I Item Development for Accessibility

Text Complexity Application

• To support assessment item development• To establish expectations for CCSS and NGSS

standards relative to language complexity• To support validation of assessments• To provide information to explore how items function for

students with differing levels of language proficiency• To support adaptive item selection algorithm

Language Complexity Tool

• What to Rate– Prompts, items and directions (MC and CR items)– For CR items the following is scored

o Prompt & directionso * Satisfactory student response (during field testing)

• What to do with the rating?– Record all 4 LC Tool scores as meta-data

© 2012 Board of Regents of the University of Wisconsin System.

© 2012 Board of Regents of the University of Wisconsin System.

Language Complexity Tool

29

Language Complexity Tool Scores

Text Density

Info Density

Length

Form & Structure

Syntactic Complexity

Vocabulary

Lexical Complexity

Record all 4 ratings

30

How To Proceed

Let’s look at some samplepassages and items.

Questions?

Confidential and Proprietary.

How To Proceed

Read through a specific set of test materials, keeping in mind the guiding questions. Do not review these materials from the point of view

of a content expert, instead review them only from the perspective of fairness.

If possible sources of bias, stereotypes, or sensitivity are detected, note them in your comments, and then they will be discussed by the group.

The recorder will take complete and precise notes for the group.

Confidential and Proprietary.