Stephen N. Elliott, PhDDunn Family Professor of AssessmentDepartment of Special EducationPeabody CollegeVanderbilt University

Selecting Accommodations Wisely: Facilitating Test Access &

Enhancing Implementation Integrity

Introduction & Background Information

• Served on NRC Committee that authored Educating One & All,

• Senior author of the Assessment Accommodation Guide,

• Developed (w/ Braden) Assessing One & All online course & book,

• Published 7 experimental studies (w/ a number of colleagues) on the effects of testing accommodations on test scores, and

• Currently serve on Technical Work Groups for NAEP Accommodations & the IES National Alternate Assessment Study.

Presentation Outline

• Words and Concepts Matter – let’s communicate!

• Key Factors that Influence Accommodation

Decision Making

• 5-Step Decision Making Process & Documentation

• Questions & Some Answers

Decisions about accommodations requires IEP teams to know...

The student’s abilities, The instructional accommodations the student

presently receives, The test (content, item type, & testing conditions) and

the skills measured by the test, The state’s Testing Guidelines, What it means to invalidate a test score, and

Previously used testing accommodations.

Michele - Case #1

Michele is an 8th grader who

has had some difficulty learning

and frequently exhibits poor

work habits. She is functioning

below grade level expectations

in almost all subjects, but does

not qualify for special services.

Tia - Case #2Tia is a 4th grader who is classified as learning disabled. Her instructional reading level is 2nd grade, but she receive all her instruction in regular classes with some support from a consulting special education teacher. She has good listening and memory skills, and is a highly motivated student.

Ben - Case #3

Ben is chronologically a 10thgrader who was diagnosed as autistic at age 3. Due to his pervasive communication difficulties, he receives muchof his education in a highly structured special education classroom with 6 other studentswith developmental disorders.

Individual Students Have Individualized Needs

• Most states require a student to have an IEP to be eligible for accommodation considerations.

• Look into the classroom for evidence about accommodations needed to support learning. Many instructional accommodations generalize to the testing event.

IEP Team Members Must Understand an Individual Student’s Instructional Support Needs before Making a Decision about Needed

Testing Accommodations!

Definition & Functionof Testing Accommodations

Definition - changes to the way a test is administered or responded to by a student. Such changes are often categorized as changes to the setting, timing, scheduling, presentation, and method of responding.

Purposes • To facilitate participate from individuals who have not taken

tests in the past.• To offset distortions in test scores caused by a disability

without invalidating the test results (i.e., to increase the validity of the inference made from a test score).

• To comply with IDEA and state regulations.

Testing Alterations:Further Defining an Accommodation

Accommodation +------------------+ Modification

• Unchanged Constructs

• Individual Need

• Differential Effects

• Sameness of Inference

Four attributes that differentiate an accommodation from a

Modification. As one moves closer to a modification, the validity of

the inferences from a test score becomes more suspect.

Hollenbeck (2002)

Access Skills & Targeted Skills

Testing accommodations facilitate access to a test and should reduce the error in test scores due to relatively poor access skills.

Appropriate testing accommodations should not change or replace the skills that the test targets or is designed to measure.

2 metaphors: Eye glasses & Access Ramp

Target SkillsTest items are designed to measure specific skills or abilities. Forexample, many mathematics items are intended to measure astudent’s ability to reason, compute, and communication a solution orresult. The skills or abilities that test developers intend the items tomeasure can be called target skills or abilities. The samemathematics items require a student to attend, read, remember someinformation, and ultimately respond by bubbling in an answer choiceor writing an extended response.These latter skills are generally Notwhat the test developers designed the mathematics items to measure,but without these skills or abilities students can not access or interactwith the test items to demonstrate whether or not they posses thetarget skills measured by the items. IEP Teams members must know what the test Measures!!!

Key Access Skills*some access skills can also be target skills

1. Attending

2. Listening

3. Reading*

4. Remembering

5. Writing*

6. Following directions

7. Working alone

8. Sitting quietly

9. Turning pages

10. Locating test items

11. Locating answer spaces

12. Erasing completely

13. Seeing

14. Processing information in a timely manner

15. Working for a sustained period of time

16. Spelling*

Testing Accommodation Guidelines• Every state has some written guidance concerning

allowable testing guidelines. • Most test publishers also have a generic set of testing

guidelines that provide conservative guidance about the influence of testing accommodations on the validity of test score inferences (e.g., http://www.ctb.com/media/articles/pdfs/general/guidelines_inclusive.pdf)

IEP Team Members Must Know Their State’s Testing Accommodation Guidelines!

You Must Know the Test

• It is critical that the IEP team be familiar with state standards and assessments before the test or assessments are administered.

• The IEP team must have a working knowledge of the test format and what skills and knowledge are being measured by statewide assessments.

• Knowledge of the test and student’s abilities drive

decisions about participation and accommodations.

Technical Qualities of a Test Reliability: A reliable test provides you consistent results.That is, it is accurate over repeated uses.

Validity: A valid test* measures what it says it measures.That is, it allows you to make an inference about a person’sknowledge or skill from his or her score on a given test.

Reliability is a necessary, but not sufficient conditionfor validity!

Approaches to Interpreting Assessments

NRT (norm-referenced testing): Comparing people to people.

CRT (criterion-referenced testing): Comparing people to a standard of performance. Virtually all tests for NCLB use a CRT approach to interpretation.

A CRT approach to test interpretation “tolerates” diversity in testing

conditions as long as the construct being measured is unchanged!

Invalidation of a test score means that changes in the test presentation, administration, content, or response format have resulted in changes in what the test measures. Therefore, the inference one makes from a given test score is not justified or comparable to other scores.

The Interaction Effect: Validity Evidence A common framework for interpreting the validity of

accommodations has been to look for an interaction between educational status (students w/disabilities & students without disabilities) and accommodation conditions (accommodated & not accommodated).

The expectation has been that accommodations would improve the average score for a group of students with disabilities, but would have little or no affect on the average score of a group of non-disabled students.

Illustration of the Interaction Paradigm

H

L

No Acc Acc

SWD

SWOD

Test Scores

IEP Teams Must Understand Research Findings on Testing Accommodations

• Most frequently used testing accommodation packages include extra time, shorten sessions, & reading support,

• Typical effects of “appropriate” accommodation packages on test scores (ES = .40 for SWDs),

• Questionable accommodations: Reading a reading test & use of calculators when mental computation is the target skill.

• Testing accommodations have some unintended consequences too!

• Implementation integrity is a growing concern.

5-Step Systematic Decision-Making and Documentation Process

The Assessment Accommodations Checklist provides a framework for IEP team’s accommodation selection, implementation planning, and documentation.

1. Step 1 – Complete Student Information

2. Step 2 – Meet with IEP Team to Select Accommodations

3. Step 3 – Communicate the Accommodation Plan with Parent(s)

4. Step 4 – Implement the Testing Accommodation Plan

5. Step 5 – Report and Evaluate the Use of the Testing Accommodations

(Elliott, Kratochwill, & Schulte, 1999)

Dos & Don’ts for IEP Teams• Do systematically use accommodations during instruction and carry

these into the assessment process. • Do base the decision about accommodations, both for instruction and

for assessment, on the needs of the student.• Do consult the district or state list of approved accommodations after

determining what accommodations the student needs. Then, reevaluate the importance of the accommodations that are not allowed. If they are important for the student, request their approval from the district or state.

• Don’t introduce a new accommodation for the first time for an assessment.

• Don’t base the decision about what accommodations a student will use on the student’s disability category.

• Don’t start from the district or state list of approved accommodations when considering what accommodations a student will use in an upcoming test.

Source: Thurlow, J. Elliott, and Ysseldyke (1998, pp. 61-62)

Making Assessment Decisions:A Summary

• The IEP team makes decisions• Decisions are made on an individual basis• Decisions are documented in the IEP• Decisions are based on access needs; that is, accommodations are

designed to affect deficits in access skills, not target skills.• Decisions about testing accommodations must not invalidate test

results; the research literature provides some guidance, but a solid understanding of the concept of validity is needed

• Decisions are made prior to the testing date• The testing event should be reviewed to determine success of

accommodation effort and a record to guide future decisions documented.

Questions & Comments

• I welcome your questions and reactions to any of the points I have made today.

• I encourage resource sharing and will gladly provide you additional resources, if I have them.

Thank you!

• I appreciate the opportunity to share my alternate assessment ideas and experiences with you.

• Contact Information:

Steve.elliott@vanderbilt.edu

615-322-2538