Date post: | 22-Dec-2015 |
Category: |
Documents |
View: | 213 times |
Download: | 1 times |
Identifying Students in Need of Modified Achievement Standards
and Developing Valid Assessments
Who are the students needing modified achievement
standards?
….and thoughts for developing eligibility criteria
Sue Bechard and Judy Snow
January 16, 2008
Washington D.C.
01.16.08 MT GSEG: Bechard & Snow 3
Who are the students needing Modified Achievement Standards (MAS)?
Can include:
□ 2% of the total student population who can be counted as proficient on MAS,
□ students with disabilities, from any of the 13 disability categories,
□ students who are addressing grade level content standards on their IEPs, but are not expected to meet grade level achievement standards in the current year
□ students who need less difficult test items, covering the same breadth of content.
US Dept of Ed., 2007
01.16.08 MT GSEG: Bechard & Snow 4
Determination of eligibility for MAS….
Must consider….□ objective evidence□ multiple measures of progress over time□ IEP goals that are based on grade level content standards□ providing students the opportunity to show what they know
and can do on an assessment that is based on grade-level academic achievement standards.
Must not consider….□ a specific disability category□ racial or economic background
US Dept of Ed., 2007
01.16.08 MT GSEG: Bechard & Snow 5
Population Identification Issues
□ On average, students with disabilities comprise approximately 10% of the total student population.
□ If 10% - 1% = 9%, what characteristics should be used to distinguish the students appropriate for the 2% option within this group?□ Are there test performance distinctions?□ Are there specific learning characteristics?□ Are there specific learning needs for access to the
general curriculum?□ What are the “modified” expectations relative to grade
level content which distinguishes this population?
01.16.08 MT GSEG: Bechard & Snow 6
Population Identification Issues (cont.)
□What is the purpose of the AA-MAS?□To allow students to do better (AYP
improvement)?□To provide better information for instructional
planning (data on what students know)?□To increase student self-esteem?□To provide greater alignment between
cognition, expectations, instruction and assessment?
□Is an AA-MAS needed at every grade level/content area?
01.16.08 MT GSEG: Bechard & Snow 7
Selected results from prior research….
□Colorado: Report from the HB 05-1246 Study Committee, December, 2005
□New England Compact (RI) Enhanced Assessment Grant, 2004-2006
□Montana General Supervision Enhancement Grant, 2005-2007
01.16.08 MT GSEG: Bechard & Snow 8
CO report: Low Performers Who Score in the Bottom 1/3 of Scale Scores
□ Students who score in the bottom one third of scale scores on CSAP are almost twice as likely to be Black or Hispanic as students of other ethnicities.
□ Only 60% of students with IEPs scoring at lowest possible scale scores were able to be matched with a test the following year; thus, they may be more mobile than their counterparts who score at higher levels.
□ For those students scoring in the bottom one-third of scale scores, and where a match the following year was able to be made, it was found that these students did make substantial longitudinal growth.
01.16.08 MT GSEG: Bechard & Snow 9
CO report: Students with IEPs Who Do Not Make Longitudinal Growth
□ On the Colorado CSAP Reading Test, there were 250 students (of 444,407) across grade levels that were determined to be “Students in the Gap”.
□ On the CSAP Math Test, there were 658 students (of 444,910) that were determined to be “Students in the Gap”.
□ The CSAP as currently administered may not reflect their academic achievements; however, if appropriate accommodations and more intensive instruction were provided, these students too may make more gains.
01.16.08 MT GSEG: Bechard & Snow 10
10
Georgia EAG
Also looked at snapshot vs. longitudinal growth:
□Low Performing: lowest performance level in at least one assessment
□Persistently Low: lowest performance level for three consecutive years
Melissa Fincher, July 2007
01.16.08 MT GSEG: Bechard & Snow 11
Rhode Island (New England Compact - NEC) EAG
Teacher judgments of class work were compared to test performances and revealed two gaps of students performing below proficient:
Performance gap □ The test may not reflect classroom performance. Teachers see
students performing proficiently in class, but test results are below proficient.
Information gap □ The test may not be helpful for instructional planning. Teachers
rate students’ class work as low as possible and test results are at “chance” level. No information is generated on what students can do.
Parker & Saxon, 2007 Bechard & Godin, 2007
01.16.08 MT GSEG: Bechard & Snow 12
NEC EAG data sources□ State assessment data – grade 8 mathematics
results from two systems□ General large-scale test results□ Demographics (special programs, ethnicity, gender)□ Student questionnaires completed at time of test□ Accommodations used at time of test
□ State special education data □ Disability classification□ Free/reduced lunch□ Attendance
□ Classroom teacher data□ Individual interviews□ Judgments of all students’ classroom work
01.16.08 MT GSEG: Bechard & Snow 13
NEC EAG findingsThe Information Gap in grade 8 mathematics comprised 2.3-
4.3% of the total population
□ Included non-disabled students. □ Test performance:
□ Students mostly guessed on the test items. □ Most used multiple accommodations.
□ Teacher perceptions: □ These students operate below grade level in class. □ Teachers are not surprised by their low test results.□ There is a disconnect in what is tested vs. what is taught.□ These students need more supports in the classroom.
□ Student perceptions: □ Think the test is harder than their classroom work. □ They try hard on the test.
01.16.08 MT GSEG: Bechard & Snow 14
NEC EAG: Special program status of students in the
Information GapBreakdown of Sub-Populations within Information Gap and Comparison Groups
Within Group IEP only ELL only IEP&ELL General Ed Information gap (n=185)
80.0% 6.5% 2.7% 10.8%-
Non-gap comparison (n=369)
69.4% 9.8% 1.6% 19.2%
Overall Population
15.1% 1.9% 0.2% 82.8%
Non-gap comparison = students who performed at “chance,” on the test but higher in the classroom.
The majority of students performing at “chance’ were students with IEPs.
01.16.08 MT GSEG: Bechard & Snow 15
NEC EAG: Disability designations of students in the Information Gap
Learning disabilities: □ Lower percentages of students with SLD
were in the information gap than in the general population.
Other disabilities:□ deaf/blind□ multiple disabilities□ hearing impairments□ mild to moderate cognitive disabilities□ combinations of disabilities.
01.16.08 MT GSEG: Bechard & Snow 16
Montana GSEG, 2005 Students in the Sample□Grade 5 students statewide □Census sample of:
□Students with an IEP□Who took Spring 2006 Grade 4 math CRT□A few (13) who scored well on the Alternate
Assessment also included, selected by scores and recommendations from IEP teams
□ CRT-M = 672 students, CRT = 199 studentsMontana Office of Public Instruction and Measured Progress
01.16.08 MT GSEG: Bechard & Snow 17
MT GSEG, 2005 data sources
□Pilot test results□Student survey□Test administrator survey□Standard setting results□Recorded discussions of standard
setting panelists□Interviews with standard setting
panelists
01.16.08 MT GSEG: Bechard & Snow 18
MT GSEG 05: Test Information Functions
Test Information
0
5
10
15
20
25
30
35
40
45
50
-4 -2 0 2 4
Theta
Info
rma
tio
n
CRT
CRT_M
01.16.08 MT GSEG: Bechard & Snow 19
MT GSEG 05: Performance Level Comparisons, CRT-M vs. CRT
Up Three Levels
7
1% All 7 went to Advanced
Up Two Levels
120
18% All 120 went from below proficient to proficient or advanced
Up One Level282
42% 88 went from nearing to proficient, 96 went to advanced
No Change 234 35%
Down One 23 3% 11 went from proficient to nearing proficiency
Down Two 5 .7%
Down Three 1 .1%
01.16.08 MT GSEG: Bechard & Snow 20
MT GSEG 05 student and teacher surveys: Difficulty
• Students taking the CRT-M found the test slightly less difficult when compared to classroom content than students taking the regular CRT.
• Teachers felt the modified test should be modified more to reach the students having with the greatest challenges.
01.16.08 MT GSEG: Bechard & Snow 21
MT GSEG 05 feasibility question: Is the CRT-M a better measure?
□Students answered more items right□Student scores went up□Students moved up to another proficiency
level□Validity indicators improved
□More data analysis and study to determine which CRT-M students benefited most
01.16.08 MT GSEG: Bechard & Snow 22
Selected considerations from current research….
□Montana (+ NEC) EAG, 2007-2009: Adapting Test Items to Increase Validity of Alternate Assessments Based on Modified Achievement Standards
□Montana GSEG, 2007-2010: Identifying Students in Need of Modified Achievement Standards and Developing Valid Assessments
01.16.08 MT GSEG: Bechard & Snow 23
MT EAG, 2007
Focus on high school reading comprehension to:□determine the processing requirements
of test passages and items (use coding strategy)
□describe the cognitive abilities and challenges of the target population
□conduct cognitive labs□develop item modifications based on
cognitive variables
01.16.08 MT GSEG: Bechard & Snow 24
Preliminary feedback from Expert Panel (01-11-08)
□ Some noted cognitive variables:□ Abstract reasoning that relies on information from entire
passage□ Long passages that require sustained attention□ Limited experience with multiple meanings of
vocabulary words□ Dense passages that require large amounts of working
memory□ Location in the passage where necessary information is
found to answer the question□ Irrelevant information in passage makes mapping and
sorting difficult□ Emotional content difficult for students with ED to
process□ Answers to questions not found in passage (e.g. reliance
on prior knowledge)
01.16.08 MT GSEG: Bechard & Snow 25
Montana GSEG, 2007
Focus on middle school reading and mathematics to:
□ Identify students in need of modified achievement standards (MAS).
□ Determine what content knowledge the student is lacking to achieve proficiency
□ Develop dynamic online assessment that provides scaffolding based on distractor selection .
01.16.08 MT GSEG: Bechard & Snow 26
Students who will be included in the MT GSEG study samples
Middle school reading and mathematics□Sample for analyzing items and
distractors: All students who took the tests of interest, disaggregated
□Sample for cognitive labs: convenience sample of 48 students (24 per content area)
□Sample for pilot test: Approximately 5% of the total population
01.16.08 MT GSEG: Bechard & Snow 27
Cumulative%
P.L.%
Cumulative%
P.L.%
-4.00 0 0.0% 0.0%
-2.99 1 0.0% 0.0%
-2.57 2 0.0% 0.0%
-2.31 3 0.0% 0.0%
-2.12 4 0.0% 0.2%
-1.97 5 0.0% 0.2%
-1.85 6 0.0% 0.2%
-1.74 7 0.1% 0.3%
-1.64 8 0.1% 0.5%
-1.56 9 0.2% 1.1%
-1.48 10 0.2% 1.1%
-1.40 11 0.3% 9.4% 1.6% 42.5%
-1.33 12 0.3% 1.8%
-1.27 13 0.5% 3.0% 1
-1.21 14 0.7% 4.2%
-1.15 15 0.9% 5.5%
-1.09 16 1.2% 8.0%
-1.03 17 1.5% 10.5%
-0.98 18 1.8% 12.0%
-0.93 19 2.2% 14.5%
-0.88 20 2.7% 17.1%
-0.83 21 3.2% 19.8%
-0.78 22 3.7% 21.9%
-0.74 203 23 4.3% 24.7%
-0.69 206 24 5.1% 26.9%
-0.65 209 25 5.8% 30.0%
-0.60 213 26 6.6% 33.3%
-0.56 216 27 7.4% 36.4%
-0.51 219 28 8.4% 39.4% Research Population-0.47 222 29 9.4% 42.5% N = 581
-0.42 225 30 10.6% 46.5% 54.9%
-0.38 228 31 11.9% 50.0% 5.2%
-0.34 232 32 13.2% 52.6%
-0.29 235 33 14.6% 55.7% 2
-0.25 238 34 16.0% 58.9%
-0.21 241 35 17.8% 61.9%
-0.16 244 36 19.6% 65.4%
-0.12 247 37 21.5% 68.7%
-0.07 251 38 23.5% 71.0%-0.03 254 39 25.8% 73.4%0.02 257 40 28.3% 75.9%0.06 260 41 31.3% 79.4%0.11 263 42 34.2% 82.2%0.16 266 43 37.6% 83.9%0.21 270 44 40.9% 85.7%0.26 273 45 44.5% 87.6%0.31 276 46 48.2% 88.9%0.37 279 47 52.2% 90.5%0.42 282 48 56.1% 92.3%0.48 285 49 60.4% 94.2%0.54 288 50 64.6% 95.2%
% of IEP Population =% of Full Population =
raw score chance
chance + error
scale score floor (RI EAG)
floor + error
26.5%
200
12.0%
43.1%
PerformanceLevel
FullPopulation
(N = 11,174)
IEPPopulation
(N = 1,058)*
ThetaScaleScore
RawScore
3
26.2%
MT CRT
Grade 10 Reading Example, 2007
01.16.08 MT GSEG: Bechard & Snow 28
Implications of research for identification of target population
Use of performance data :□Longitudinal performance data□Students who are so low performing,
nothing is known about them□Match between classroom
performance and test performance□Distractor analyses
01.16.08 MT GSEG: Bechard & Snow 29
Implications of research for identification of target population (cont.)
Use of other data:
□Teacher judgment data□Opportunity to learn variables
□Mobility□Attendance□Program placement
□Performance data analyzed by cognitive modeling information
□Data from standards-driven IEPs
01.16.08 MT GSEG: Bechard & Snow 30
Data collected for “The Whole IEP Process” (C.
Massanari)
What is the desired outcome for this student? Three to four years from
now Student’s desired post-
school outcome
What are the skills and knowledge essential to meeting the desired outcome?
What are the expectations of the general curriculum relative to the student’s age/grade? Content Expectations for learning and
demonstration of learning Extracurricular activities or
events available
01.16.08 MT GSEG: Bechard & Snow 31
How do skills and knowledge essential to meeting the desired outcome compare with the general curriculum, including content and expectations for learning? Where are the similarities/connections?Where are the differences?Where within the general curriculum,
including extracurricular, are the opportunities for learning the needed skills and knowledge?
What are the student’s present levels of performance? What skills and knowledge does
the student already possess? What other strengths does the
student present? What are the areas of challenge? What accommodations,
modifications, or other supports have proven beneficial for this student?
Given all the information we have discussed thus far, what do we think are reasonable goals for this year? What are the objectives for each
goal? What instructional accommodations
are needed? What modifications to the general
curriculum are needed? How will progress be reported and
how often?
Given the information we have discussed thus far, how will the student participate in state and district-wide assessments? With peers as given With peers and with
accommodations or modifications
Alternate assessment
01.16.08 MT GSEG: Bechard & Snow 32
Also, consider how model of 1% eligibility guidelines might apply
For example: Montana's eligibility questions for the CRT-Alt. The student MUST:
□ Program: Have an active IEP□ Learning characteristics: Have cognitive abilities
and adaptive behaviors which require substantial adjustments to general curriculum
□ Learning objectives and expected outcomes: Focus on functional application of skills
□ Delivery of instruction: Requires direct and extensive instruction
01.16.08 MT GSEG: Bechard & Snow 33
Consider how model of 1% eligibility guidelines might apply (cont.)
Montana's eligibility questions for the CRT-Alt. Decisions must NOT be based on:
□ Excessive or extended absence □ Disability category□ Social, cultural, or economic differences □ Amount of time receiving special education
services□ Expectation of failure on general test
01.16.08 MT GSEG: Bechard & Snow 34
So…2% eligibility considerations might address:
Learning characteristics: □What are the cognitive abilities and adaptive
behaviors of the target population? □What adjustments are needed for the student
to participate in the general curriculum (e.g., accommodations/modifications)
Learning objectives and expected outcomes: □How does the student demonstrate application
of learned knowledge, skills, and abilities?
01.16.08 MT GSEG: Bechard & Snow 35
2% eligibility considerations (cont.)
Delivery of instruction: □What are the deconstructions of constructs
necessary for the student to master the grade level content?
□What adjustments must be made to simplify the materials used in instruction?
Academic achievement:□How is the progress of this student different
from the pattern of progress typical for all students at the targeted grade level?
01.16.08 MT GSEG: Bechard & Snow 36
References□ Bechard, S. and Godin, K. (2007). Identifying and Describing
Students in the Gaps in Large-Scale Assessment Systems. Paper submitted for publication.
□ Colorado Department of Education. (2005, December). Assessing “students in the gap” in Colorado: Report from the HB 05-1246 Study Committee. Denver: Author.
□ Montana Office of Public Instruction and Measured Progress. (2007, May). Determining the Feasibility of an Alternate Assessment Based on Modified Achievement Standards: A Planning Project and Pilot Test [Final Report for Montana’s General Supervision Enhancement Grant CFDA 84.373X – Priority B]. Helena: MT.
□ Parker, C. E., & Saxon, S. (2007). “They Come to the Test, and There is Nothing to Fold”: Teacher Views of Students in the Gaps and Large-Scale Assessments. Paper submitted for publication.
□ Title I—Improving the Academic Achievement of the Disadvantaged; Individuals With Disabilities Education Act (IDEA). Final rule. 72 Fed. Reg. 17748–17781, pts. 200 and 300 (2007, April 9).