OFFICE OF ELEMENTARY AND SECONDARY EDUCATION...must submit satisfactory evidence to address the...

UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF ELEMENTARY AND SECONDARY EDUCATION

The Honorable Tony Evers January 13, 2017 State Superintendent Wisconsin Department of Public Instruction 125 South Webster Street Madison, WI 53703 Dear Superintendent Evers: Thank you for your participation in the U.S. Department of Education’s (Department) assessment peer review process under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB). The Every Student Succeeds Act (ESSA) maintains the essential requirements from NCLB that each State annually administer high-quality assessments in at least reading/language arts, mathematics, and science that meet nationally recognized professional and technical standards. Therefore, as you know, the Department reinstituted peer review of State assessment systems so that each State receives feedback from external experts on the assessments it is currently administering. We appreciate the efforts required to prepare for the peer review, which occurred in June 2016. State assessment systems provide essential information that States, districts, principals, and teachers can use to identify the academic needs of students, target resources and supports toward students who need them most, evaluate school and program effectiveness, and close achievement gaps among students. A high-quality assessment system also provides useful information to parents about their child’s advancement against and achievement of grade-level standards. The Department’s peer review of State assessment systems is designed to provide feedback to States to support the development and administration of high-quality assessments. On October 6, 2016, the Department sent a letter to chief State school officers outlining the outcomes for States related to the assessment peer review. I am writing to provide you feedback on your State’s recent submission of evidence. External peer reviewers and Department staff evaluated the Wisconsin Department of Public Instruction’s (WIDPI) submission and found, based on the evidence received, that the components of your assessment system meet some, but not all of the statutory and regulatory requirements of section 1111(b)(1) and (3) of the ESEA, as amended by NCLB. Based on the recommendations from this peer review and our own analysis of the State’s submission, I have determined the following:

• Reading/language arts (R/LA) and mathematics general assessments in high school (ACT): Partially meets requirements.

• R/LA and mathematics alternate assessments based on alternate academic achievement standards (AA-AAAS) in grades 3-8 and high school (Dynamic Learning Maps-Year-End Model (DLM-YE)): Substantially meets requirements

The components that substantially meet requirements meet most of the requirements of the statute and regulations but some additional information is required. The Department expects that WIDPI should be able to provide this additional information within one year. The component that partially meet requirements does not meet a number of the requirements of the statute and regulations and WIDPI will need to provide substantial additional information to demonstrate it meets

Page 2 – The Honorable Tony Evers the requirements. The Department expects that WIDPI may not be able to submit all of the required information within one year. The specific list of items required for WIDPI to submit is enclosed with this letter. Because one of the State’s components has partially met the requirements, the Department is placing a condition on the State’s Title I grant award related to those components of the assessment system. To satisfy this condition, WIDPI must submit satisfactory evidence to address the items identified in the enclosed list. WIDPI must submit a plan and timeline within 30 days for when it will submit all required additional documentation for peer review. The Department will also host regular (e.g., quarterly) progress calls with the State to discuss the State’s progress on its timeline. If, following the peer review of the additional evidence, adequate progress is not made, the Department may take additional action. Additionally, the Office of Special Education and Rehabilitative Services (OSERS) will monitor progress on matters pertaining to requirements in the Individuals with Disabilities Education Act (IDEA) related to the participation of students with disabilities in Title I assessments. Insufficient progress to address such matters may lead OSERS to place a condition on WIDPI’s federal fiscal year 2017 IDEA Part B grant award. The Department notes that WIDPI submitted a waiver request for assessing speaking and listening that was approved on June 24, 2016, for the 2016−2017, 2017−2018, and 2018−2019 school years. In addition, the full peer review notes from the review are enclosed. These recommendations to the Department formed the basis of the Department’s determination. Please note that the peers’ recommendations may differ from the Department’s feedback; we encourage you to read the full peer notes for additional suggestions and recommendations for improving your assessment system beyond what is noted in the Department’s feedback. Department staff will reach out to your assessment director in the next few days to discuss the peer notes and the Department’s determination and to answer any questions you have. Thank you for your ongoing commitment to improving educational outcomes for all students. I look forward to our continued partnership as we move ahead with this critical work. I appreciate the work you are doing to improve your schools and provide a high-quality education for your students. If you have any questions, please contact Lisa Sadeghi or Porscheoy Brice of my staff at: [email protected]. Sincerely, /s/

Ann Whalen Senior Advisor to the Secretary

Delegated the Duties of Assistant Secretary for Elementary and Secondary Education

Enclosures cc: Visalakshi Somasundaram, Director of Student Assessment

Page 3 – The Honorable Tony Evers

3

Critical Elements Where Additional Evidence is Needed to Meet the Requirements for Wisconsin’s Assessment System Critical Element Additional Evidence Needed 1.5 – Participation Data

WIDPI must provide: • Evidence of test participation for the State assessments in science.

2.1 – Test Design and Development

For the reading/language arts (R/LA) and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that the test design measures the full range of the State’s grade-

level academic content standards (e.g., evidence of alignment of the test design blueprint to academic content standards). This evidence should include information about the State’s plan to assess the full breadth of the State’s R/LA standards, including speaking and listening. (Note: WIDPI has received a speaking and listening waiver; therefore, the Department does not expect WIDPI to submit additional evidence regarding speaking and listening during the period of the waiver).

• Evidence for the R/LA tests that describe the use of writing and reading test scores to support the intended interpretations and use of the results for R/LA accountability purposes.

For R/LA AA-AAAS in grades 3-8 and high school (DLM-YE), WIDPI must provide: • Evidence that the assessment design measures the State’s academic

content standards, including the language domain, or presents an explanation as to why this domain was not included.

3.1 – Overall Validity, including Validity Based on Content

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that independently establishes alignment, specifically that:

o Each assessment is aligned to its test blueprint, and each blueprint is aligned to the full range of State’s academic content standards, including speaking and listening in R/LA (Note: WIDPI has received a speaking and listening waiver; therefore, the Department does not expect WIDPI to submit additional evidence regarding speaking and listening during the period of the waiver); or

o Each assessment is aligned to the full range of the State’s academic content standards, and the procedures the State follows to ensure such alignment during test development; and

o Describes a systematic process and timeline to address any gaps or weaknesses identified through analysis of alignment.

• See evidence in 2.1 above regarding the use of writing and reading test scores to support the intended interpretations and use of the results for R/LA accountability purposes.

3.3 – Validity Based on Internal Structure

For the R/LA general assessments in high school (ACT), WIDPI must provide: • Evidence that establishes the ACT reading, English, and writing tests as a

single R/LA construct. Evidence may include: o Reports of analyses of the internal structure of the assessments (e.g.,

tables of item correlations) that show the extent to which the


Critical Element Additional Evidence Needed interrelationships among subscores are consistent with the State’s academic content standards for relevant student groups; OR

o Reports of analyses that show the dimensionality of the assessment is consistent with the structure of the State’s academic content standards and the intended interpretations of results; OR

o Evidence that ancillary constructs needed for success on the assessments do not provide inappropriate barriers for measuring the achievement of all students, such as evidence from cognitive labs or documentation of item development procedures; OR

o Reports of differential item functioning (DIF) analyses that show whether particular items (e.g., essays, performance tasks, or items requiring specific knowledge or skills) function differently for relevant student groups.

3.4 – Validity Based on Relationships with Other Variables

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that shows the assessment scores are related as expected with

criterion and other variables for all student groups (e.g., reports of analyses that demonstrate positive correlations between State assessment results and assessments of the same content area administered by some or all districts in the State).

4.1 – Reliability

For the R/LA general assessments in high school (ACT), WIDPI must provide: • Evidence that supports the reliability for the composite R/LA test scores

using the State’s data from test administration, such as: o Reliability estimates for the State overall and major reporting sub-

groups. o Standard error of measurement for the State overall and major

reporting sub-groups. o Estimates of classification accuracy and decision consistency for the

State overall and major reporting sub-groups.

For R/LA AA-AAAS in grades 3-8 and high school (DLM-YE), WIDPI must provide: • Evidence of monitoring and refinement of the diagnostic classification

models from subsequent test administrations. 4.2 – Fairness and Accessibility

For R/LA AA-AAAS in grades 3-8 and high school (DLM-YE), WIDPI must provide: • Evidence of the development and selection of reading passages that

includes information about steps that test developers have taken to ensure reading passages are accessible to students with significant cognitive disabilities.

• Evidence of the development and selection and/or creation of graphic components in the assessment (e.g., drawn or photographed images) that includes information about steps that test developers have taken to ensure passages from general grade-level texts are made accessible to students with the most significant cognitive disabilities.

4.4 – Scoring For the R/LA general assessments in high school (ACT), WIDPI must provide:


Critical Element Additional Evidence Needed • Evidence on the hand scoring for the writing tests, such as rater recruiting

criteria, training, range finding/calibration, validity papers, and procedures to reconcile discrepant ratings among human scorers.

For R/LA AA-AAAS in grades 3-8 and high school (DLM-YE), WIDPI must provide: • Evidence of monitoring procedures used for scoring DLM-YE writing

items, including measures of inter-rater reliability. 4.7 – Technical Analysis and Ongoing Maintenance

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence of regular internal and external technical review of the ACT

testing program in the State, such as minutes from technical advisory committee (TAC) meetings and documentation of roles and responsibilities of TAC members.

5.1 – Procedures for Including Students with Disabilities

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that clarifies what specific accessibility tools are available to all

students, including students with disabilities, taking the ACT tests. For R/LA AA-AAAS in grades 3-8 and high school (DLM-YE), WIDPI must provide: • Evidence of clear explanations for parents of the differences between

assessments based on grade-level academic achievement standards and AA-AAAS, including any effects of State and local policies on a student’s education resulting from taking an AA-AAAS.

5.2 – Procedures for Including ELs

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • As noted in element 5.1, evidence that clarifies what specific accessibility

tools are available to all students, including ELs, taking the ACT tests. 5.3 – Accommodations

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence of a process to determine that the accommodations it provides

(i) are appropriate and effective for meeting the individual student’s need(s) to participate in the assessments, (ii) do not alter the construct being assessed, and (iii) allow meaningful interpretations of results and comparison of scores for students who need and receive accommodations and students who do not need and do not receive accommodations.

5.4 – Monitoring Test Administration for Special Populations

For the R/LA and mathematics general assessments in high school (ACT) and the R/LA and mathematics AA-AAAS (DLM-YE), WIDPI must provide: • Evidence of a process for monitoring testing of students with disabilities

and ELs to ensure that they are appropriately included in assessments and receive accommodations that are: o Consistent with the State’s policies for accommodations; o Appropriate for addressing a student’s disability or language needs

for each assessment administered; o Consistent with accommodations provided to the students during

instruction and/or practice;


Critical Element Additional Evidence Needed o Consistent with the assessment accommodations identified by a

student’s IEP Team or 504 team for students with disabilities, or another process for an EL; and

o Administered with fidelity to test administration procedures. 6.2 – Achievement Standards-Setting

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that provides greater detail about the achievement standard

setting process, including: o A description of the standards-setting method and process used by the

State; o The rationale for the method selected; o Documentation that the method used for setting cut scores allowed

panelists to apply their knowledge and experience in a reasonable manner and supported the establishment of reasonable and defensible cut scores;

o Documentation of the process used for setting cut scores and developing performance-level descriptors aligned to the State’s academic content standards;

o A description of the process for selecting panelists; o Documentation that the standards-setting panels consisted of panelists

with appropriate experience and expertise, including: Content experts with experience teaching the State’s academic

content standards in the tested grades; Individuals with experience and expertise teaching students with

disabilities, English learners and other student populations in the State;

As appropriate, individuals from institutions of higher education and individuals knowledgeable about career-readiness; and

A description, by relevant characteristics, of the panelists (overall and by individual panels) who participated in achievement standards setting.

6.3 – Challenging and Aligned Academic Achievement Standards

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that the State’s academic achievement standards are aligned

with the State’s academic content standards. • Evidence that the State’s academic achievement standards are

challenging. 6.4 – Reporting

For the R/LA and mathematics general assessments in high school (ACT), WIDPI must provide: • Evidence that test score reports and supporting material (a) reflect the

State’s test reporting categories (i.e., a single score for R/LA), and (b) provide information on the State’s academic achievement levels.

• Evidence that the score reports are available in alternative formats. • Evidence of a process and timeline for delivering reports to students,

parents, teachers, principals, and other stakeholders as soon as practicable after each test administration.

• Evidence of materials that support parents and educators in the use and interpretation of test scores.


STATE ASSESSMENT PEER REVIEW NOTES FOR WISCONSIN

U. S. Department of Education

Peer Review of State Assessment Systems

June 2016 State Assessment Peer Review

Notes

U. S. Department of Education Office of Elementary and Secondary Education

Washington, D.C. 20202

Note: Peer review notes provide the combined recommendations of the individual peers to the U.S. Department of Education (Department), based on the statute and regulations, the Department’s peer review guidance, and the peers’ professional judgement of the evidence submitted by the State. These assessment peer review notes, however, do not necessarily reflect the final set of additional evidence, if any, that a State may need to submit to demonstrate that its assessment system meets all of the critical elements for assessment peer review. Although the peer notes inform the Secretary’s consideration of each State’s assessment system, the Department makes the final decision regarding whether the assessment system meets the requirements in the statute and regulations. As a result, these peer notes may not completely align with the final determination made by the Department.


2 Consistent with the note on page 1, the evidence requested by the peer reviewers does not necessarily reflect the final set of additional evidence, if any, that a State may need to submit to demonstrate that its assessment system meets all of the critical elements for the assessment peer review. As a result, a State should refer to the letter to the State, including the list of additional evidence needed, if any, from the Department.

Contents

SECTION 1: STATEWIDE SYSTEM OF STANDARDS AND ASSESSMENTS . 3 1.1 – State Adoption of Academic Content Standards for All Students .... 3 1.2 – Coherent and Rigorous Academic Content Standards ...................... 4

1.3 – Required Assessments(REVIEWED BY DEPARTMENT STAFF ONLY) ........... 5

1.4 – Policies for Including All Students in Assessments(REVIEWED BY

DEPARTMENT STAFF ONLY) .................................................................................. 7

1.5 – Participation Data (REVIEWED BY DEPARTMENT STAFF ONLY) .................... 9

SECTION 2: ASSESSMENT SYSTEM OPERATIONS ..................................... 10

2.1 – Test Design and Development ............................................................ 10 2.2 – Item Development ................................................................................ 12 2.3 – Test Administration ............................................................................. 13

2.4 – Monitoring Test Administration (REVIEWED BY DEPARTMENT STAFF ONLY)14 2.5 – Test Security ........................................................................................ 15 2.6 – Systems for Protecting Data Integrity and Privacy ........................... 17

SECTION 3: TECHNICAL QUALITY – VALIDITY............................................. 18 3.1 – Overall Validity, including Validity Based on Content ...................... 18 3.2 – Validity Based on Cognitive Processes ............................................. 20

3.3 – Validity Based on Internal Structure .................................................. 21 3.4 – Validity Based on Relationships with Other Variables ..................... 22

SECTION 4: TECHNICAL QUALITY - OTHER ................................................. 24 4.1 – Reliability .............................................................................................. 24

4.2 – Fairness and Accessibility .................................................................. 26 4.3 – Full Performance Continuum .............................................................. 27

4.4 – Scoring ................................................................................................. 28 4.5 – Multiple Assessment Forms ............................................................... 29 4.6 – Multiple Versions of an Assessment ................................................. 30

4.7 – Technical Analysis and Ongoing Maintenance ................................. 31

SECTION 5: INCLUSION OF ALL STUDENTS ................................................ 32

5.1 – Procedures for Including Students with Disabilities ........................ 32 5.2 – Procedures for including ELs ............................................................. 34 5.3 – Accommodations ................................................................................. 35 5.4 – Monitoring Test Administration for Special Populations ................. 36

SECTION 6: ACADEMIC ACHIEVEMENT STANDARDS AND REPORTING . 37 6.1 – State Adoption of Academic Achievement Standards for All Students........................................................................................................................ 37

6.2 – Achievement Standards-Setting ......................................................... 38 6.3 – Challenging and Aligned Academic Achievement Standards ......... 39 6.4 – Reporting .............................................................................................. 40



SECTION 1: STATEWIDE SYSTEM OF STANDARDS AND ASSESSMENTS

Critical Element Evidence (Record document and page # for

future reference)

Comments/Notes/Questions/Suggestions Regarding State Documentation or Evidence

1.1 – State Adoption of Academic Content Standards for All Students

The State formally adopted challenging academic content standards for all students in reading/language arts, mathematics and science and applies its academic content standards to all public elementary and secondary schools and students in the State.

Evaluate for all subjects 1. #128 – Adoption of the CCSS for

ELA/LIT in History, Social Studies, Science, and Technical Subjects and the CCSS for Mathematics

2. #103 - WI CCSS and WI Common Core Essential Elements in ELA and Mathematics

3. #140 – WI ESEA Flexibility Request: Renewal Request (pp. 19-36)

4. #132 – ESEA Flexibility Renewal Letter 5. #133 – Assessment in WI required

assessments 6. #064 – About WI Accountability 7. #065 – School Accountability for 2014-15 8. #066 - School Accountability for 2015-16

Please note that this review is for (a) the Wisconsin’s ACT in reading/language arts and mathematics as the general assessments at grade 11 and (b) the Wisconsin’s Dynamic Learning Maps (DLM) in reading/language arts and mathematics as the alternate assessments at grades 3-8, and 11.

1. Wisconsin provides evidence to support that the state formally adopted the Common Core State Standards for English Language Arts & Literacy in History, Social Studies, Science, and Technical Subjects and the Common Core State Standards for Mathematics and the College and Career Readiness in 2010 (#128).

2. Wisconsin provides evidence to support that the state formally adopted the Alternate Academic Standards as the Wisconsin Common Core Essential Elements (CCEE) in English language arts and Mathematics in 2010 (#103).

3. The ESEA Flexibility Request Renewal (#132) confirms the adoption and the transition to the CCSS and CCR, the implementation of the new statewide assessments, and the alignment of the Wisconsin CCEE to the Wisconsin CCSS (#132 & #140).

Section 1.1 Summary Statement _X__ No additional evidence is required.




future reference)


1.2 – Coherent and Rigorous Academic Content Standards The State’s academic content standards in reading/language arts, mathematics and science specify what students are expected to know and be able to do by the time they graduate from high school to succeed in college and the workforce; contain content that is coherent (e.g., within and across grades) and rigorous; encourage the teaching of advanced skills; and were developed with broad stakeholder involvement.

Evaluate for all three subjects 1. #128 – Adoption of the CCSS for

ELA/LIT in History, Social Studies, Science, and Technical Subjects and the CCSS fro Mathematics

2. #103 - WI CCSS and WI Common Core Essential Elements in ELA and Mathematics

1. Wisconsin provides evidence to adopt the coherent and rigorous the Common Core State Standards in ELA/LIT and the Common Core Essential Elements (CCEEs) for student with significant cognitive disabilities.

2. It would be more helpful if this submission

includes information about the process by which feedback from stakeholders was used to revise the standards in the adoption of the academic content standards.

3. It would be more helpful if this submission includes document regarding to the alignment between the Wisconsin CCSS and the Wisconsin CCEE.

Section 1.2 Summary Statement __X_ No additional evidence is required.



Critical Element—REVIEWED BY DEPARTMENT STAFF ONLY

Evidence —REVIEWED BY DEPARTMENT

STAFF ONLY (Record document and page # for

future reference)

Comments/Notes/Questions/Suggestions Regarding State Documentation or Evidence —REVIEWED BY DEPARTMENT STAFF ONLY

1.3 – Required Assessments The State’s assessment system includes annual general and alternate assessments (based on grade-level academic achievement standards or alternate academic achievement standards-AAAS) in:

Reading/language arts and mathematics in each of grades 3-8 and at least once in high school (grades 10-12);

Science at least once in each of three grade spans (3-5, 6-9 and 10-12).

File #111 (p. 1)- general overview of assessment

program:

Wisconsin Forward Exam- 3rd

– 8th

grades:

ELA, math/ 4th

& 8th

grades: science/ 4th

,

8th

& 10th

: social studies

DLM- For 1% SWD students: 3rd

- 11th

grades: ELA, math/ 4th

& 8th

-11th

grades:

science/ 4th

, 8th

& 10th

grades: social

studies

ACT Aspire- 9th

and 10th

grades

ACT Plus Writing- 11th

grade: reading,

math, English, science, and writing

ACT Work Keys- 11th

grade

File #133 (p. 1)- general overview of assessment

program:

Same information as File #111

File #72 (p. 1)- description of high school

assessments

ACT Aspire Early High School- 9th

and

10th

grades: English, reading, math,

science, and writing

ACT Plus Writing- 11th

grade: reading,

math, English, science, and writing

ACT Work Keys- 11th

grade: applied

mathematics, locating information, and

reading for information

File #72 (p. 1)- describes DLM- Measures the

academic progress of students with significant

cognitive disabilities in the subject areas of:

ELA and mathematics 3rd

-11th

grades/

science 4th

& 8th

– 11th

grades/ social

studies 4th

, 8th

& 10th

grades.

State has required general and AA-AAAS assessments in high school for R/LA and math that are part of this reivew. State has required AA-AAAS in R/LA and math gr. 3-8 that are part of this review. State has required general assessments in grades3-8 in R/LA and math; and in general and AA-AAAS science 3-5, 6-8. and 9-12 but these assessments are not a part of this peer review.






future reference)


Section 1.3 Summary Statement—REVIEWED BY DEPARTMENT STAFF ONLY _x__ No additional evidence is required.






future reference)


1.4 – Policies for Including All Students in Assessments The State requires the inclusion of all public elementary and secondary school students in its assessment system and clearly and consistently communicates this requirement to districts and schools.

For students with disabilities(SWD), policies state that all students with disabilities in the State, including students with disabilities publicly placed in private schools as a means of providing special education and related services, must be included in the assessment system;

For English learners (EL): o Policies state that all English learners must

be included in the assessment system, unless the State exempts a student who has attended schools in the U.S. for less than 12 months from one administration of its reading/ language arts assessment;

o If the State administers native language assessments, the State requires English learners to be assessed in reading/language arts in English if they have been enrolled in U.S. schools for three or more consecutive years, except if a district determines, on a case-by-case basis, that native language assessments would yield more accurate and reliable information, the district may assess a student with native language assessments for a period not to exceed two additional consecutive years.

Evaluate for all factors in left hand column-SWD

and EL File #081 (pp 10-11)- EL information for ACT- requirement for all students to participate is stated, referencing the EL exemption File #068 (p. 1-4)- Accommodations for SWD for ACT File #109 (pp 2, 3 & 5)- DLM information in FAQ format that covers various issues for inclusion, and methods for test delivery to SWDs.

WI DPI states that they will include information on the Wisconsin Forward Exam in future peer reviews.






future reference)


Section 1.4 Summary Statement-REVIEWED BY DEPARTMENT STAFF ONLY __x_ No additional evidence is required.






future reference)


1.5 – Participation Data The State’s participation data show that all students, disaggregated by student group and assessment type, are included in the State’s assessment system. In addition, if the State administers end-of-course assessments for high school students, the State has procedures in place for ensuring that each student is tested and counted in the calculation of participation rates on each required assessment and provides the corresponding data.

File #129- ACT/DLM reporting screenshots from http://wisedash.dpi.wi.gov/Dashboard/Page/Home

Allows side-by-side views of assessment data (participation percentage and achievement) disaggregated by subgroup

File #134- 2014-15 participation rate tables

Shows participation rate for 3rd – 8th grades and 11th grade in Math and ELA. No science or writing scores shown in this evidence. No data for 9th and 10th grade assessments.

May need to provide data for science assessments at appropriate grade levels.

Section 1.5 Summary Statement-REVIEWED BY DEPARTMENT STAFF ONLY _X_ The following additional evidence is needed/provide brief rationale: May need to provide data for science assessments at appropriate grade levels.

http://wisedash.dpi.wi.gov/Dashboard/Page/Home



SECTION 2: ASSESSMENT SYSTEM OPERATIONS


future reference)


2.1 – Test Design and Development The State’s test design and test development process is well-suited for the content, is technically sound, aligns the assessments to the full range of the State’s academic content standards, and includes:

Statement(s) of the purposes of the assessments and the intended interpretations and uses of results;

Test blueprints that describe the structure of each assessment in sufficient detail to support the development of assessments that are technically sound, measure the full range of the State’s grade-level academic content standards, and support the intended interpretations and uses of the results;

Processes to ensure that each assessment is tailored to the knowledge and skills included in the State’s academic content standards, reflects appropriate inclusion of challenging content, and requires complex demonstrations or applications of knowledge and skills (i.e., higher-order thinking skills);

If the State administers computer-adaptive assessments, the item pool and item selection procedures adequately support the test design.

Evaluate for all factors in left hand column—all

tests and grades documented on cover sheet

Alignment: 1. #005 – How ACT assessments Align with

State CCR standards (pp.2-3) 2. #006 – The alignment of CCSS and ACT’s

CCR System 3. #054 – ACT Alignment Wisconsin

Purpose and Intended Interpretations:

1. #049 – ACT Technical Manual (pp.1-2; pp. 24-25)

2. #052 – Using your ACT results Test Blueprint:

1. #033 – ACT Reading Test Blueprint 2. #017 – ACT English and Writing Test

Blueprint 3. #027 - ACT Mathematics Test Blueprint 4. #049 – ACT Technical Manual (pp. 9-11;

16) Process:

1. #049 – ACT Technical Manual (pp. 7-15; pp. 17-37)

2. #011 – ACT National Curriculum Survey (p. 2)

For the alternate assessments – Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11, evidence was submitted on WI’s behalf by the state of WV.

For WI’s DLM Submission: Evidence about Test Design and Development of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Additional information is needed to clarify the purposes and intended uses of ACT scores (#049, #052).

2. The evidence of the alignment between the ACT items and the Wisconsin CCSS is based on the analysis by the test developer. A plan for an independent alignment study is expected (#005, #006, #054).

3. Insufficient information is provided on how the ACT ELA score was created and how the ACT ELA scores were used in assessment related activities (e.g., standard setting, reporting and interpretation of test scores).

4. No evidence is provided in regard to assessing Listening and Speaking or an application for waiver.




future reference)


Section 2.1 Summary Statement For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale:

Clarify the purposes WI intends to use the ACT scores for, such as school accountability.

A plan for an independent alignment study (or studies) that involves WI educators to determine the degree to which ACT aligns to the CCSS in ELA/LIT and in Mathematics. An improvement plan for the future item/test development to address areas of weak alignment is expected to follow the alignment study.

Detail how the combined ELA score is created and interpreted.




future reference)


2.2 – Item Development The State uses reasonable and technically sound procedures to develop and select items to assess student achievement based on the State’s academic content standards in terms of content and cognitive process, including higher-order thinking skills.

Evaluate for all factors in left hand column —all


1. #049 – ACT Technical Manual (pp. 7-16) 2. #022 – Forms Construction Guide (pp. 1-2;

2-3; 2-6; 2-9; 2-17) 3. #016 ACT Item Writer’s Guide for English

MC (pp. 2-8; 9-14; 32-46) 4. #035 - ACT Item Writer’s Guide for

Reading MC (pp. 4-10; 11-14; 21-27) 5. #019 – ACT English Essay Writer’s Guide 6. #030 – ACT Guide to Reading Test Passage

Selection 7. #028 – Item writer’s Guide for the ACT

Mathematics Test (pp. 6-15; 16-18; 19-23) 8. #004 – Your Guide to the ACT Assessment

(p. 3) 9. #043 – Sample Item Writer Assignment (p.

1) 10. #003 – 2011 Annual Item Writer Report

(pp. 5-6) Alternate (DLM) – Reading/language arts and mathematics grades 3-8, 11. Evidence for the assessment was submitted on WI’s behalf by the state of WV.

For WI’s DLM Submission: Evidence about Item Development of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission: The following comments from reviewers support the requests for additional evidence for the Critical Element 2.1 Test Design and Development.

1. WI provides ACT documents about the process for item development, such as item writer selection, training, item development guide, reading passage selection guide with details.

2. The alignment of the ACT item pool to the CCSS should be investigated, specifically focus on content and cognitive complexity.

3. The application of the Universal Design Principles could improve item development process.

Section 2.2 Summary Statement For WI’s ACT Submission: __X_ No new evidence is requested for this Critical Element, but the evidence requested under Critical Element 2.1 applies here as well.




future reference)


2.3 – Test Administration The State implements policies and procedures for standardized test administration, specifically the State:

Has established and communicates to educators clear, thorough and consistent standardized procedures for the administration of its assessments, including administration with accommodations;

Has established procedures to ensure that all individuals responsible for administering the State’s general and alternate assessments receive training on the State’s established procedures for the administration of its assessments;

If the State administers technology-based assessments, the State has defined technology and other related requirements, included technology-based test administration in its standardized procedures for test administration, and established contingency plans to address possible technology challenges during test administration.



1. #114 – Logistical Manual for the implementation of Online Assessments in WI

2. Samples of Newsletters # 117 - #121

3. Samples Interactive discussion #122 - #127

4. #074 – ACT State and District Testing Schedule of Events

5. Sample ACT Updates #098 - #102

6. #050 – The ACT Test Administration Manual (pp. 2-4; 5-6; 8-11; 66)

7. #050 - The ACT Test Administration Manual (pp. 9-10; 66-68)

8. #097 – ACT Testing Wisconsin 9. #084 – Providing Local Test Arrangements

on the ACT 10. #070 – Guiding Principles for ACT-

Approved Accommodations on the ACT

Alternate (DLM) – Reading/language arts and mathematics grades 3-8, 11. Evidence for the assessment was submitted on WI’s behalf by the state of WV.

For WI’s DLM Submission:

1. Evidence about Test Administration of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV.

2. It would be more helpful if WI provides supplemental information about the communication on the DLM test administration.

For WI’s ACT Submission:

1. WI provides sufficient evidence about test administration in policy, manuals/guidelines, training, timeline, and communication with LEAs for update information.

2. WI provides sufficient evidence about the accessibility for SWD, IEP, and EL students.

Section 2.3 Summary Statement For WI’s ACT Submission: _X__ No additional evidence is required.





STAFF ONLY(Record document and page # for

future reference)


2.4 – Monitoring Test Administration The State adequately monitors the administration of its State assessments to ensure that standardized test administration procedures are implemented with fidelity across districts and schools.

File #115 (pp 1-2)- District Coordinators- handout for training program to ensure proper testing procedures are followed File #116- Coordinator orientation presentation- provides resources for administration and monitoring of testing File #050- ACT Test Administration Manual

p. 3- equal treatment, fair testing practices, authorized observers

pp 28-32- room, test distribution, timing, announcements, etc.

pp 39-54- instructions File #058- DLM Assessment Coordinator Manual:

p. 7- coordinator checklist

p. 19- preparation information

pp 24-25- monitor and support information File #062- DLM Test Administration Manual

pp 51-53- allowable practices

pp 55-59- first steps and training File #136- DLM Test Admin. Monitoring Report- reporting tool/spreadsheet

1. File #061- DLM Test Security Incident Report Form- standardized reporting form

Evidence meets requirements for this element.

Section 2.4 Summary Statement—REVIEWED BY DEPARTMENT STAFF ONLY _x__ No additional evidence is required.




future reference)


2.5 – Test Security The State has implemented and documented an appropriate set of policies and procedures to prevent test irregularities and ensure the integrity of test results through:

Prevention of any assessment irregularities, including maintaining the security of test materials, proper test preparation guidelines and administration procedures, incident-reporting procedures, consequences for confirmed violations of test security, and requirements for annual training at the district and school levels for all individuals involved in test administration;

Detection of test irregularities;

Remediation following any test security incidents involving any of the State’s assessments;

Investigation of alleged or factual test irregularities.

Evaluate for all factors in left hand column for both

grade-level and AA-AAAS

For both ACT and DLM:

1. #131 – WI Test Security Manual (pp. 7;

15; 2-5; 8-15; 21; 15-17; 20; 6; 2-3)

For ACT:

1. #078 – District Assessment Coordinator

Confidentiality Agreement

2. #089 – School Assessment Coordinator


3. #090 – District Staff/School Staff

confidentiality Agreement Form

4. #094 – Proctor/Test Administrator

confidentiality Agreement

5. #095 – Test Coordinator Confidentiality

Agreement

6. #79 – District/School Technology

Coordinator confidentiality Agreement

7. #083 – ACT/Aspire/WorkKeys test

Security Incident Report Form

8. #050 – The ACT Test Administration

Manual (pp. 5-7; 8-12; 13-15; 22-23; 24-

27; 28-32; 39-57; 81)


Manual (pp. 9-10; 66-68)

10. #096 – ACT High School Assessments –

Test Security

11. #026 – Procedure for Investing Testing

Irregularities and Questioned Test Scores


Manual (pp.33-38)


Manual (pp. 9-10; 66-68)


1. Sufficient evidence is proved about Test Security through Prevention, Detection, and Investigation (#131, #110).


1. Sufficient evidence is proved about test security through Prevention, Detection, and Investigation (#131, #083)




future reference)


14. #049 – ACT Technical Manual (p. 16)

15. #037 – 2015-2016 Terms and Conditions:

Testing Rules and Policies for the ACT

(pp. 2-3)

16. #026 - Procedure for Investing Testing

Irregularities and Questioned Test Scores

(pp. 3-4)


Manual (p. 2)

There are policy, procedure, and implementation

for test security.

For DLM:

1. #142 – District Assessment Coordinator


2. #143 - District Technology Coordinator


3. #144 – District/School Report User


4. #145 - School Assessment Coordinator


5. #146 – School Technology Coordinator


6. #131 – Test Security Manual

7. #110 – DLM Test security

8. #061 – DLM test Security Incident Report

Form

Section 2.5 Summary Statement

For WI’s ACT Submission: _X__ No additional evidence is required




future reference)


2.6 – Systems for Protecting Data Integrity and Privacy The State has policies and procedures in place to protect the integrity and confidentiality of its test materials, test-related data, and personally identifiable information, specifically:

To protect the integrity of its test materials and related data in test development, administration, and storage and use of results;

To secure student-level assessment data and protect student privacy and confidentiality, including guidelines for districts and schools;

To protect personally identifiable information about any individual student in reporting, including defining the minimum number of students necessary to allow reporting of scores for all students and student groups.



For both ACT and DLM:

1. #115 – Information for New District Assessment Coordinators

2. #116 - New District Assessment Coordinator Orientation

3. #063 – Student Data Privacy Main menu 4. #104 – WISEdash Support – About Data

Redaction and Student Privacy 5. #105 WISE Data Requests 6. #106 – Welcome to the Data Warehouse &

Decision Support Team Page 7. #130 – Student Privacy

For ACT:

1. #050 – ACT Test Administration Manual (pp. 13-15)

2. #032 – ACT privacy policy 3. #025 – ACT Information Security Program

Summary For DLM:

1. #060 – Data Use Agreement between WI and UKA Center

2. Evidence for the assessment was submitted on WI’s behalf by the state of WV.


1. WI provides evidence to support the Systems for Protecting Data Integrity and Privacy. However, additional information is needed in regard to the security policy and practice for data held by the state.


1. WI provides evidence to support the system for protecting data integrity and privacy. However, additional information is needed in regard to the security policy and practice for data held by the state.

Section 2.6 Summary Statement For WI’s DLM Submission and ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

WI specific information on security policies and practices regarding WI’s data servers and databases (e.g., how the servers are secured from external security threats and details on long term data storage and management).



SECTION 3: TECHNICAL QUALITY – VALIDITY


future reference)


3.1 – Overall Validity, including Validity Based on Content The State has documented adequate overall validity evidence for its assessments, and the State’s validity evidence includes evidence that the State’s assessments measure the knowledge and skills specified in the State’s academic content standards, including:

Documentation of adequate alignment between the State’s assessments and the academic content standards the assessments are designed to measure in terms of content (i.e., knowledge and process), the full range of the State’s academic content standards, balance of content, and cognitive complexity;

If the State administers alternate assessments based on alternate academic achievement standards, the assessments show adequate linkage to the State’s academic content standards in terms of content match (i.e., no unrelated content) and the breadth of content and cognitive complexity determined in test design to be appropriate for students with the most significant cognitive disabilities.



1. #049 – ACT Technical Manual (pp. 64-

137)

2. #055 – ACT Writing Test Technical

Report (pp. 13-17)

3. #034 – Reading Content Panelists:

Instruction for Review

4. #006 – The Alignment of Common Core

and ACT’s College and Career Readiness

System (p. 6; Appendix A)

5. #054 – ACT Alignment Wisconsin (pp. 3-

6; 11; 14; 16-17; 24-26; 33; 49-61)

1. The PLDs seem to be definition for each

level without content standards and

performance expected from students at

each performance level

2. The PLDs are prepared by ACT and

reviewed by WI panels

3. The cut scores are set based on the

predictive approach about student success

in college, rather than to address what

student know and be able to do based on

the content standards.

4. The process for PLDs development is not

clear

For Alternate Assessment: Alternate (DLM) – Reading/language arts and mathematics grades 3-8, 11. Evidence for the assessment was submitted on WI’s behalf by the state of WV.

For WI’s DLM Submission: Evidence about Overall Validity, including Validity based on Content of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. The evidence of the alignment between the ACT items and the Wisconsin CCSS is based on the analysis by the test developer.

2. A plan for an independent alignment study with the participation of WI educators is expected to address adequate evidence in alignment between ACT and the WI CCSS and support that Wisconsin ACT is designed to measure in terms of content (i.e., knowledge and process), the full range of the State’s academic content standards, balanced content, and cognitive complexity.

3. The results from independent alignment study should be used for a future plan in the improvement of item/test development.

4. Insufficient information is provided in regard to how to create the ACT ELA score and use the ELA scores in related activities (e.g., standard setting, reporting and interpretation of test scores).

5. No evidence is provided in regard to assessing Listening and Speaking or an application for waiver.




future reference)


Section 3.1 Summary Statement For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

As noted later on under Critical Elements 6.2 and 6.3, information that details the development of the PLDs and the appropriateness of the PLDs addressing expectations based on CCSS.

As noted earlier under Critical Element 2.1, evidence on alignment and the creation of the combined ELA score. The requests for additional evidence are copied below from 2.1 Item Development. - Clarify the purposes WI intends to use the ACT scores for, such as school accountability. - A plan for an independent alignment study (or studies) that involves WI educators to determine the degree to which ACT aligns to the CCSS in

ELA/LIT and in Mathematics. An improvement plan for future item/test development to address areas of weak alignment is expected to follow the alignment study.

- Detail how the combined ELA score is created and interpreted.




future reference)


3.2 – Validity Based on Cognitive Processes The State has documented adequate validity evidence that its assessments tap the intended cognitive processes appropriate for each grade level as represented in the State’s academic content standards.



For ACT:

1. #049 – ACT Technical Manual (pp. 8; 13-15)

2. #022 – Forms Construction Guide (pp. 2.15-2.19; Appendix A; 2.2-2.5)


For WI’s DLM Submission: Evidence about Validity based on Cognitive Process of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Evidence is provided regarding validity based on cognitive processes in item development, item tryout, item analysis, and test form construction at the high school level.

Section 3.2 Summary Statement For WI’s ACT Submission: _X__ No additional evidence is required.




future reference)


3.3 – Validity Based on Internal Structure The State has documented adequate validity evidence that the scoring and reporting structures of its assessments are consistent with the sub-domain structures of the State’s academic content standards on which the intended interpretations and uses of results are based.


tests and grades documented on cover sheet For ACT:

1. #047 – Correlations among Subscores on the ACT

2. #012 – Differential Item Functioning Analysis

3. #055 – ACT Writing Test Technical Report (pp. 13-17)


For WI’s DLM Submission: Evidence about Validity based on Internal Structure of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Insufficient information is provided in regard to validity based on the internal structure. Principle components analyses or other related approaches can be used to provide evidence about the internal structure of ACT.

2. According to the WI’s submission, the ELA score is a composition of English, Reading, and Writing. The inter-correlations are based on the content standards measured under each test (English, Reading, and Writing) using South Carolina’s data (#047). It is assumed that once the way the ELA score created is clarified, new evidence will be provided on the internal structure of ELA scores based on WI student performance.


As noted previously under 2.1 and 3.1, how the combined ELA score is created is unclear. Thus the applicability of the evidence on internal structure is similarly unclear. Once the way the ELA score created is clarified, new evidence should then be provided on the internal structure of that ELA score or an explanation relating the evidence from this submission to that ELA scores.




future reference)


3.4 – Validity Based on Relationships with Other Variables The State has documented adequate validity evidence that the State’s assessment scores are related as expected with other variables.



1. #049 – ACT Technical Manual (pp. 67-69) 2. #010 – A multidimensional Perspective of

College Readiness 3. #024 – Influence of Achievement in Core

High School Course on ACT Scores 4. #049 - ACT Technical Manual (pp. 97-100) 5. #046 – Development of STEM Readiness

Benchmarks to Assist Educational and Career Decision Making

6. #051 – Updating the ACT College Readiness benchmarks

7. #023 – Who goes to graduate school? 8. #014 – Differential effects of using ACT

College Readiness Assessment Scores and High School GPA to predict First-Year College GPA

9. #009 – College Performance and Retention: A Meta-Analysis

10. #007 – Choosing among Multiple Achievement Measures

11. #013 – Different Assessments, Different Results

12. #044 – Selection Decisions for the ACT and SAT Score

13. #031 – The Relative Predictive Validity of ACT Scores and High Score Grades …


For WI’s DLM Submission: Evidence about Validity based on Relationships with Other Variables of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. WI provides some evidence on the

influence of achievement on the core high school course on ACT scores, and the relationships between high school GPA and ACT college readiness assessment score (#009, #014, #031).

2. WI should develop a plan to conduct similar investigations using WI student data, instead of relying solely on using national data.

3. A plan should be developed to collect additional validity evidence in the relationships with other variables from different measures, such as interim and formative assessments.




future reference)



A plan to document validity evidence on relationships with other variables using data from WI students when the data become available.



SECTION 4: TECHNICAL QUALITY - OTHER


future reference)


4.1 – Reliability The State has documented adequate reliability evidence for its assessments for the following measures of reliability for the State’s student population overall and each student group and, if the State’s assessments are implemented in multiple States, for the assessment overall and each student group, including:

Test reliability of the State’s assessments estimated for its student population;

Overall and conditional standard error of measurement of the State’s assessments;

Consistency and accuracy of estimates in categorical classification decisions for the cut scores and achievement levels based on the assessment results;

For computer-adaptive tests, evidence that the assessments produce test forms with adequately precise estimates of a student’s achievement.



1. #049 – ACT Technical Manual (pp. 51-64; 51-62; 61; 51-61)

2. #008 – ACT Classification Consistency and Accuracy for WI Students

3. #055 - ACT Writing Test Technical Report (pp. 1-2)


For WI’s DLM Submission: Evidence about Reliability of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Evidence on reliability is documented in the ACT Technical Manual (#049) and the ACT Writing Technical Report (#055) for the overall precision of test scores (e.g., reliability and conditional standard error of measures) based on the national data of college-bound student performance.

2. Additional empirical evidence should be

provided about reliability, SEM, and the accuracy and consistency of classifications of ACT scores for the accountability reporting subgroups using data from WI students.

3. Additional empirical evidence should be provided about reliability, SEM, and the accuracy and consistency of classifications of the composite ELA scores for the overall population and the accountability reporting subgroups based on WI student performance.




future reference)



Additional evidence about the reliability, SEM, and classification accuracy and consistency by accountability reporting subgroup based on WI student data.

Evidence on reliability for the composite ELA scores, rather than its components, both for the overall WI population and for subgroups.




future reference)


4.2 – Fairness and Accessibility The State has taken reasonable and appropriate steps to ensure that its assessments are accessible to all students and fair across student groups in the design, development and analysis of its assessments.



1. #049 – ACT Technical Manual (p.3) 2. #016 – ACT Item Writer’s Guide for

English MC (p.6) 3. #035 – ACT Item Writer’s Guide for

Reading MC (p. 7) 4. #019 – ACT Essay Writer’s Guide (pp. 6-8) 5. #030 – ACT Guide to Reading Test Passage

Selection (p. 6) 6. #028 – Item writer’s Guide for the ACT

Mathematics Test ( pp. 6-7) 7. #012 – Differential item Functioning

Analysis


For WI’s DLM Submission: Evidence about Fairness and Accessibility of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. WI provides evidence showing that appropriate steps have been taken to ensure the fairness and accessibility in testing as well as in the process of item/test development.

2. WI provides empirical evidence on DIF

analysis from operation to identify potential item bias based on the data from South Carolina students (#012).

3. WI should provide additional empirical evidence in DIF analysis based on WI’s student performance.


Results from, or a plan for conducting, DIF analysis based WI student data.




future reference)


4.3 – Full Performance Continuum The State has ensured that each assessment provides an adequately precise estimate of student performance across the full performance continuum, including for high- and low-achieving students.



1. #049 – ACT Technical Manual (pp. 14; 54-55)


For WI’s DLM Submission: Evidence about Full Performance Continuum of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Due to the purpose and intended use of ACT score for high-stakes school accountability, WI should consider a plan to improve the precision of measurement for all students, particularly for low-achieving students.

Section 4.3 Summary Statement For WI’s ACT Submission: _X__ No additional evidence is required




future reference)


4.4 – Scoring The State has established and documented standardized scoring procedures and protocols for its assessments that are designed to produce reliable results, facilitate valid score interpretations, and report assessment results in terms of the State’s academic achievement standards.



1. #049 – ACT Technical Manual (pp.38-63; 38-50; 51-62; 16; 45)

2. #055 – ACT Writing Technical Report (pp. 12-13)


For WI’s DLM Submission: Evidence about Scoring of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. WI provides general information and the procedure for scoring student essays on the ACT writing assessment.

2. Additional information should be provided for scoring WI students’ responses to essay, such as raters’ recruiting, selection criterion, and training, and rang finding, validation papers, and rater consistency.


Information on the hand scoring of direct writing, such as rater recruiting criteria, training, rang finding, validity papers, and procedures dealing with discrepant ratings.




future reference)


4.5 – Multiple Assessment Forms If the State administers multiple forms within a content area and grade level, within or across school years, the State ensures that all forms adequately represent the State’s academic content standards and yield consistent score interpretations such that the forms are comparable within and across school years.



1. #022 – Form Construction Guide (pp. 2.15-2.19; Appendix A; 2.2-2.5; 2.6-2.7; 2.7-2.12; Appendix B, section 5-7)

2. #049 – ACT Technical Manual (pp. 50-51) 3. #018 – Stability Checks in Random Group

Equating


For WI’s DLM Submission: Evidence about Multiple Assessment Forms of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. WI provides evidence about technically sound process for the construction of test forms to ensure the comparability of test construct across multiple forms.

2. WI provides evidence about the technical sound procedures for equating, such as sampling, equating method, and exploring potential technical issues to ensure the comparability of test scores across multiple forms. .

Section 4.5 Summary Statement For WI’s ACT Submission: _X__ No additional evidence is required or,




future reference)


4.6 – Multiple Versions of an Assessment If the State administers assessments in multiple versions within a content area, grade level, or school year, the State:

Followed a design and development process to support comparable interpretations of results for students tested across the versions of the assessments;

Documented adequate evidence of comparability of the meaning and interpretations of the assessment results.


tests and grades documented on cover sheet WI administers the P/P version of ACT


For WI’s DLM Submission: Evidence about Multiple Versions of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission: No multiple versions are used.

Section 4.6 Summary Statement For WI’s ACT Submission: _X__ No additional evidence is required




future reference)


4.7 – Technical Analysis and Ongoing Maintenance The State has a system for monitoring and maintaining, and improving as needed, the quality of its assessment system, including clear and technically sound criteria for the analyses of all of the assessments in its assessment system (i.e., general assessments and alternate assessments).



1. #048 – ACT Technical Advisory Committee

2. #011 – ACT National Curriculum Survey (pp. 1-2)

3. #049 – ACT Technical Manual (p. 3, 5, 64)


For WI’s DLM Submission: Evidence about Technical Analysis and Ongoing Maintenance of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. WI provides some information about the ACT Technical Advisory Committee (TAC). Additional information is needed about the WI’s TAC, such as the committee members, sample meeting agenda, and recommendations particularly on the technical issues of the implementation of ACT in Wisconsin.

2. WI may consider developing a plan for the ongoing analyses of WI student data to support the use of ACT for high-stakes school accountability.


Information on WI’s TAC, including the TAC membership and future meetings. In addition, the TAC’s recommendations on future analyses and maintenance, as well as the general implementation of the ACT in WI, should also be provided.

Section 4.7 Summary Statement- DLM EOY

__X_ No additional evidence is required



SECTION 5: INCLUSION OF ALL STUDENTS


future reference)


5.1 – Procedures for Including Students with Disabilities The State has in place procedures to ensure the inclusion of all public elementary and secondary school students with disabilities in the State’s assessment system, including, at a minimum, guidance for individual educational plan (IEP) Teams to inform decisions about student assessments that:

Provides clear explanations of the differences between assessments based on grade-level academic achievement standards and assessments based on alternate academic achievement standards, including any effects of State and local policies on a student’s education resulting from taking an alternate assessment based on alternate academic achievement standards;

States that decisions about how to assess students with disabilities must be made by a student’s IEP Team based on each student’s individual needs;

Provides guidelines for determining whether to assess a student on the general assessment without accommodation(s), the general assessment with accommodation(s), or an alternate assessment;

Provides information on accessibility tools and features available to students in general and assessment accommodations available for students with disabilities;

Provides guidance regarding selection of appropriate accommodations for students with disabilities;

Includes instructions that students eligible to be

Evaluate for all factors in left hand column —Addresses general assessments w or w/o accommodations and AA-AAAS For both ACT and DLM:

1. #112 - Form I-7-A Sample IEP Form and Participation Guidelines for Alternate Assessments

2. #082 – Form I-7 The ACT Plus Writing and ACT WorkKeys

3. #113 – Form I-7 DLM sample IEP form for DLM

For ACT:

1. #093 – WI ACT with Writing Accommodations and Support Matrix

2. #069 – WI ACT Plus Writing Accommodations Decision Tree

3. #001 – Test Accessibility and Accommodations User Guide

4. #021 – Frequent Asked Questions: ACT State and District Testing (p. 14)

5. #086 – Parent Letter for WI Non-college reportable/Ineligible Accommodations Notification

For Alternate Assessment:

1. #056 – Accessibility Manual for the DLM Alternate Assessment 2014-2015


1. WI provides evidence about the Procedures for Including Student with Disabilities. However, documentation is needed to inform parents about the alternate assessments and their potential consequences should their child or children take an alternate assessment.


1. WI provides evidence about the procedures for including student with disabilities on ACT, such as Accommodation Decision Tree (#069) and the guidelines for accommodations (#001).

2. WI also provides a notification letter to

parents in regard to Non-college reportable/Ineligible Accommodations on ACT.

3. Clarifications are needed on the ACT accessibility tools and features available to students in general and assessment accommodations available for students with disabilities.




future reference)


assessed based on alternate academic achievement standards may be from any of the disability categories listed in the IDEA;

Ensures that parents of students with the most significant cognitive disabilities are informed that their student’s achievement will be based on alternate academic achievement standards and of any possible consequences of taking the alternate assessments resulting from district or State policy (e.g., ineligibility for a regular high school diploma if the student does not demonstrate proficiency in the content area on the State’s general assessments);

The State has procedures in place to ensure that its implementation of alternate academic achievement standards for students with the most significant cognitive disabilities promotes student access to the general curriculum.

Section 5.1 Summary Statement For WI’s DLM Submission: _X__ The following additional evidence is needed/provide brief rationale

Documentation of WI guidance to parents whose child or children may take an alternate assessment. This guidance should clearly state any and all potential consequences of taking an alternate assessment.

For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale:

Based on the evidence provided (e.g., #001 – Test Accessibility and Accommodations User Guide) the accessibility tools appear to be either limited or non-existent. The documentation should be revised to clarify, in detail, what accessibility tools are available to students taking the ACT.




future reference)


5.2 – Procedures for including ELs The State has in place procedures to ensure the inclusion of all English learners in public elementary and secondary schools in the State’s assessment system and clearly communicates this information to districts, schools, teachers, and parents, including, at a minimum:

Procedures for determining whether an English learner should be assessed with accommodation(s);

Information on accessibility tools and features available to all students and assessment accommodations available for English learners;

Guidance regarding selection of appropriate accommodations for English learners.

Evaluate for all factors in left hand column —Addresses general assessments w or w/o accommodations and AA-AAAS For both ACT and DLM:

1. #066 – School Accountability for 2015-2016

For ACT:

1. #049 – ACT Technical Manual (p.14) 2. #054 – Accommodations on the ACT Test

(p. 1) 3. #001 – Test Accessibility and

Accommodations User Guide 4. #081 – The ACT High School Assessments

FAQ (pp. 10-11) 5. #068 – The CAT High School Assessments

Accommodations and Supports 6. #069 – The ACT Plus Writing

Accommodations Decision Tree 7. #093 – WI ACT with Writing

Accommodations and Support matrix for ELs

For DLM:

1. #056 – Accessibility Manual for the DLM 2014-2015 (pp. 16-17; 30, 40)

2. #109 – DLM FQA (pp. 4-5) for ELs


1. WI provides sufficient evidence about the procedures for including EL students.


1. WI provides documents about the procedures for including EL students, such as an Accommodation Decision Tree (#069), the guidelines for accommodations (#001), and the accommodations and support matrix for ELs.

2. Additional evidence is needed in regard to the accessibility tools and features available to all students and assessment accommodations available for English learners on ACT.

Section 5.2 Summary Statement For WI’s DLM Submission: _X__ No additional evidence is required. For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

As noted in Critical Element 5.1, there is insufficient evidence provided regarding the accessibility tools and features available to all students taking ACT.




future reference)


5.3 – Accommodations The State makes available appropriate accommodations and ensures that its assessments are accessible to students with disabilities and English learners. Specifically, the State:

Ensures that appropriate accommodations are available for students with disabilities(SWD) under the Individuals with Disabilities Education Act (IDEA) and students covered by Section 504;

Ensures that appropriate accommodations are available for English learners (EL);

Has determined that the accommodations it provides (i) are appropriate and effective for meeting the individual student’s need(s) to participate in the assessments, (ii) do not alter the construct being assessed, and (iii) allow meaningful interpretations of results and comparison of scores for students who need and receive accommodations and students who do not need and do not receive accommodations;

Has a process to individually review and allow exceptional requests for a small number of students who require accommodations beyond those routinely allowed.

Evaluate for all factors in left hand column —Addresses general assessments w or w/o accommodations and AA-AAAS; For ACT:

1. #054 – Accommodations and the ACT Test (pp. 1-2; 5)

2. #093 – WI ACT with Writing Accommodations and Support Matrix

3. #015 – Policy for documentation 4. #001 – Test Accessibility and

Accommodations User Guide 5. #021 – FAQ (p. 11; 15)


For WI’s DLM Submission: Evidence about Accommodations of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Although WI provides policy, procedures, and accommodations for ACT, additional evidence is needed about the availability and the appropriate use of accommodations for students with special needs and the process for allowing accommodations beyond those routinely allowed for individual students.

2. WI did not provide evidence that provided

accommodations are appropriate and effective, do not alter the construct, and allow for meaningful interpretations and comparisons. A detailed plan is expected to ensure that appropriate and effective accommodations are available for students with disabilities under the IDEA, students covered by Section 504, and for English learners to support meaningful interpretations of test scores.

Section 5.3 Summary Statement For WI’s DLM Submission: _X__ No additional evidence is required For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

Evidence of a process, or intent to implement, to review and allow requests for accommodations beyond those routinely allowed.

A plan detailing examination(s) designed to ensure that accommodations are appropriate and effective, do not alter the construct, and allow for meaningful interpretations and comparisons. Such examinations often look for differential boost or measurement invariance.




future reference)


5.4 – Monitoring Test Administration for Special Populations The State monitors test administration in its districts and schools to ensure that appropriate assessments, with or without appropriate accommodations, are selected for students with disabilities under IDEA, students covered by Section 504, and English learners so that they are appropriately included in assessments and receive accommodations that are:

Consistent with the State’s policies for accommodations;

Appropriate for addressing a student’s disability or language needs for each assessment administered;

Consistent with accommodations provided to the students during instruction and/or practice;

Consistent with the assessment accommodations identified by a student’s IEP Team or 504 team for students with disabilities, or another process for an English learner;

Administered with fidelity to test administration procedures.

Evaluate for all factors in left hand column —Addresses general assessments w or w/o accommodations and AA-AAAS; For ACT:

1. #015 – Policy for Documentation 2. #002 – Welcome to the Accommodations

Q & A Session for 2016 Testing 3. #050 – ACT Test Administration Manual

(pp. 3; 97-98; 105; 107) For Alternate Assessment:

1. #057 – DLM Accessibility Profile Report Layout


1. Although WI provides documents about test administrations, additional evidence is needed regarding the monitoring of DLM administration for special populations, such as policy, process, as well as implementation and a summary report.


1. Although WI provides documents about the available accommodated test administration, as well as the summary report by ACT, additional evidence is needed regarding monitoring test administration of ACT for special populations, such as policy, process, implementation, and a summary report.

Section 5.4 Summary Statement For WI’s DLM Submission: __X_ The following additional evidence is needed/provide brief rationale

Documentation on the procedure or process WI has for monitoring test administrations, and acting on, the results of that monitoring. For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

Documentation on the procedure or process WI has for monitoring test administrations, and acting on, the summary reports ACT’s generates on administration.



SECTION 6: ACADEMIC ACHIEVEMENT STANDARDS AND REPORTING


future reference)


6.1 – State Adoption of Academic Achievement Standards for All Students The State formally adopted challenging academic achievement standards in reading/language arts, mathematics and in science for all students, specifically:

The State formally adopted academic achievement standards in the required tested grades and, at its option, also alternate academic achievement standards for students with the most significant cognitive disabilities;

The State applies its grade-level academic achievement standards to all public elementary and secondary school students enrolled in the grade to which they apply, with the exception of students with the most significant cognitive disabilities to whom alternate academic achievement standards may apply;

The State’s academic achievement standards and, as applicable, alternate academic achievement standards, include: (a) At least three levels of achievement, with two for high achievement and a third of lower achievement; (b) descriptions of the competencies associated with each achievement level; and (c) achievement scores that differentiate among the achievement levels.


tests and grades documented on cover sheet For both ACT and DLM:

1. #129 – WI Statewide Reporting For ACT:

1. #077 – Cabinet Decision Paper: AT Cut Scores

2. #087 – Summary: ACT Performance Level Cut Scores for WI (pp. 3-6)

3. #071 – WI ACT Achievement Level Descriptors

For Alternate Assessment:

1. #059 – DLM Cut Scores Approval 2. #138 – About the Data: DLM performance

levels 3. #137 – DLM 2015 Year-End Model

Standard Setting Report


1. WI provides evidence to support the state’s adoption of the DLM academic achievement standards for all students (#059, $137).


1. WI provides evidence to support the state’s adoption of the ACT academic achievement standards for all students (#077, #087).

Section 6.1 Summary Statement For WI’s DLM Submission: _X__ No additional evidence is required For WI’s ACT Submission: _X__ No additional evidence is required




future reference)


6.2 – Achievement Standards-Setting The State used a technically sound method and process that involved panelists with appropriate experience and expertise for setting its academic achievement standards and alternate academic achievement standards to ensure they are valid and reliable.



1. #087 – Summary: ACT Performance Level Cut Scores for WI

2. #008 – ACT Classifications Consistency and Accuracy for WI Students

3. #091 – Who should be involved in Recommending ACT Cut Scores?


For WI’s DLM Submission: Evidence about Achievement Standard Setting of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. WI provides insufficient evidence about the review and approval process for determining the academic achievement standards (#087).

2. A standard setting technical report or a similar document is expected to describe the methodology, training, and the process for setting/approval cut scores. These academic achievement standards should clearly reflect the expectations from students about what they know and be able to do at each achievement level based on the WI SSCC.

Section 6.2 Summary Statement For WI’s DLM Submission: _X__ No additional evidence is required For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

Evidence on the process by which the WI standard setting panelists reviewed, revised and approved the academic achievement standards.

Generally, such practice is captured in a standard setting technical report, which should provide fine detail on the ways in which WI standard-setting panelists created the academic content standards. If a standard setting technical report does not exist, WI should present evidence of a plan for the creation of such documentation.

Evidence that clarifies how the combined ELA cut scores were defined (i.e., the process for combined ELA score, instead of the process for English, Reading and Writing separately).




future reference)


6.3 – Challenging and Aligned Academic Achievement Standards The State’s academic achievement standards are challenging and aligned with the State’s academic content standards such that a high school student who scores at the proficient or above level has mastered what students are expected to know and be able to do by the time they graduate from high school in order to succeed in college and the workforce.

If the State has defined alternate academic achievement standards for students with the most significant cognitive disabilities, the alternate academic achievement standards are linked to the State’s grade-level academic content standards or extended academic content standards, show linkage to different content across grades, and reflect professional judgment of the highest achievement standards possible for students with the most significant cognitive disabilities.



1. #087 – Summary: ACT Performance Level Cut Scores for WI

2. #075 – Establishing Performance Level Standards for the ACT in WI


For WI’s DLM Submission: Evidence about Challenging and Aligned Achievement Standard Setting of the Dynamic Learning Maps reading/language arts and mathematics grades 3-8, 11 was submitted on WI’s behalf by the state of WV. For WI’s ACT Submission:

1. Insufficient evidence is provided about the process for the development and review of the ACT Performance Level Descriptors (PLDs) (#075, #087).

2. Additional evidence is needed to show that

the Performance Level Descriptors clearly define the expectations from students based on the WI CCSS at each achievement level.

3. The achievement standards should be aligned with the WI CCSS that a high school student who scores at the proficient or above level on ACT has mastered what students are expected to know and be able to do by the time they graduate from high school in order to succeed in college and the workforce.

Section 6.3 Summary Statement For WI’s DLM Submission: _X__ No additional evidence is required or, For WI’s ACT Submission: _X__ The following additional evidence is needed/provide brief rationale

As noted under Critical Element 3.1 and elaborated here, the state should provide evidence demonstrating that the PLDs reference specific content within the Common Core State Standards, so that each level of the PLDs clearly states what students should know and be able to do.




future reference)


6.4 – Reporting

The State reports its assessment results, and the reporting facilitates timely, appropriate, credible, and defensible interpretations and uses of results for students tested by parents, educators, State officials, policymakers and other stakeholders, and the public, including:

The State reports to the public its assessment results on student achievement at each proficiency level and the percentage of students not tested for all students and each student group after each test administration;

The State reports assessment results, including itemized score analyses, to districts and schools so that parents, teachers, principals, and administrators can interpret the results and address the specific academic needs of students, and the State also provides interpretive guides to support appropriate uses of the assessment results;

The State provides for the production and delivery of individual student interpretive, descriptive, and diagnostic reports after each administration of its assessments that: o Provide valid and reliable information

regarding a student’s achievement; o Report the student’s achievement in terms

of the State’s grade-level academic achievement standards (including performance-level descriptors);

o Provide information to help parents, teachers, and principals interpret the test results and address the specific academic needs of students;

o Are available in alternate formats (e.g., Braille or large print) upon request and, to


tests and grades documented on cover sheet For both ACT and DLM:

1. #135 – WISEdash for District Homepage – Interpretive guidance

For ACT: 1. #042 – The ACT Plus Writing Student

Report 2. #038 – The ACT High School Report 3. #040 – The ACT High School Report

Checklist 4. #039 – The ACT Profile 5. #041 – The ACT Electronic Student

Record State Testing 6. #052 –Using Your ACT Results 7. #036 – ACT Score Report Descriptions

For DLM:

1. #141 – Understanding Your child’s DLM Score Report


1. WI provides sufficient information about Reporting of test scores on the DLM and with guidelines for interpretations.

For WI’s ACT Submission: 1. Additional evidence is needed about reporting of ACT results and provide valid and reliable information regarding a student’s achievement level with appropriate interpretations. 2. A document should be developed that multiple assessment reports for various audience, in which timeline is provided to specific audience. 3. An interpretive guide of test scores should be provided. 4. Reports of test results should be customized to facilitate the use by teachers/educators, administrators, parents, and to the general public with different formats.




future reference)


the extent practicable, in a native language that parents can understand;

The State follows a process and timeline for delivering individual student reports to parents, teachers, and principals as soon as practicable after each test administration.

Section 6.4 Summary Statement For WI’s DLM Submission: _X__ No additional evidence is required for WI’s DLM submission. For WI’s ACT Submission: __X_ The following additional evidence is needed/provide brief rationale:

Documentation showing all reports and supporting material (a) reflect WI’s reporting categories (i.e., a score for ELA), and (b) provide information on WI academic achievement levels.

Information showing that the score reports are available in alternative formats.

A process and timeline for delivering reports to students, parents, teachers, principals, and other stakeholders as soon as practicable after each test administration.

An interpretive guide.

Materials that support parents, e.g., a letter to parents explaining the ACT and their student’s scores.

Reports tailored for educators (e.g., classroom level reports) or a plan to develop reports tailored for educators.

STATE ASSESSMENT PEER REVIEW NOTES FOR DYNAMIC LEARNING MAPS-YEAR END MODEL

1

U. S. Department of Education

Peer Review of State Assessment Systems

June 2016 State Assessment Peer Review

Notes DLM End of Year Consortium Evidence

U. S. Department of Education Office of Elementary and Secondary Education

Washington, D.C. 20202

Note: Peer review notes provide the combined recommendations of the individual peers to the U.S. Department of Education (Department), based on the statute and regulations, the Department’s peer review guidance, and the peers’ professional judgement of the evidence submitted by the State. These assessment peer review notes, however, do not necessarily reflect the final set of additional evidence, if any, that a State may need to submit to demonstrate that its assessment system meets all of the critical elements for assessment peer review. Although the peer notes inform the Secretary’s consideration of each State’s assessment system, the Department makes the final decision regarding whether the assessment system meets the requirements in the statute and regulations. As a result, these peer notes may not completely align with the final determination made by the Department.



Contents

SECTION 2: ASSESSMENT SYSTEM OPERATIONS ....................................... 3

2.1 – Test Design and Development .............................................................. 3 2.2 – Item Development .................................................................................. 6 2.3 – Test Administration ............................................................................... 8 2.4 – Monitoring Test Administration (reviewed by Department staff only)11 2.5 – Test Security ........................................................................................ 12

2.6 – Systems for Protecting Data Integrity and Privacy ........................... 13

SECTION 3: TECHNICAL QUALITY – VALIDITY............................................. 14 3.1 – Overall Validity, including Validity Based on Content ...................... 14 3.2 – Validity Based on Cognitive Processes ............................................. 16

3.3 – Validity Based on Internal Structure .................................................. 17 3.4 – Validity Based on Relationships with Other Variables ..................... 18

SECTION 4: TECHNICAL QUALITY - OTHER ................................................. 19 4.1 – Reliability .............................................................................................. 19

4.2 – Fairness and Accessibility .................................................................. 21 4.3 – Full Performance Continuum .............................................................. 23 4.4 – Scoring ................................................................................................. 24

4.5 – Multiple Assessment Forms ............................................................... 25 4.6 – Multiple Versions of an Assessment ................................................. 26

4.7 – Technical Analysis and Ongoing Maintenance ................................. 27

SECTION 5: INCLUSION OF ALL STUDENTS ................................................ 28

5.1 – Procedures for Including Students with Disabilities ........................ 28 5.2 – Procedures for including ELs ............................................................. 31 5.3 – Accommodations ................................................................................. 33

5.4 – Monitoring Test Administration for Special Populations ................. 35

SECTION 6: ACADEMIC ACHIEVEMENT STANDARDS AND REPORTING . 37

6.1 – State Adoption of Academic Achievement Standards for All Students........................................................................................................................ 37 6.2 – Achievement Standards-Setting ......................................................... 39

6.3 – Challenging and Aligned Academic Achievement Standards ......... 40 6.4 – Reporting .............................................................................................. 41



SECTION 2: ASSESSMENT SYSTEM OPERATIONS


future reference)


2.1 – Test Design and Development The State’s test design and test development process is well-suited for the content, is technically sound, aligns the assessments to the full range of the State’s academic content standards, and includes:

Statement(s) of the purposes of the assessments and the intended interpretations and uses of results;

Purposes and Intended Interpretations and Uses of Results File 06: page 1, page 5

PLEASE NOTE: The peer reviewers wish to acknowledge the magnitude and significance of this endeavor to create and implement a assessment for students with significant cognitive disabilities based on an articulated learning map aligned to content standards for the general student population. While the feedback that follows contains questions and a few requests for additional evidence, as well as some suggestions for consideration in the future, peers were cognizant of the enormous amount of work and time that went into the DLM assessment. The technical manual provides a clear statement of the purpose and intended interpretations and uses of the results of the DLM assessment.

Test blueprints that describe the structure of each assessment in sufficient detail to support the development of assessments that are technically sound, measure the full range of the State’s grade-level academic content standards, and support the intended interpretations and uses of the results;

Test Structure and Blueprints File 06: pp. 41-46 File 08 File 09 File 10 File 11

Test blueprints clearly present the structure of the DLM testlets and the assessment as a whole. The DLM year-end-model assesses EEs in five conceptual areas across two claims. Coverage is summarized on page 2 of File 10. The DLM year-end-model assesses EES in mathematics across all four major claims (File 11 p. 1). Peer reviewers were unable to gain a sufficiently clear understanding of how EEs were “prioritized” for inclusion in the assessment (which includes a subset of EEs rather than all per grade level). Although various documents (e.g., Sample Student




future reference)


Report in File 07, Appendix E.1) illustrate that not all EEs eligible for assessment are addressed by every student, it would have been extremely helpful to peer reviewers for DLM to have provided more detail, and in the appropriate sections of the submission, on what constitutes a typical assessment experience at the student level. This might take the form of a summary that describes the minimum-maximum number of testlets, the minimum-maximum number of items per testlet, and the minimum-maximum coverage across EEs in both ELA and Math. To frame differently, it would have helped to see how the distribution of EEs presented in Files 10 and 11 is operationalized for individual students. The picture of the DLM assessment at the global level is far clearer and richer than the picture of the assessment at the student level. Peers could not find evidence that the EEs address Speaking and Listening, which are among the domains in the CCSS. While the ELA blueprints include a few EEs that correspond to certain CCSS Language standards, these are identified in the blueprint under one or another of the five Conceptual Areas (CAs) covered by the DLM assessment (sometimes C.1.2. and other times C.2.1.) The peers could not find evidence that the CCSS domain of Language is explicitly addressed.

Processes to ensure that each assessment is tailored to the knowledge and skills included in the State’s academic content standards, reflects appropriate inclusion of challenging content, and requires complex demonstrations or applications of knowledge and skills (i.e., higher-order thinking skills);

Representation of Knowledge and Skills in the Assessment and the Standards

File 06: pp. 5-8, 24-28, 38-41, 46, 61-64

Evidence conveys the degree and nature of coverage of the EEs (learning targets for students with significant cognitive disabilities [SWSCD]) and their correspondence to CCSS.




future reference)


If the State administers computer-adaptive

assessments, the item pool and item selection procedures adequately support the test design.

Item Pool and Selection Procedures File 06: pp. 60-61, 83-84, 101-106, 112-114

The adaptive delivery of testlets via KITE is well designed as described in the Technical Manual (112-14). While computer-adaptive procedures described made clear how linkage level would be adjusted based on performance, the peers were unable to find evidence to explain how this might impact EE coverage at the student level.

Section 2.1 Summary Statement _X__ The following additional evidence is needed/provide brief rationale:

DLM should explain why some CCSS ELA domains are not directly addressed (Language) or are not addressed at all (Speaking/Listening)



Critical Element Evidence (Record document and page # for future reference) Comments/Notes/Questions/Suggestions Regarding State Documentation or Evidence

2.2 – Item Development The State uses reasonable and technically sound procedures to develop and select items to assess student achievement based on the State’s academic content standards in terms of content and cognitive process, including higher-order thinking skills.

Item Development File 06: Introduction, pp. 20-21, 46,47-48, 37-38, 69-70, 69, 65-68, 68-69, 61-64, 60-61, 85-87, 46-47, 75-76, 76-77, 82-83, 89-93,126-130, 219-233, 210-217 File 18: pp. 9-10, 11-15

Evidence was sufficient for this section. Training materials for item writers are described and in some instances, provided in their entirety. One concern, however, is that peers could not find much evidence, beyond simple criteria for writing keys and distractors, that item development has attended fully and well to matters related to determining a correct/complete response (e.g., specifically how to screen for flaws in item options (ambiguous options, multiple options where not intended, etc.). Training might be enhanced with examples of well-written and poorly written items. If more training resources are available (e.g., the section of File 18 on bias and sensitivity review), then these should be cited, as they are likely to only strengthen the submission. The assumption must be made that all items are treated as dichotomous items, although this is never stated/made explicit. Why and how DLM decided against awarding partial credit for multiple select items and others with multiple correct response options should be included in evidence for this Critical Element. The testlet design (with % items correct to indicate “mastery”) may not permit this, but perhaps that should/could be explained. More detail on the duration of training of item writers—as well as a typical training agenda—would be useful as evidence of sound procedures to develop and select items. Some background/rationale for item types and



Critical Element Evidence (Record document and page # for future reference) Comments/Notes/Questions/Suggestions Regarding State Documentation or Evidence

features (e.g., use of three response options in MC item, use of color photograph as text feature in ELA testlets) would be helpful—In other words, documentation of the research/best practice behind key item features/formats. Peers suggest that additional cognitive labs to investigate possible option order effect on student response be considered.

Item Selection File 06: pp. 77- 82, p. 93, 46- 47, 75-83, 93-101, 97- 98, 101-106, 98

The Technical Manual (p. 233) indicates the intention to follow up on DIF analysis by expanding in future years; DLM should be encouraged to submit this supplementary analysis when available. Overall, the evidence was sufficient for this section. However, while observations are described as part of validity studies, peer reviewers did not see any evidence of the use of observation during field-testing to inform item development, review and revision. This additional source of information might be helpful and should be considered for future rounds of item development.

Section 2.2 Summary Statement __X_ No additional evidence is required




future reference)


2.3 – Test Administration The State implements policies and procedures for standardized test administration, specifically the State:

Has established and communicates to educators clear, thorough and consistent standardized procedures for the administration of its assessments, including administration with accommodations;

Standardized Procedures File 05: pages 9-12, 89-116, 78-150, 149-159, 22-38, 38-50, 49-50 File 02: pp. 7-13 File 03: p. 8 File 04: pp. 6-7

The evidence provided collectively identifies all of the steps necessary (and the resources to guide key individuals involved) to conduct standardized administrations of the assessment. The Test Administration manual is made easier to follow with supporting visuals (screen shots from Educator Portal and KITE) and numerous “hints” in sidebars to address specific needs/issues. DLM provides states with live updates through “state landing page” and updates on website. Detail is provided on the range of testing devices that may be used. DLM provides some information on handling such matters as extended inactivity when KITE is open, and exiting and returning. One incident summary (involving incorrect testlet information pages) is provided among evidence to illustrate/document contingency plans. State landing pages and updates on the DLM website appear to serve as a mechanism—if needed—to deal with unexpected technology challenges during test administration.

Communication File 17: a) Sample state landing page from DLM website; b) Test updates – website and email example

Administration with Accommodations File 01: pp. 19-22, 15-18 File 05: pp. 32-37




future reference)


Has established procedures to ensure that all individuals responsible for administering the State’s general and alternate assessments receive training on the State’s established procedures for the administration of its assessments.

Training File 06: pp.251-254, 250-251, 254, 248-249, 124-125 File 07: Appendices G.1, C.14 File 16: p. 5 File 02: pp. 7-13 File 03: p. 8 File 04: pp. 6-7

Training resources are provided as evidence for all key personal: Test Administrators, Data Stewards, and Technical Liaisons. Required training for test administrators consists of eight modules on such topics as accessibility, preparing for the test, computer delivered testlets and teacher delivered testlets. Detail on required performance (80%) on post-test quizzes is provided. Peer reviewers could not find any information on “next steps” or consequences if trainee failed to reach that performance on one or more of the post-test quizzes. More detail would be helpful on how training can ensure that ALL teachers of SWSCD will be able to administer DLM to their students.

If the State administers technology-based assessments, the State has defined technology and other related requirements, included technology-based test administration in its standardized procedures for test administration, and established contingency plans to address possible technology challenges during test administration.

Technology Requirements File 06: pp.110, 251-254 File 07: Appendix G.1 File 04: pp. 9-13, 8 File 05: pp. 149-150, 62-65 File 02: p. 55

Contingency plans for technology-based assessment administration File 06: p. 111, 123-125, 133-134,193-195 File 07: Appendix C.7

Good systems in place for addressing localized administration issues (Technical Manual, p. 111) and internet connectivity issues (see Technical Liaison Manual p. 12). Peer reviewers would like to have seen more




future reference)


File 04: pp. 6-7, 12 File 17:

information on contingency plans based on potential disruptions of service/functioning of technology.

Section 2.3 Summary Statement __X_ No additional evidence is required from the DLM consortium






future reference)


2.4 – Monitoring Test Administration The State adequately monitors the administration of its State assessments to ensure that standardized test administration procedures are implemented with fidelity across districts and schools.

policies and procedures apply to all

grade-level and AA-AAAS in all subjects

The DLM consortium submitted evidence of procedures to monitor test administration (File 06 p. 123-125 and p. 225-231) and made a variety of materials and resources available to partner states for the purpose of monitoring test administration. These included a test administration monitoring protocol and an observation protocol for use by SEAs and LEAs. Among other things, the monitoring protocols captured accessibility supports used, level of engagement and barriers to engagement (File07 Appendix C 13). In addition, training tools were provided on the use of these protocols. Monitoring of test administrations was also possible at the SEA and LEA levels through the DLM Educator Portal which permitted checking on progress toward test completion at the student level. Again, training on this feature was submitted (File 07 appendix C 14). Focused monitoring of the test administration was conducted both by DLM and at the SEA and LEA levels, indicating fidelity of test administration (File 06 p.225-231). Errors in routing of students to testlets was also monitored and procedures provided to test administrators to rectify errors (File 06 pp. 142-143 and pp. 193-195) . Finally, states were provided with summaries of these errors as addenda to score reports (File 06 pp. 193-195)

Section 2.4 Summary Statement—REVIEWED BY DEPARTMENT STAFF ONLY __x_ No additional evidence is required from the DLM consortium-State specific.




future reference)


2.5 – Test Security The State has implemented and documented an appropriate set of policies and procedures to prevent test irregularities and ensure the integrity of test results through:

Prevention of any assessment irregularities, including maintaining the security of test materials, proper test preparation guidelines and administration procedures, incident-reporting procedures, consequences for confirmed violations of test security, and requirements for annual training at the district and school levels for all individuals involved in test administration;

Evidence of Prevention of Irregularities File 06: pp. 46-47, 68-69, 69-70, 75-76, 77, 132-133, 134, 133, 135, 252 File 07: Appendices B.3 and C.3 File 05: p. 121-12 File 02: p. 36

Evidence of detection, remediation, and investigation of test irregularities focused on data breaches. Page 252 of File 06 provides a broad statement of expectations regarding security in context of training.

Detection of test irregularities;

Evidence of Detection of Irregularities File 06: pp.135-136 File 07: Appendix C.15

Evidence from DLM is sufficient in this section; more evidence is expected from States.

Investigation of alleged or factual test irregularities

Evidence of Investigation of Remediation Following Incidents File 06: p. 133, 134-135 File 07: Appendices C.5, C.6

Evidence from DLM is sufficient in this section; more evidence is expected from States.

Remediation following any test security incidents involving any of the State’s assessments;

THIS PORTION OF THE CRITICAL ELEMENT IS

ADDRESSED IN INDIVIDUAL STATE SUBMISSIONS. N/A

Section 2.5 Summary Statement _X__ No additional evidence is required from the DLM consortium.




future reference)


2.6 – Systems for Protecting Data Integrity and Privacy The State has policies and procedures in place to protect the integrity and confidentiality of its test materials, test-related data, and personally identifiable information, specifically:

To protect the integrity of its test materials and related data in test development, administration, and storage and use of results;



Security and Integrity of Test Materials File 06: pp. 133-134, 134- 135

Detailed evidence documenting policies and procedures to protect integrity and confidentiality of data is provided. Evidence includes definition of minimum number of students necessary to allow reporting of scores for all students/student groups. There is evidence of an appropriately hierarchical system of access to data based on scope of responsibility.

To secure student-level assessment data and protect student privacy and confidentiality, including guidelines for districts and schools;

File 06: 133-134, 135 File 07: Appendices C.4, C.5, C.6

Evidence is sufficient for this section.

To protect personally identifiable information about any individual student in reporting, including defining the minimum number of students necessary to allow reporting of scores for all students and student groups.

PII Protection in Reporting File 06: pp.134-135, 186-188, 191 File 07: Appendices C.4, C.5, C.6

Evidence is sufficient for this section.

Section 2.6 Summary Statement __X_ No additional evidence is required from the DLM consortium.



SECTION 3: TECHNICAL QUALITY – VALIDITY


future reference)


3.1 – Overall Validity, including Validity Based on Content The State has documented adequate overall validity evidence for its assessments, and the State’s validity evidence includes evidence that the State’s assessments measure the knowledge and skills specified in the State’s academic content standards, including:

Documentation of adequate alignment between the State’s assessments and the academic content standards the assessments are designed to measure in terms of content (i.e., knowledge and process), the full range of the State’s academic content standards, balance of content, and cognitive complexity;

Validity Framework and Overall Evaluation File 06: pp. 5-7, 279-280 (Tables 108 and 109), 264-278), p. 281 (Table 110), 283-284, 282-283, 263-264

Overall, the interpretation and use argument was clearly tied to four validity claims (how the scores could be used) and these guided validation efforts. The submission provides evaluative evidence of technical quality through an overview of the review process, criteria used, and results (pp. 75-76; 77-82; 82-83). The External Alignment Study (File 15) provides detail on fidelity to the content in the grade-level standards (see pp. 4-8 for Executive Summary). It is worth noting that DLM acknowledges the need to do further alignment studies (see Technical Manual p. 267) since earlier study was done on limited sample rather than on operational testlets as administered. Over time peer reviewers would like to see more evidence of alignment between instructional content and assessment content. The Technical Manual (282-284) included a detailed account of anticipated areas for continuous improvement and future research. Follow-up on consequential validity evidence is advisable, since admittedly limited based on 2014-15 administration.

If the State administers alternate assessments based on alternate academic achievement standards, the assessments show adequate linkage to the State’s academic content standards in terms of content match (i.e., no unrelated

Measurement of Academic Content Standards File 06: Introduction, pp. 5-7, 10-21, 14-17, 17-19, 26-27, 38-41, 41- 46, 61-64, 46-47, 68-69, 75-82

Evidence demonstrates that the EEs of the DLM (the equivalent of alternate academic content standards) are adequately linked to State academic content standards in terms of content match (i.e., no unrelated content) and the breadth of content and




future reference)


content) and the breadth of content and cognitive complexity determined in test design to be appropriate for students with the most significant cognitive disabilities.

cognitive complexity determined in test design to be appropriate for students with the most significant cognitive disabilities. Testlets are constructed from items that each address an EE aligned to college and career ready standards.

Evaluative Evidence File 06: pp. 75-76, 77-82, 82-83 File 12 File 13 File 15: pp. 8-9, 16 (Table 5), 15-16, Appendix B, 4-8 File 07: Appendix H.1





future reference)


3.2 – Validity Based on Cognitive Processes The State has documented adequate validity evidence that its assessments tap the intended cognitive processes appropriate for each grade level as represented in the State’s academic content standards.

Assessments Tap Intended Cognitive Processes File 06: pp. 61-64, 68-69, 46-47, 69-70, 75-76, p. 230 (Table 95), Conclusion, 270-271 File 18: pp. 9- 19, Appendix A

The use of Essential Element Concept Maps (EECMs) in item and testlet development is intended to ensure that the assessments tap intended cognitive processes as represented in State academic content standards. This is confirmed in external review. For classification purposes, Anderson and Krathwohl’s (2001) revision to Bloom’s taxonomy is used by DLM. The assignment of cognitive levels for 2014-15 items/testlets was reviewed and confirmed by various stakeholders (item writers/reviewers, teachers, advisory panel members).

Interaction with Testlet Content File 06: pp. 219-223, 224-225

Additional cognitive labs to investigate possible option order effect on student response.

Fidelity of Administration File 06: p. 254, 251-254, 147 (Table 55), 117-121,123-124, 225- 229 File 07: Appendix C.12

The submission contains adequate evidence of administration fidelity was provided.

Accessibility File 06: pp. 125-132, 148-150, 119 (Table 43), 121, 150 (Table 57), p. 230 (Table 95), 252-253, 258- 259

The submission contains acceptable evidence of accessibility.





future reference)


3.3 – Validity Based on Internal Structure The State has documented adequate validity evidence that the scoring and reporting structures of its assessments are consistent with the sub-domain structures of the State’s academic content standards on which the intended interpretations and uses of results are based.

Scoring and reporting structures’ consistency with sub-domain structures File 06: Introduction, pp. 14-17, 43-46, 61-64, 85 (Table 21), 58-160, 190-191, 193, 25-34, 35-36, 28-29, 214-215, 263-264 File 07: Appendices E.1, E2 File 15: pp. 22-23

Adequate evidence has been provided. The data files provide for accountability and school improvement purposes indicate overall performance level results for each content area and highest linkage level mastered for each EE (See Chapter VII of Technical Manual).

Consistency of Measurement File 06: pp. 203-204, 205-206, 207-209

Evidence is adequate for the consistency of the scoring and reporting with the sub-domain structures of the consortium content standards.

Differential Item Functioning (DIF) File 06: pp. 231-236, 283-284

Evidence of appropriate procedures is provided; DLM has only considered gender because of sample size. Peers suggest that as more data are available (recommended as >200 per class), further analyses be conducted on other categories (e.g., race, ethnicity, different disability classifications, etc.)





future reference)


3.4 – Validity Based on Relationships with Other Variables The State has documented adequate validity evidence that the State’s assessment scores are related as expected with other variables.

Score Relationship to Other Variables File 06: p. 85 (Table 21), 327 (Table 102)

DLM acknowledges that evidence of the relationship between student responses on the assessment and other measures is limited, given that the first operational administration was in 2014-2015. However, they provide information about test administrators’ judgments regarding difficulty level of testlets (Technical Manual, pp. 236-237). Per the DLM Consortium’s self-analysis of this Critical Element: “To date, evidence on the relationship between student responses on the DLM assessments and other measures is limited to teacher evaluations of student academic knowledge and skills as measured by the First Contact survey, and teacher perception of testlet difficulty.” Recognizing that that the submission reflects only the 2014-15 administration, peer reviewers would like to see included other evidence such as correlations between student performance on DLM and States’ previous alternative assessment or another measure (for consortium members who have such data available).




SECTION 4: TECHNICAL QUALITY - OTHER


future reference)


4.1 – Reliability The State has documented adequate reliability evidence for its assessments for the following measures of reliability for the State’s student population overall and each student group and, if the State’s assessments are implemented in multiple States, for the assessment overall and each student group, including:

Test reliability of the State’s assessments estimated for its student population;

Reliability for Student Population File 06: pp. 161- 162, 190- 193, 203-204, 205-206, 207-209)

General comment: DLM made a very strong assumption about the fungible item parameters (items at the same linkage level have the same intercept and main effect). Peers have not found any evidence that this assumption was tested. DLM calculates reliability by using simulation. They use a model but peers did not see evidence that the model fits adequately to the data. Peers acknowledge that DLM plans to examine model fit (File 06, p. 36) and support that plan. Evidence includes documentation of involvement of TAC in decisions regarding the scoring model (p.162).

Overall and conditional standard error of measurement of the State’s assessments;

Overall and Conditional Standard Error of Measurement File 06: pp. 196-200, 283-284

DLM indicates that due to the model chosen, they will report classification consistency instead of overall and conditional standard error.

Consistency and accuracy of estimates in categorical classification decisions for the cut scores and achievement levels based on the assessment results;

Achievement Levels File 06: pp. 203-204

From Table 74 in File 06 (p. 204), results appear to be adequate. These analyses need to be extended to subgroups as more data are available.

For computer-adaptive tests, evidence that the assessments produce test forms with adequately precise estimates of a student’s achievement.

Computer-Adaptive Tests File 06: pp. 207-209

From Table 77 in File 06 (page 208), results are acceptable. Again, these analyses need to be extended to subgroups as more data are available.

Section 4.1 Summary Statement __X_ The following additional evidence is needed/provide brief rationale:

Beyond the one paragraph on p. 162 of File 06, provide clarification as to what was done so far and anticipated plans for what will be done in the future with




future reference)


regard to model fit.




future reference)


4.2 – Fairness and Accessibility The State has taken reasonable and appropriate steps to ensure that its assessments are accessible to all students and fair across student groups in the design, development and analysis of its assessments.

Accessibility File 06: 61-64, 46-61, 37, 69-70, 83, 69, 61-64, 82, 126-130, 136-139, 219-223, 225- 229 File 01: 15-18 File 18: 11-16 File 19: 5-16

External review of testlets is described in the Technical Manual (pp. 78-82). Content review criteria are provided (p. 79). These seem very general and perhaps limited in scope (e.g., nothing to direct writers to ordering of response options, inadvertent cueing). This has a potential impact on fairness and accessibility. The ability to evaluate the sufficiency of evidence related to fairness and accessibility is impacted by the fact that there is no reference in evidence to the process by which text passages are created (when, by whom, using what specific selection and creation/revision processes, specification, etc.) beyond indicating that they are based on/drawn from grade appropriate selections and then reduced in cognitive complexity level. (See p. 48; “short narrative passages were constructed from books commonly taught in general education, and short informational texts were written to relate to thematic elements from narratives”). There is also no reference to the processes involved in providing images/graphic adjuncts to items and testlets. Peer reviewers could find no information on when, how, and by whom images included in the assessment are selected and/or created, nor could they find any evidence of specifications or review criteria for this component of items and testlets.

Fairness File 06: pp. 69-70, 78-82, 130-132, 133, 219-223, 225-229, 231-236

See comments about DIF (3.3)




future reference)


File 05: pp. 51-53

Section 4.2 Summary Statement _X__ The following additional evidence is needed/provide brief rationale:

Include detail on development and selection of reading passages to address accessibility per above. Peers need to see steps that test developers have taken to ensure passages from general grade level texts are made accessible to SWSCD.

DLM needs to provide information to address the selection and/or creation of graphic components in the assessment (e.g., drawn or photographed images) and include criteria used to evaluate this component to ensure fairness and accessibility.




future reference)


4.3 – Full Performance Continuum The State has ensured that each assessment provides an adequately precise estimate of student performance across the full performance continuum, including for high- and low-achieving students.

Adequately Precise Estimate Across the Continuum File 06: p. 83, 112-114, 161-162, 162, 184-185,189-190, 203-204, 205-206, 207-209 File 14: 62- 63

The design of the DLM (nodes, linkage levels, EEs, etc.) and test administration placement (see first contact survey, Technical Manual pp. 83) and adaptive delivery (Technical Manual pp. 112-114) supports the capacity of the assessment to provide an adequately precise estimate of student performance across the full performance continuum as reflected in the linkage levels.





future reference)


4.4 – Scoring The State has established and documented standardized scoring procedures and protocols for its assessments that are designed to produce reliable results, facilitate valid score interpretations, and report assessment results in terms of the State’s academic achievement standards.

Standardized Scoring Procedures File 06: pp. 47-48, 52-56, 151-152, 161-162, 172-178,

190- 193, 194-195, 193-195, 193-195, 203-204, 205- 206, 207-209, 253, 254.

File 05: pp. 38-50 File 17: pp. 8-10)

The evidence provided makes clear the automated scoring procedures for the majority of items in the DLM assessment system. However, additional information may be warranted for those writing testlets (File 06, pp. 47-48) and other testlets for which teachers make score judgments while the testlet is being administered. While there is documentation of support for teacher fidelity in the training modules (pp. 253 and 254) and teacher input on student response was evaluated as part of DLM’s validity studies, peer reviewers were unable to find any evidence of monitoring procedures for this particular aspect of scoring to ensure reliable results (e.g.. inter-rater reliability). That is, where the Test Administer must “choose the description that matches the highest level of evaluation of the student’s writing” (File 05, p. 47), it is not clear whether, or how often, these choices that impact scoring are checked/confirmed.

Section 4.4 Summary Statement __X_ The following additional evidence is needed/provide brief rationale:

Because some writing testlets and other testlets involve teacher judgment on scores, DLM needs to provide an explanation of what scoring monitoring procedures (e.g., the equivalent of “read-behinds”) are being used, or what one(s) were considered but rejected and the rationale for that decision (fidelity of scoring).




future reference)


4.5 – Multiple Assessment Forms If the State administers multiple forms within a content area and grade level, within or across school years, the State ensures that all forms adequately represent the State’s academic content standards and yield consistent score interpretations such that the forms are comparable within and across school years.

Assessment Forms Represent Academic Content Standards File 06: pp. 112-114, 142-143, 125 File 07: Appendix C.7

While the DLM assessments are customized to each student—who is assigned a series of testlets rather than a fixed test form—each battery selectively represents an approved minimum number of Essential Elements which correspond to state academic content standards. The adaptive delivery method is designed to ensure coverage of the test blueprint (pp. 112-114). However, peers noted that the ELA test blueprint is presented by grade, and it is not clear what the EE coverage for each student might be (see FILE 10: ELA Blueprint—specifically page 2). See comments under 2.1. Peer reviewers were unable to find evidence to explain the comparability of coverage among students, given that each will address different EEs (although distribution of EEs across Conceptual Areas appears to be the same).

Assessment Forms Yield Consistent Score Interpretations File 06: pp. 101-106, 97-98, 61-64, 69-70, 93-101,161-162)

Because the calibrations were done separately for each linkage level, it was unclear to peer reviewers how estimated parameters were linked to the same scale. Peers felt it would be helpful if clarification were provided as to how item parameters were put on the same scale for a given linkage level.





future reference)


4.6 – Multiple Versions of an Assessment If the State administers assessments in multiple versions within a content area, grade level, or school year, the State:

Followed a design and development process to support comparable interpretations of results for students tested across the versions of the assessments;

Comparable Interpretation of Results File 06: pp. 60-61, 69-70, 69, 61-64

The evidence supports comparable interpretation of results for students taking the general form of testlets, the version for students who are blind or have visual impairment (BVI), and blind/visually impaired students who read braille. The evidence provided focuses on the item writing process and resources (Technical Manual, pp. 69-70) but does not explicitly address the implications for developing alternate versions of testlets that have a considerable visual load—but based on sample items/testlets interspersed in the Technical Manual, it appears that many include images (drawings or photographs). It is not clear how the determination of general forms of testlets that would not introduce accessibility barriers for blind students is made, prior to transcription. Peers suggest including more detail on role/impact of graphic components in items/testlets and how this is addressed in multiple versions of the assessment.

Documented adequate evidence of comparability of the meaning and interpretations of the assessment results.

Documented Evidence of Comparability File 06: pp. 97-98, 101-106, 126, 121, 130- 132, 161-162

As more data become available, peers suggest conducting modality study(ies) comparing test administration modes.





future reference)


4.7 – Technical Analysis and Ongoing Maintenance The State has a system for monitoring and maintaining, and improving as needed, the quality of its assessment system, including clear and technically sound criteria for the analyses of all of the assessments in its assessment system (i.e., general assessments and alternate assessments).

Monitoring, Maintaining, and Improving Quality of Assessment File 06: pp. 64-75, 97- 98, 97, 135-136, 136- 139, 231-236,283-284, 279 (Table 108), 280 (Table 109) File 15 File 20: pp. 2-17 File 23 File 14: Appendix B, pp. 62-63 File 15: pp. 24-25, 25-30 File 2222T

In DLM notes under this section of their submission, they reference State partners’ responsibilities for maintenance of EEs (page 15 in Section 4). Peers were unclear as to what this entails. As DLM moves to subsequent years’ administrations, peers were interested in knowing how year-to-year equating would be conducted, based on the design features of this assessment. Procedures for obtaining reliability evidence are based on AERA Standards for Educational and Psychological Testing (2014); see Technical Manual p. 203. DLM appears to have a solid system in place for identifying and implementing future studies to inform/enhance the assessment; However, it is not sufficiently clear to peers what role state partners play beyond input on topics (since the TAC “provides input on conceptualization, preliminary/exploratory analyses, and final products”), based on evidence provided (FILE 23: TAC Materials). Peers suggest that DLM clarify/provide more detail on roles/responsibilities of partners in future research.




SECTION 5: INCLUSION OF ALL STUDENTS


future reference)


5.1 – Procedures for Including Students with Disabilities The State has in place procedures to ensure the inclusion of all public elementary and secondary school students with disabilities in the State’s assessment system, including, at a minimum, guidance for individual educational plan (IEP) Teams to inform decisions about student assessments that:

Provides clear explanations of the differences between assessments based on grade-level academic achievement standards and assessments based on alternate academic achievement standards, including any effects of State and local policies on a student’s education resulting from taking an alternate assessment based on alternate academic achievement standards;


ADDRESSED IN INDIVIDUAL STATE SUBMISSIONS.

States that decisions about how to assess students with disabilities must be made by a student’s IEP Team based on each student’s individual needs;



Provides guidelines for determining whether to assess a student on the general assessment without accommodation(s), the general assessment with accommodation(s), or an alternate assessment;

Guidelines for choice of assessment File 06: pp. 252-253 File 07: Appendix C.16, G1

THE REVIEWER MAY FIND ADDITIONAL

EVIDENCE OF STATE-SPECIFIC GUIDANCE IN

INDIVIDUAL STATE SUBMISSIONS.

Technical Manual Appendix C.16 sets forth three basic guidelines for assigning a student to the alternate assessment, as well as extraneous factors that should not come into play; The Technical Manual indicates these are all reinforced during test administrator training and this is borne out in Module 1 (FILE 07 pp. 199-227)




future reference)


Provides information on accessibility tools and features available to students in general and assessment accommodations available for students with disabilities;

Information on Supports and Accommodations File 06: pp. 125-132, 252-253 File 07: Appendix G.1 File 01: pp. 15-18 THE REVIEWER MAY FIND ADDITIONAL



Peers noted that this was well addressed in the Technical Manual pp. 125-132 and Module 3 (FILE 07 pp. 242-278) and in Accessibility Manual (pp. 15-18) in section on DLM accessibility features.

Provides guidance regarding selection of appropriate accommodations for students with disabilities;

Guidance on Selection of Accommodations File 01: pp. 11-12, 19-22, 30 File 06: pp. 252-253 File 07: Appendix G.1 File 05: pp. 95-106, 60, 76 THE REVIEWER MAY FIND ADDITIONAL



Peers took note of well documented procedures; a particular strength is the DLM practice of allowing test administrators to change PNP selections and evaluate the effectiveness of accommodations determined for each student (see p. 30 of FILE 01 for questions to guide this evaluation).

Includes instructions that students eligible to be assessed based on alternate academic achievement standards may be from any of the disability categories listed in the IDEA;

Disability Categories for Alternate Assessment File 07: Appendix C.16 THE REVIEWER MAY FIND ADDITIONAL



See under Guidelines for choice of assessment, above.

Ensures that parents of students with the most significant cognitive disabilities are informed that their student’s achievement will be based on alternate academic achievement standards and of


ADDRESSED IN INDIVIDUAL STATE SUBMISSIONS. N/A




future reference)


any possible consequences of taking the alternate assessments resulting from district or State policy (e.g., ineligibility for a regular high school diploma if the student does not demonstrate proficiency in the content area on the State’s general assessments);

The State has procedures in place to ensure that its implementation of alternate academic achievement standards for students with the most significant cognitive disabilities promotes student access to the general curriculum.

Promote Access to the General Curriculum File 06: pp. 255-261, 254-261, 251-252 File 07: Appendix G.1 File 05: pp. 20-22 File 08 File 09 THE REVIEWER MAY FIND ADDITIONAL

EVIDENCE OF STATE-SPECIFIC PROCEDURES IN


The submission includes ample evidence that the design of the alternate assessment promotes access to grade level content standards.





future reference)


5.2 – Procedures for including ELs The State has in place procedures to ensure the inclusion of all English learners in public elementary and secondary schools in the State’s assessment system and clearly communicates this information to districts, schools, teachers, and parents, including, at a minimum:

Procedures for determining whether an English learner should be assessed with accommodation(s);

Determining Appropriateness of Accommodation File 06: pp. 181-183 File 01: pp. 13-30




There are procedures in place for determining accommodations that are appropriate for all SWSCD including ELS (File 01, pp. 13-30). ELs with significant cognitive disabilities may have translation provided outside the DLM system (p. 16, 17). Detail is provided in Testlet Information page (TIP) about any exceptions to allowable translation (e.g, a vocabulary item); test administrators are also permitted to translate words and provide synonyms and definitions in preferred language (FILE 05 p. 48-49).

Information on accessibility tools and features available to all students and assessment accommodations available for English learners;

Information on Supports and Accommodations File 06: pp. 125-132, 252-253 File 07: Appendix G.1 File 05: pp. 35-36, 48-49 File 01 THE REVIEWER MAY FIND ADDITIONAL



Evidence provided is adequate for this factor.

Guidance regarding selection of appropriate accommodations for English learners.

Guidance on Selection of Accommodations File 06: pp. 125-132, 252-253 File 01: pp. 15-18, 30 File 05: p. 159, 60


Consortia-level evidence provided by DLM is adequate for this section. States must provide additional evidence.




future reference)








future reference)


5.3 – Accommodations The State makes available appropriate accommodations and ensures that its assessments are accessible to students with disabilities and English learners. Specifically, the State:

Ensures that appropriate accommodations are available for students with disabilities( SWD) under the Individuals with Disabilities Education Act (IDEA) and students covered by Section 504;

Evidence of appropriate accommodation availability File 06: pp. 125-132, 254 File 01: pp. 11-12, 15-18 File 05: pp. 51-53

Based on detail in the Technical Manual and Accessibility Manual, test administrators are trained annually on IEP decision-making, which drives selection of accommodations on the assessment.

Ensures that appropriate accommodations are available for English learners (EL);

Evidence of appropriate accommodations for English learners File 06: pp. 120-121, 181-183

Translation is available as an accommodation, appropriately implemented by the test administrator, given the small % of students with significant cognitive disabilities who are EL (no translated forms)

Has determined that the accommodations it provides (i) are appropriate and effective for meeting the individual student’s need(s) to participate in the assessments, (ii) do not alter the construct being assessed, and (iii) allow meaningful interpretations of results and comparison of scores for students who need and receive accommodations and students who do not need and do not receive accommodations;

Appropriateness and effectiveness of accommodations File 06: pp. 125-132, 143-147, 126-130, 130-132, 127-128, 217-218, 270-271, 283-284 File 05: pp. 159, 55, 150-151 File 01: p. 30 File 21

Decisions on supports are well-grounded to support flexibility and equity of use, along with multiple means of engagement, representation, action and expression (pp. 130-132). Other key principles include student use of normal response mode and familiar, individualized manipulatives as required. One commendable feature is use of released testlets on which students can practice, in order to determine which accommodations will be most useful for him/her (KITE User Guide, pp. 150-151). Submission indicates plans (and rationales) for continuing research to improve use and effectiveness of accommodations; please note that rather than this detail appearing in File 06, pp. 217-18 as noted under DLM evidence, it actually appears on p. 150.




future reference)


Has a process to individually review and allow exceptional requests for a small number of students who require accommodations beyond those routinely allowed.

Exceptional accommodations requests: THIS PORTION OF THE CRITICAL ELEMENT IS


N/A





future reference)


5.4 – Monitoring Test Administration for Special Populations The State monitors test administration in its districts and schools to ensure that appropriate assessments, with or without appropriate accommodations, are selected for students with disabilities under IDEA, students covered by Section 504, and English learners so that they are appropriately included in assessments and receive accommodations that are:

Consistent with the State’s policies for accommodations;

Accommodations and participation decisions are consistent with state policy

File 01: pp. 11-12, 15-18 File 07: Appendix C.16




DLM provides guidelines through which state consortium members can monitor participation and accommodation assignment.

Appropriate for addressing a student’s disability or language needs for each assessment administered;

Appropriateness of accommodations and participation decisions for addressing student needs File 01: pp. 11-12, 15-18 THE REVIEWER MAY FIND ADDITIONAL



Similarly, DLM provides guidance on the selection of accommodations based on student needs and preferences (Accessibility Manual pp. 11-2 and 15-18).

Consistent with accommodations provided to the students during instruction and/or practice;

Consistent with accommodations during instruction and/or practice File 01: pp. 11 – 12, 19-22 File 06: pp. 136-150, 283-284 THE REVIEWER MAY FIND ADDITIONAL



DLM recommends that accommodations for use during assessment be consistent with those implemented in instructional contexts, but this must be monitored by each state. However, the consortium does collect indirect evidence of consistency between accommodations for assessment and instruction via a survey (Technical Manual p. 150).

Consistent with the assessment accommodations identified by a student’s IEP Team or 504 team

Consistent with accommodations identified by team

DLM offers, via the KITE Educator Portal, the means by which state and local educators may




future reference)


for students with disabilities, or another process for an English learner;

File 02: pp. 44-46 THE REVIEWER MAY FIND ADDITIONAL



produce lists of students’ accommodations on their PNP profile. States may use this information for monitoring.

Administered with fidelity to test administration procedures.

Administered with fidelity to procedures File 06: p.p. 124, 225-229




Along with SEA and LEA staff, DLM staff participates in monitoring the use of accommodations (Technical Manual p. 124).




SECTION 6: ACADEMIC ACHIEVEMENT STANDARDS AND REPORTING


future reference)


6.1 – State Adoption of Academic Achievement Standards for All Students The State formally adopted challenging academic achievement standards in reading/language arts, mathematics and in science for all students, specifically:

The State formally adopted academic achievement standards in the required tested grades and, at its option, also alternate academic achievement standards for students with the most significant cognitive disabilities;

THIS CRITICAL ELEMENT IS ADDRESSED IN

INDIVIDUAL STATE SUBMISSIONS. N/A

The State applies its grade-level academic achievement standards to all public elementary and secondary school students enrolled in the grade to which they apply, with the exception of students with the most significant cognitive disabilities to whom alternate academic achievement standards may apply;



The State’s academic achievement standards and, as applicable, alternate academic achievement standards, include: (a) At least three levels of achievement, with two for high achievement and a third of lower achievement; (b) descriptions of the competencies associated with each achievement level; and (c) achievement scores that differentiate among the achievement levels.



Section 6.1 Summary Statement __x_ No additional evidence is required from DLM consortium-state specific.






future reference)


6.2 – Achievement Standards-Setting The State used a technically sound method and process that involved panelists with appropriate experience and expertise for setting its academic achievement standards and alternate academic achievement standards to ensure they are valid and reliable.

Technically Sound Method File 14: pp. 19-20, 23- 24, 26-27, 31-32, 38-39, 33-34, 34-35, 34, 41, 53-57, 57-58, 44-49, 60-64, Appendix G, Appendix B, 62063 File 06: pp. 203-204

Submission provides a clear explanation of application of well-established standard setting procedures to the unique features of the DLM assessment (FILE 14, pp. 19-20)

Panelist experience and expertise File 14: pp.28, 30-31, 70-74, 51, Appendix L (201)

Standard-setting involved a range of participants, including TAC and state partners, and SEA staff, who reviewed and approved cuts. Panelist characteristics are well-defined in terms of professional role, experience with SWSCD, race, gender, geographic representation (FILE 14, pp. 30-31) Peer reviewers noticed, however, that representation was skewed toward females and Caucasians. While that may reflect the population that participants were drawn from, it would have been useful to make this clear. DLM should make an effort to better balance participation in the future.





future reference)


6.3 – Challenging and Aligned Academic Achievement Standards The State’s academic achievement standards are challenging and aligned with the State’s academic content standards such that a high school student who scores at the proficient or above level has mastered what students are expected to know and be able to do by the time they graduate from high school in order to succeed in college and the workforce.

If the State has defined alternate academic achievement standards for students with the most significant cognitive disabilities, the alternate academic achievement standards are linked to the State’s grade-level academic content standards or extended academic content standards, show linkage to different content across grades, and reflect professional judgment of the highest achievement standards possible for students with the most significant cognitive disabilities.

Challenging Alternate Academic Achievement Standards File 06: pp.17-19, 26-27, 38-41 File 15: pp. 23-24 File 18: pp. 63-67 File 06: pp. 163-164, 179- 180, 165

Evidence demonstrates that EEs (extended content standards) were developed based on CCSS and later aligned with CETE learning maps. Performance level descriptors were developed through a process informed by research and professional judgment.

Differentiated content across grades

File 06: pp. 10-13, 26-27, 179-180

PLDs are clearly based on grade level content (FILE 06, pp. 179-80) and are aligned across grades to ensure increasing complexity.





future reference)


6.4 – Reporting

The State reports its assessment results, and the reporting facilitates timely, appropriate, credible, and defensible interpretations and uses of results for students tested by parents, educators, State officials, policymakers and other stakeholders, and the public, including:

The State reports to the public its assessment results on student achievement at each proficiency level and the percentage of students not tested for all students and each student group after each test administration;

Reporting Results

File 06: pp. 165, 93-195, 282-283 File 07: Appendix E.7 THE REVIEWER MAY FIND ADDITIONAL

EVIDENCE OF STATE- SPECIFIC REPORTING IN


States are provided with detailed data files and resources to SEA staff on how they can be used (FILE 07, Appendix E7). It might be useful to peer reviewers to be able to access examples of the additional resources mentioned in Appendix E7 that are available to states on their website—even if limited to including static documents. Peers noted that the screenshot provided on p. 192 of FILE 07 gives some idea of a wide array of resources available to member states.

The State reports assessment results, including itemized score analyses, to districts and schools so that parents, teachers, principals, and administrators can interpret the results and address the specific academic needs of students, and the State also provides interpretive guides to support appropriate uses of the assessment results;

Assessment results reported to support appropriate uses of results File 06: pp. 161- 162, 190- 193, 238-239, 244- 246 File 07: Appendix E.2, E1 THE REVIEWER MAY FIND ADDITIONAL

EVIDENCE OF STATE- SPECIFIC REPORTING IN


Submission notes indicate “preliminary evidence indicates that teachers are able to use ISRs to guide instructional planning” and reference FILE 06 pp. 244-246. Some indication of plans to obtain follow-up information would be useful.

The State provides interpretive guides to support appropriate uses of the assessment results.

Interpretive Guides File 06: pp. 191-192 File 07: Appendices E.3, E.9, E.10, E.4, E.5

File 20 THE REVIEWER MAY FIND ADDITIONAL

EVIDENCE OF STATE- SPECIFIC INTERPRETIVE

GUIDES IN INDIVIDUAL STATE SUBMISSIONS.

The last page of FILE 20 provides screenshot of a list of scoring and reporting resources available to states on the DLM website; access to these documents would enhance this submission.

The State provides for the production and Delivery of Student Reports Overall, DLM provides ample evidence to address




future reference)


delivery of individual student interpretive, descriptive, and diagnostic reports after each administration of its assessments that: o Provide valid and reliable information

regarding a student’s achievement; o Report the student’s achievement in terms

of the State’s grade-level academic achievement standards (including performance-level descriptors);

o Provide information to help parents, teachers, and principals interpret the test results and address the specific academic needs of students;

o Are available in alternate formats (e.g., Braille or large print) upon request and, to the extent practicable, in a native language that parents can understand;

File 06: pp. 203-209, 238-239, 191-192, 265-274 File 07: Appendix E3 THE REVIEWER MAY FIND ADDITIONAL



most aspects of this factor. Peer reviewers noted how detailed the score report is and wonder whether reliability evidence supports this much detail in score reporting. Peer reviewers ask DLM to consider conveying to parents that there is some error in scores (to address the reliability of information regarding a student’s achievement).

The State follows a process and timeline for delivering individual student reports to parents, teachers, and principals as soon as practicable after each test administration

Process and Timeline File 06: pp. 190-193, 267 File 07: Appendix E6




Evidence from DLM is sufficient for this section.

Section 6.4 Summary Statement __X_ No additional evidence is required from the consortium.

Date post:	25-Sep-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

OFFICE OF ELEMENTARY AND SECONDARY EDUCATION...must submit satisfactory evidence to address the...

Documents