Report prepared by H. Gary Cook, Ph.D. Wisconsin Center ... Year 1 Technical... · Hawaii State...

Hawaii State Alternate Assessment Technical Report—Year 1

State of Hawaii Department of Education

Report prepared by

H. Gary Cook, Ph.D. Wisconsin Center for Education Research

January 19, 2007

FINAL VERSION V.5

HSAA Year 1 Technical Report--Final v5.doc 2

Table of Contents Introduction ................................................................................................................................................... 3 Test Development ......................................................................................................................................... 3 Test Administration ....................................................................................................................................... 5 Scoring HSAA ............................................................................................................................................... 7 Alignment of HSAA Items to APIs and APIs to HCPSIII Standards ............................................................. 9

Alignment Participants ............................................................................................................................. 10 Categorical Concurrence ......................................................................................................................... 11

Test-to-Standards Alignment ............................................................................................................... 11 Standards-to-Standards Alignment ...................................................................................................... 11

Depth of Knowledge ................................................................................................................................ 12 Test-to-Standards Alignment (Special Education) ............................................................................... 12 Standards-to-Standards Alignment ...................................................................................................... 12

Range of Knowledge (Test-to-Standards) ............................................................................................... 13 Balance of Representation (Test-to-Standards) ...................................................................................... 13 Coverage (Standards-to-Standards) ....................................................................................................... 13 Source of Challenge ................................................................................................................................ 13 Reading Alignment Results ..................................................................................................................... 14 Mathematics Alignment Results .............................................................................................................. 18

Standard Setting ......................................................................................................................................... 24 HSAA Item and Test Analyses .................................................................................................................... 26 Validation Studies ....................................................................................................................................... 34

Factor Analysis of HSAA ......................................................................................................................... 34 HSAA Teacher Survey ............................................................................................................................ 43 HSAA Parent Survey ............................................................................................................................... 45 Item Importance Ratings ......................................................................................................................... 46 IEP Alignment Ratings ............................................................................................................................. 51

Appendix A: Alignment Materials ................................................................................................................ 57 Appendix B: Web Alignment Tool Notes and Source of Challenge Output ................................................ 72

Appendix C: Standard Setting Proficiency (Performance) Level Descriptors………………………………..75


Introduction Both the Individuals with Disabilities Education Act (IDEA) and the No Child Left Behind Act of 2001 (NCLB) requires states to assess all public school students, including those students with the most significant disabilities. Both acts enjoin states to include, to the extent possible, all students in the state’s regular educational assessment program. However, for those students whose educational goals are not consistent with their peers, states are required to create and administer alternate assessments that most validly assess these students’ educational goals. The state of Hawaii has had a task-based alternate assessment program (Hawaii Alternate Assessment, or HAA) which has been used for students with the most significant disabilities. In a review of the HAA program in 2004, the Office of Special Education Programs (OSEP) conditionally approved the state’s alternate assessment system; however, several inconsistencies with federal guidelines were uncovered. Based on this report, the state took the opportunity to review its alternate assessment program. In its review, the state conferred with Dr. Steve Elliott at Vanderbilt University to gain an understanding of the different types of alternate assessment models. Upon review, the state submitted a proposal for an IDEA General Supervision Enhancement Grant utilizing an alternate assessment model developed by Dr. Elliott in two states: Wisconsin and Idaho (see Sueoka, Elliott and Cook, 2004). The federal government approved the state’s grant application, and Hawaii embarked upon the development of its new alternate assessment program (Hawaii State Alternate Assessment, or HSAA) in the fall of 2004. The work presented here outlines findings from the first live administration of the HSAA given during the spring of 2006. The report has seven sections. First, a brief description of the test development process will be given. Next a brief description of the administration of HSAA will be provided. A description of the scoring procedures used for the HSAA will be next. During the spring of 2006, alignment studies between the HSAA items to the Alternate Performance Indicators (APIs), and APIs to the HCPS III standards were conducted. Findings from these studies will be summarized. The next section will describe the standard setting session conducted for the HSAA in May, 2006. Following this, item and test statistical analyses on the HSAA will be presented, and the last section will describe a preliminary criterion-related validity study comparing the HSAA to the Social Skills Rating System (SSRS) and the Academic Competence Evaluation Scales (ACES) as well as present survey findings related to test importance and usability.

Test Development The most recent review of state alternate assessment practices indicates that 28 percent of states are using a teacher-completed rating scale for their alternate assessment for students with disabilities. A substantial body of evidence on the validity of teachers’ judgments of student behavior and academic performance provides support for this approach (e.g., Hoge and Coladarci, 1989). In addition, alternate assessments in states using rating scales (e.g., Idaho and Wisconsin) have been judged as adequately aligned to state content standards using the nationally recognized Webb approach to alignment (Roach, Elliott, & Webb, 2005). Three components underlie each of these approaches to alternate assessment: (a) the collection of classroom-based evidence as an indication of knowledge and skills representative of academic content standards, (b) a scoring rubric for evaluating the knowledge and skills, and (c) a framework for summarizing the level of proficiency exhibited by the evidence collected. The state of Hawaii’s Alternate Assessment Grant Proposal focused on developing a teacher-completed rating scale as the state’s alternate assessment tool. The development of this assessment involved the following steps:

1. Establish “Essence” statements to capture the fundamental concept in the grade-level benchmarks, then develop Alternate Performance Indicators (APIs) that link to the state’s grade-level academic standards/benchmarks in reading/language arts and mathematics.

2. Identify key knowledge and skills that enable students to demonstrate whether they have achieved the various APIs. Also identify sample classroom activities that provide students the opportunity to learn these skills.


3. Write a large pool of test items that are universally accessible and adequately represent and sample the knowledge and skills for the various content areas.

4. Develop the blueprint that the assessment would be based upon. See Table 1. 5. Develop a rating scale format and related proficiency rating and summary score rubrics. 6. Design and pilot test the new alternate assessment to evaluate the utility of the instrument

and the reliability of the ratings. Revise the rating scales and/or scoring rubrics based on the pilot test results and feedback from a focus group of users. Continue to conduct validity studies to establish the content, concurrent, construct, and instructional validity of the scores from the assessment.

7. Conduct an alignment study using Webb’s method as well as an IEP sampling study to establish evidence for content validity.

8. Develop alternate achievement (proficiency) standards and a score-reporting system that is compatible with the state’s regular assessment system.

9. Write administrative guides to facilitate reliable administration and scoring. 10. Design professional development materials to train teachers how to use the new assessment.

Conduct professional development training sessions to facilitate appropriate administration, scoring, and use of the resulting alternate assessment scores.

11. Implement the assessment statewide and conduct a consumer implementation survey. 12. Design studies to evaluate the reliability and validity of the results of the assessment during

and after the completion of the first year of statewide use. 13. Develop score reports and tools (e.g., brochures, website) to communicate with teachers and

parents about the purpose and outcomes of the alternate assessment. 14. Provide regular professional development opportunities so potential users will be well trained

to make decisions about who should participate in the HSAA and to administer, score, and report results of the assessment for AYP purposes.

In March 2005, the Alternate Assessment Workgroup used drafts of the HCPS III to understand the grade-level content standards and benchmarks that apply to all students, including those with significant cognitive disabilities. The workgroup reviewed the content standards and grade-level benchmarks for grades 3–8 and 10 in reading and mathematics. The writing content standards and grade-level benchmarks were also reviewed for grades 3, 5, 8, and 10. The workgroup determined the “essence” of the grade-level benchmarks and how this applies to students with significant cognitive disabilities. The essence describes the essential or critical function of the benchmark for that content area. The content areas are:

♦ Reading/Language Arts (Reading Conventions and Skills, Comprehension, and Literary Response and Analysis; Writing Conventions and Skills and Rhetoric) ♦ Mathematics (Number and Operations; Measurement; Geometry and Spatial Sense; Patterns, Functions, and Algebra; and Data Analysis, Statistics, and Probability) ♦ Science (development starts in summer 2006)

A representative group of Hawaii Special Education (SPED) teachers and resource teachers; curriculum resource consultants, specialists, and teachers; parents; and administrators developed a list of APIs for each grade-level benchmark. These alternate indicators identify a performance at a less complex, introductory level (or entry level) in relation to the grade-level benchmarks. Alternate assessment items were developed based on the APIs that link to or extend the grade level benchmarks to create an enhanced HSAA. An online review of these indicators and items was conducted in July 2005 using Harcourt Assessment, Inc.’s HIRO (Harcourt Item Review Online) web-based program. In addition, sample classroom tasks were developed to indicate how a student might demonstrate performance on these APIs. The blueprint for the HSAA Reading and Mathematics assessments is presented in Table 1.


Table 1: HSAA 2006 Blueprint for Reading and Mathematics Grades 3-5 Grade 6-8, 10 Reporting Category Items Points Items Points Mathematics

Number and Operations 19 57 15 45 Measurement 12 36 12 36 Geometry and Spatial Sense 18 54 15 45 Patterns, Functions, and Algebra 12 36 18 54 Data Analysis, Statistics, and Probability 14 42 15 45

Mathematics Total 75 225 75 225 Reading

Conventions and Skills 12 36 12 36 Reading Comprehension 13 39 13 39 Literary Response and Analysis 15 45 15 45

Reading Total 40 120 40 120

Total for Mathematics and Reading 155 465 155 465

Test Administration The first live administration of the HSAA was given between February and April 2006. Table 2 presents the number of students who participated in this assessment. In total, 368 Hawaii students participated in this assessment. Equal numbers of students participated in subject tests at the Elementary and Secondary levels. A total of 316 Hawaii educators participated in scoring the alternate assessment as either first or second raters. Table 2: HSAA 2006 Student Participants by Subject by Form Elementary Form Secondary Form

Grade Math Reading Math Reading 3 67 67 4 62 62 5 57 57 6 45 45 7 53 53 8 51 51 10 33 33

Total 186 186 182 182 The number of students participating in the Elementary form of HSAA is slightly more than those participating in the Secondary form. Tables 3 through 5 display the makeup of students participating in the administration of the HSAA.


Table 3: HSAA 2006 Student Participants by Gender

Grade Gender

Total F M 3 29 38 67 4 32 30 62 5 20 37 57 6 20 25 45 7 10 43 53 8 26 25 51

10 11 22 33 Total 148 220 368

As indicated in Table 3, the majority of students participating in the HSAA (59.8%) were males. In Table 4, five race/ethnicity groups had 10% or more students participating in the alternate assessment. At the Elementary level, the order of participations (from greatest to least) is as follows: Filipino (27%), Part Hawaiian (18.8%), Other (15%), Japanese (11.8%), and White (11.8%). The same five groups represented over 10% of students participating at the Secondary level, but the order differs slightly: Part Hawaiian (21.4%), Other (18%), Filipino (17%), Japanese (11.5%), and White (10.4%).

Table 4: HSAA 2006 Student Participants by Race/Ethnicity Race Statistics Elementary Secondary Total

American Indian Count 2 0 2 % within Level 1.08 0.00 0.54

Black Count 3 6 9 % within Level 1.61 3.30 2.45

Chinese Count 7 8 15 % within Level 3.76 4.40 4.08

Filipino Count 41 31 72 % within Level 22.04 17.03 19.57

Hawaiian Count 9 2 11 % within Level 4.84 1.10 2.99

Part Hawaiian Count 35 39 74 % within Level 18.82 21.43 20.11

Japanese Count 22 21 43 % within Level 11.83 11.54 11.68

Korean Count 2 3 5 % within Level 1.08 1.65 1.36

Portuguese Count 4 4 8 % within Level 2.15 2.20 2.17

Spanish or Hispanic Count 4 5 9 % within Level 2.15 2.75 2.45

Samoan Count 6 8 14 % within Level 3.23 4.40 3.80

White Count 22 19 41 % within Level 11.83 10.44 11.14

Other Count 28 34 62 % within Level 15.05 18.68 16.85

Indo Chinese Count 1 2 3 % within Level 0.54 1.10 0.82

Total Count 186 182 368 % within Level 100 100 100


Within and across grades, the three most prevalent disability types are mental retardation, multiple disabilities and autism. Table 5: HSAA 2006 Student Participants by Disability Type

Disability Type Statistics Level

Total Elementary Secondary

Autism Count 38 42 80 % within Level 20.43 23.08 21.74

Deafness Count 1 0 1 % within Level 0.54 0.00 0.27

Developmental Delay Count 6 0 6 % within Level 3.23 0.00 1.63

Emotional Disturbance Count 2 1 3 % within Level 1.08 0.55 0.82

Mental Retardation Count 61 80 141 % within Level 32.80 43.96 38.32

Multiple Disability Count 63 44 107 % within Level 33.87 24.18 29.08

Other Health Impairment

Count 13 10 23 % within Level 6.99 5.49 6.25

Specific Lang Disability

Count 2 0 2 % within Level 1.08 0.00 0.54

Total Count 186 182 368 % within Level 100 100 100

Scoring HSAA To rate HSAA, teachers were asked to collect one of six pieces of evidence from which students would be rated on each assessment item: work samples, tests, observations, interviews, media samples (videotape, audiotape, or photos) or records review. From obtained samples, teachers then were to rate collected evidence on a four-point scale (0 to 3). Table 6 presents the item scoring rubric. Table 6: HSAA Item Scoring Rubric Proficiency Rating Description

0 = Nonexistent (Can’t Do Currently) Student may be aware or attends to a task but is currently unable to perform any part of the skill or demonstrate any knowledge.

1 = Emerging (Starting to Do)

Student attends to a task and can respond to some part of the knowledge and skills required of it given significant physical, verbal, visual, or other support. The student’s response is correct or accurate a low percentage of the time even with repeated opportunities and feedback in a limited number of settings.

2 = Progressing (Can Do Partially)

Student exhibits some of the requisite knowledge and skills and requires some support. Performance may be seen in several familiar settings and improvement in accuracy has occurred with repeated opportunities and instructional feedback.

3 = Mastered (Can Do Well and Consistently)

Student exhibits the knowledge and skills required by the task and a majority of time correctly generalizes it with only limited assistance or prompting. The student routinely performs the skill in a variety of settings. The student requires little or no feedback to accurately demonstrate the knowledge and skills required.


Once a rating was chosen for an item, that value (0, 1, 2, or 3) was recorded on the HSAA Test Rating Form. The following four-step process was followed to obtain final scores for the HSAA. Step 1: A qualified rater (Rater 1) rates all items and sums the item-level ratings to obtain a total score.

This total score is then translated into a Proficiency Level. Rater 1 selects a colleague (Rater 2/Reliability Checker) to double-check ratings and total scores.

Step 2: Rater 2 reviews all the IEP-aligned evidence and the evidence provided for eight items (one item

for each reading standard and mathematics strand). On a separate form, Rater 2 rates each of these items and then compares his/her ratings to those of Rater 1. Raters 1 and 2 discuss major disagreements. Rater 2 then reviews all other ratings of Rater 1 to ensure that every item has been rated and to learn more about the student’s performance in the content area. Based on the evidence and the ratings of Rater 1, Rater 2 selects the Overall Proficiency Level that best characterizes the student’s skills.

Step 3: Raters 1 and 2 use the Interrater Agreement Chart to determine agreement or disagreement on

the student’s Overall Proficiency Level. Disagreements are discussed and resolved, and a final consensus is reached. Original rating forms and copies of the item-level evidence are submitted to the Harcourt Assessment Scoring Center for an item-level reliability check based on the evidence submitted.

Step 4: A qualified rater at the Harcourt Assessment Scoring Center reviews the rating forms and

evidence for each student to ensure completeness. The rater then scores all evidence submitted to establish the overall interrater reliability of the results.

Interrater reliability statistics between the first and second raters were calculated for each item identified as IEP aligned. The Kappa Coefficient1 was used as the tool to evaluate rater agreement. The Kappa Coefficient is used to identify independent agreement between two raters on nominal scales. Kappa was designed to account for chance agreement between raters. It is a highly useful metric "when one is concerned that the percent-agreement statistic may be artificially inflated due to the fact that most observations fall into a single category2." Cohen’s Kappa is calculated as follows:

ˆ1A E

E

P PkP−

=−

.

Where PA is the observed agreement between raters and PE is the expected agreement between raters. Kappa values are bounded between 0 and 1. The higher the coefficient, the better the agreement. Much has been written on interpreting Kappa. A generally accepted guideline is substantial agreement is exhibited with Kappa coefficients greater than 0.65. Kappas greater than 0.80 are thought to represent very good agreement, and Kappas of 1.0 represent complete agreement. Table 7 summarizes Rater 1 and Rater 2 agreement across all HSAA forms. Table 7: Summary of Rater Agreement Indices for HSAA 2006

HSAA Form Kappa Coefficient Descriptive Statistics Across All Items

Mean Minimum Maximum Range Reading Elementary 0.908 0.772 1.000 0.228 Reading Secondary 0.954 0.838 1.000 0.162 Mathematics Elementary 0.910 0.504 1.000 0.496 Mathematics Secondary 0.959 0.826 1.000 0.174

1 Cohen J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, pp. 37-46. 2 Stemler, Steven E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4). Retrieved August 30, 2006 from http://PAREonline.net/getvn.asp?v=9&n=4.


Overall, there is good agreement in scoring for HSAA across forms. Mean Kappa values are substantially high. With the exception of Mathematics Elementary (minimum value of 0.504—represented by only one item), the minimum Kappa values exhibit substantial agreement. This high degree of interrater reliability is largely due to the aggressive training required of raters of the HSAA. The Hawaii Department of Education requires HSAA raters to undergo extensive training and to pass an exam to be certified to rate the HSAA. The agreement statistics summarized above exhibit the high quality of that training. Individual items’ Kappa coefficients are presented in the item and test analysis section of this report. As indicated in step 4 above, a third qualified rater from the Harcourt Assessment Scoring Center reviewed rating forms and evidence sources for a subset of items. Rater 1 was instructed to send in three pieces of evidence per student for Reading and five pieces of evidence per student for mathematics. Only items that were judged by teachers to be IEP aligned and provided with evidence were rated by the third rater. Too few students had items that were rated by a third rater to calculate meaningful Kappa values by item. However, there was sufficient n-count to calculate the overall percent agreement rates across the matrix that contains all items by all students. The column labeled, “Total Scored by Rater 3” in Table 8 represents the number of papers that were scored by Rater 3 across all items and students. Table 8: Percent Agreement Rate between Rater 1 and Rater 3

Test Total Scored by Rater 3 Exact Adj Non_Adj Reading Elementary 535 88% 11% 1% Reading Secondary 526 81% 16% 3% Math Elementary 862 86% 11% 2% Math Secondary 805 82% 15% 4% As indicated in the table, a third rater scored between 526 to 862 pieces of evidence depending upon the grade and content area. Exact agreement indicates that Rater 3 agrees with the rating provided by Rater 1. Adjacent agreement means that Rater 1 and Rater 3 differed by one point in their proficiency ratings, and Non-Adjacent Agreement indicates that Rater 1 and Rater 3 differed by two points in their proficiency rating. The exact agreement rates were very high ranging from 81% for Secondary Reading to 88% for Elementary Reading. When the agreement rate was not exact, Rater 3 typically scored the student higher than Rater 1. For Reading Elementary, Reading Secondary, Math Elementary, and Math Secondary, respectively, Rater 3 scored the student higher 66%, 61%, 70%, and 57% of the time. Table 9 indicates the number of evidence sources that were sent in per student for the third rating. For reading, most raters (n=170) sent in the three required pieces of evidence. For math, the majority of raters sent in the required five pieces of evidence (n=152 for elementary math and n=134 for secondary math), but there were a large number of teachers who sent in less than five pieces of evidence. Table 9: Number of Evidence Sources per Student Test 0 1 2 3 4 5 Total Reading Elementary 0 7 9 170 -- -- 186 Reading Secondary 2 4 6 170 -- -- 182 Math Elementary 0 5 3 13 13 152 186 Math Secondary 5 1 9 16 17 134 182

Alignment of HSAA Items to APIs and APIs to HCPSIII Standards Alignment of a state’s assessment and its content curriculum has grown in prominence in evaluating large-scale assessments. Alignment studies provide support for the content validity evidence of a state’s assessment as it specifically relates to standards. The No Child Left Behind Act of 2001 and guidance


associated with that law (U.S. Department of Education, 20043) specify what states need to do to assure the alignment of assessments to their content standards. A variety of alignment methodologies have been proposed in the literature (see Council of Chief State School Officers, September 20024; LaMarca, P.M., Redfield, D., Winter, P.C., Bailey, A., and Hansche Despriet, L., 20015; Rothman, R., Slattery, J.B., Vranek, J.L., Resnick, L.B., 20026; and Webb, 19977). A commonly used method is one created by Dr. Norman Webb of the Wisconsin Center for Education Research. The alignment study reported here utilizes that methodology. There is a distinction in the alignments presented here, however. Webb alignments typically focus on aligning state academic achievement standards to state academic assessments, usually in the areas of reading and mathematics. The alignments presented here differ in two ways. First, the goal of this study is to identify the relationship between the state’s Alternate Performance Indicators (APIs) and the state’s alternate assessment—Hawaii State Alternate Assessment, or HSAA. Here the level of content and cognitive expectations greatly differ from assessments and standards typically aligned in the Webb alignment methodology. Roach, Elliott and Webb (2005)8 have applied Webb’s methodology to alternate assessments with moderate success. Nonetheless, slight modifications to of the Webb process seem necessary to accommodate different content and cognitive expectations. Second, the state of Hawaii wanted to understand the relationship between its APIs and its state academic content standards, specifically reading and mathematics. To support this, another alignment, a standard-to-standard alignment, was conducted. A standard-to-standard alignment is a modification of the Webb procedure, which examines the relationship between two sets of content expectations or standards (Cook, 20059). The Webb alignment process and modifications to support special education and standard-to-standard alignments are described after the studies’ participant lists are presented.

Alignment Participants The following is a list of alignment participants. On the left is displayed the group in which each panelist participated. Elementary groups aligned grades 3 through 5 and Secondary groups aligned grades 6, 7, 8, and 10. Also listed is the school building or site where panelists taught, the grade levels they taught at, and the agency where they were employed. Group Panelist Building/Site Grade Level Agency Gender Ethnicity

Elementary Math Cynthia Mew * State Literacy RT State Literacy RT/Math State Female

Asian

Elementary Math Elsa Valentini Kapolei Elem FSC K-5/Bilingual Leeward Female

Asian

Elementary Math Jennifer Shibuya Kaewai El SMI K-5 Honolulu Female

Asian

3 U.S. Department of Education. (April 28, 2004). Standards and Assessments Peer Review Guidance: Information and Examples for Meeting Requirements of the No Child Left Behind Act of 2001 Washington, D.C.: Author. 4 Council of Chief State School Officers. (September 2002). Models for Alignment Analysis and Assistance to States. Washington, D.C.: Author. 5 LaMarca, P.M., Redfield, D., Winter, P.C., Bailey, A., and Hansche Despriet, L. (2001). State Standards and State Assessment Systems: A Guide to Alignment. A study of the State Collaborative on Assessment & Student Standards (SCASS) Comprehensive Assessment Systems for ESEA Title I (CAS). Washington, D.C.: Council of Chief State School Officers. 6 Rothman, R., Slattery, J.B., Vranek, J.L., Resnick, L.B. (2002). Benchmarking and Alignment of Standards and Testing, CSE Technical Report 566. Los Angeles, CA: Center for Student and Evaluation, National Center for Research on Evaluation, Standards and Student Testing. 7 Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education (NISE Research Monograph No. 6). Madison: University of Wisconsin–Madison, National Institute for Science Education. Washington, DC: Council of Chief State School Officers. 8 Roach, A.T., Elliott, & Webb, N. (2005). Alignment of an alternate assessment with state academic standards: Evidence for the content validity of the Wisconsin Alternate Assessment. Journal of Special Education, 38 (4), 218-231. 9 Cook, H. G. (2005). Research Report #0504: Milwaukee Public Schools Alignment Study of Milwaukee Public Schools’ Learning Targets in Reading and Math to Wisconsin Student Assessment System Criterion-Referenced Test Frameworks in Reading and Math. Milwaukee, WI: Milwaukee Public Schools Office of Assessment and Accountability.


Group Panelist Building/Site Grade Level Agency Gender Ethnicity

Elementary Math Lynda Tarantino Mountain View FSC K-5 Hilo Female

Other

Elementary Math Lynne Gutierrez Manana Elem MR/SLD K-6 Leeward Female

Asian

Elementary Math Sonny Gamponia Haiku Elem Pre-5 Speech Maui Male

Asian

Elementary Read Amy Nakagawa* Windward District SPED RT Windward Female

Asian

Elementary Read Beverly Wong * Windward District SPED RT/Literacy Windward Female

Asian

Elementary Read Janine Esposo Wilcox Elem SPED 1-5 Kauai Female

Asian

Elementary Read Lori Nelson Nanakulu Elem M/Fragile K-2 & 4 Leeward Female

White

Elementary Read Tiffany Teshima Kaewai El FSC K-5 Honolulu Female

Asian

Secondary Math Bae Marchan Ilima Inter SPED MR 7-8/Bilingual Leeward Female

Asian

Secondary Math Beverly McGuire Molokai High Grade 9-12 Molokai Female

White

Secondary Math Dewey Gottileb * Math RT Leeward Secondary Math Spec. Regular Ed Leeward

Male

Other

Secondary Math Janice Rogers Aiea High SPED DH Central Female

Asian

Secondary Math Melanie Coates State Intervention Team K-12 Maui Female

White

Secondary Math Sara Vidad-Castillo Mililani Middle Grade 6-8 Central Female

Asian

Secondary Math Teresa Cowden Kohala Middle Sped 6-8 Kona Female

White

Secondary Read Aurora Agcaoili Waianae Inter FSC 7-8 Leeward Female

Asian

Secondary Read Jelna Shelton Mililani Middle Grade 8-12 Central Female

Pacific Islander

Secondary Read Joanne Fujio * State Literacy RT Kauai State Literacy RT Regular Ed/ESLL Kauai

Female

Asian

Secondary Read Linda Maeda-Lee McKinley High FSC 10 Honolulu Female

Asian

Secondary Read Pam Hirata King Intermediate FSC 7-8 Windward Female

Asian

*Leader or Co-leader/ RT = Resource Teacher

Categorical Concurrence

Test-to-Standards Alignment Different states have different ways to describe content expectations. In Hawaii, content expectations have three hierarchical levels: strand, standard, and topic. Strand is the highest level of content expectation with areas such as Number Operations in math or Reading in reading. Standards are below strands, e.g., Number Sense in Number Operations or Conventions and Skills in Reading. Below standards are topics, e.g., Numbers and Number Systems in Number Sense or Concepts in Print for Conventions and Skills. Typically, alignment is done at the lowest level and generalized upwards. That is the process used here. Categorical Concurrence then is the number of items associated with strands, standards or topics. Webb suggests that an acceptable Categorical Concurrence would be at least six test items for each content strand (using Hawaii’s terms). For this alignment, we adopt this criterion. Recall that content expectations for this alignment are the state’s APIs. So to be acceptable, we expect that at least 6 HSAA items are associated with each API strand to be acceptable.

Standards-to-Standards Alignment Unlike alignment of tests to standards, which presumes many test items will associate with topics and standards, the alignment of standards to standards should be more of a one-to-one correspondence. That is, to meet Categorical Concurrence criterion one would expect the number of standards from the target content expectations to closely equal the number of standards from the content expectations being


aligned. That is the criterion adopted here. To allow for some variance in rater judgment, an acceptability criterion of 90% is set. That is, 90% of the number of HCPSIII standards within each strand must be present in the APIs. For example, there are 3 standards in the strand Number Operations; thus, an average of 2.7 (90%) APIs need to be aligned to this strand to be acceptable.

Depth of Knowledge

Test-to-Standards Alignment (Special Education) Each content strand, standard, or topic has an associated cognitive complexity. For example, asking a student to identify which characters are in a story is a much less demanding task than asking a student to explain characters’ motives or opinions in the context of that story. In Webb's framework, the complexity of task is termed Depth of Knowledge and is identified by four levels: Level 1: Recall and recognition, Level 2: Skills and concepts, Level 3: Strategic thinking, and Level 4: Extended thinking. With special education students these levels may not be specific enough, especially at lower levels. There is a need to further expand levels to better evaluate cognitive tasks and activities expected of student with special needs. Webb’s four cognitive levels are further delineated into 6 stages as seen in the table below: Table 10: Comparison of DOK Levels and Special Education Stages

Level 1: Recall and Recognition Stage 1: Respond Stage 2: Reproduce Stage 3: Recall

Level 2: Skills and Concepts Stage 4: Skills and Concepts Level 3: Strategic Thinking Stage 5: Strategic Thinking Level 4: Strategic Thinking Stage 6: Strategic Thinking Notice that Webb’s Level 1 is represented by three stages. Again, this is done to support the specific types of elementary cognitive tasks one expects of special needs students. Copies of the special education DOK descriptions can be seen in Appendix A. According to Webb (2001) the DOK “criterion between standards and assessment is met if what is elicited from students on the assessment is as demanding cognitively as what students are expected to know and do as stated in the standards.” The acceptable level for DOK is .50 or above—i.e., 50% of HSAA items associated with API strands should be at or above the DOK levels of that strand.

Standards-to-Standards Alignment By design tests are created to “sample” student behavior, and with tests users make inferences. Seldom if ever is the entire domain of possible test items placed on one exam. That would make tests extremely lengthy. Seldom are proficient cutscores for educational measures set such that students must correctly answer all the items. Similarly, the DOK acceptability criterion for a test-to-standards alignment presumes sampling and moderate cutpoints—hence the criterion is set at 50% (0.50). This same logic does not hold for standards-to-standards alignments. The instruments being aligned in a standards-to-standards alignment are expectations of student performance, not samples of students’ behavior. A different criterion is needed to fully align, since we are, in a sense, aligning like things. Following this logic, one might expect a high proportion of standards to match DOK levels. However, the alignment reported here is with APIs, which were designed to be LESS cognitively rigorous than HCPSIII standards. What then should be an acceptable DOK level for the alignment between alternate performance indicators and Hawaii’s state standards? Federal Non-Regulatory Guidance states that “[a]n alternate


achievement standard sets an expectation of performance that differs in complexity from a grade-level achievement standard”10 (p.20). Federal Guidance further indicates that “[t]he alternate achievement standards must be challenging for students with the most significant cognitive disabilities and defined in a way that supports individual growth through a linkage to different content across grades” (p.21). There are three relevant criteria set forth in Federal Guidance: 1) less cognitively demanding, 2) challenging for students with the most significant cognitive disabilities, and 3) capable of supporting individual growth. Using these three criteria, we suggest that an acceptable DOK criterion would be 30%. If 30% of the aligned APIs are at or above the DOK levels of their aligned standards, the DOK criterion is met. Certainly, an argument could be made for a different criterion. Since there is little experience in aligning alternate performance expectations and state standards, there is little external research to draw upon. A level of 30% is less cognitively demanding but also seems to be challenging and capable of supporting individual growth for these students.

Range of Knowledge (Test-to-Standards) Webb states that “[t]his criterion is met if a comparable span of knowledge expected of students by a standard is the same as, or corresponds to, the span of knowledge that students need in order to correctly answer the assessment items/activities.” For the Range criterion to be met, at least 50% of the topics within an API standard have to have at least one associated assessment item.

Balance of Representation (Test-to-Standards) Balance of Representation (or Balance) is met if “the degree to which one [topic] is given emphasis on the assessment is comparable to the emphasis given to the other [topics] within a standard.” An index is calculated to obtain Balance. The intent of the Balance index is to identify the degree to which item coverage is spread evenly across standards/topics within a goal. According to Webb, this index must be .70 or higher to show acceptable Balance.

Coverage (Standards-to-Standards) Unlike test-to-standards alignments, standards-to-standards alignments examine the relationship between two sets of expectations. Because of the one-to-one correspondent nature of this type of alignment, the Range and Balance statistics do not adequately describe breadth. A better evaluation of breadth would be to identify the number of topics within a standard that panelists agree align. For example, let us say two or more panelists identify that a specific topic is covered by one or more standards. There is some agreement then that that indicator is “covered.” The number of indicators covered in this fashion would provide a picture of the breadth of alignment between the two sets of standards. That is the criterion adopted here. If 75% or more of topics are “covered” within a standard, Coverage will be considered acceptable. Values between 60% and 74% will be considered weak coverage, and values less than 60% are considered unacceptable.

Source of Challenge A test item is designed to evaluate specific topics. Assuming a student has not guessed at the answer, if that student correctly answers an item we expect that he or she understands or has mastered the assessed standard/ topic. If a student who has not mastered a concept gets the right answer to an item for the wrong reason, or if a student who has mastered a concept gets the wrong answer for the right reason, Webb claims that these items are a Source of Challenge. During the alignment process, raters identify items that may represent a Source of Challenge. Comments from panelists including Source of Challenge issues can be seen in Appendix B. Table 11 summarizes alignment criteria.

10 US Department of Education (2005). ALTERNATE ACHIEVEMENT STANDARDS FOR STUDENTS WITH THE MOST SIGNIFICANT COGNITIVE DISABILITIES: Non-Regulatory Guidance. Washington, D.C. Author.


Table 11: HSAA Alignment Acceptability Criterion

Test-to-Standards Alignment (HSAA items to APIs Alignment)

Acceptability Categorical Concurrence

Depth of Knowledge Range Balance

YES 6 or more items per standard

50% or more of the items at or above the standard’s DOK

50% of the standards have at least one associated item

An index value of .70 or greater.

WEAK 4 to 6 items

40%-49% of the items at or above the standard’s DOK

40%-49% of the standards have at least one associated item

An index value between .60 and .69.

NO Less than 4 items per standard

Fewer than 40% of items at or above the standard’s DOK

Fewer than 40% of the standards have at least one associated item

An index value less than .60

Standards-to-Standards Alignment (APIs to HCPSIII Alignment)

Acceptability Categorical Concurrence Depth of Knowledge Coverage

YES At least 90% the number of standards

30% or more of the items at or above the standard’s DOK

Coverage of 75% or greater

WEAK 70%-89% the number of standards

20-29% of the items at or above the standard’s DOK

60%-74% Coverage

NO Less than 70% the number of standards

Fewer than 20% of items at or above the standard’s DOK

Less than 60% Coverage

Reading Alignment Results Tables 12, 13 and 14 present findings from the API to HCPSIII Reading alignment studies. Each table has two series of columns. The first set of columns present alignment statistics and the second displays alignment findings based on the criteria set forth in Table 11. In these and subsequent tables, CAT refers to Categorical Concurrence and DOK to Depth of Knowledge. We see that Coverage for the strand Literary Response and Analysis for 3rd grade does not meet the acceptance criterion, meaning that the breadth of coverage of HCPSIII 3rd grade reading standards by 3rd grade APIs is somewhat limited. Table 12: Alignment of Reading APIs to Reading HCPS III Standards Grade 3 with 6 Panelists

Strand Alignment Statistics Alignment Findings

CAT DOK Coverage CAT DOK Coverage Conventions and Skills 100% 30% 83% YES YES YES

Reading Comprehension 100% 46% 100% YES YES YES

Literary Response and Analysis 92% 53% 50% YES YES NO

Table 10 displays the 4th grade Reading APIs to HCPSIII Standards alignment. Here, the Depth of Knowledge level for Conventions and Skills does not meet criterion.


Table 13: Alignment of Reading APIs to Reading HCPS III Standards Grade 4 with 6 Panelists


CAT DOK Coverage CAT DOK Coverage Conventions and Skills 100% 10% 100% YES NO YES


Literary Response and Analysis 100% 47% 100% YES YES YES

In Table 14, we see that the Depth of Knowledge level is unacceptable for Conventions and Skills, and the Coverage of Reading Comprehension is weak. At the end of this section of the report a summary of major alignment findings will be presented for Reading. Table 14: Alignment of Reading APIs to Reading HCPS III Standards Grade 5 with 5 Panelists



Reading Comprehension 100% 51% 67% YES YES WEAK


Table 15 shows results for the HSAA Reading Elementary form to the Reading Elementary APIs alignment. Recall that this is a test-to-standards alignment and has four criteria: Categorical Concurrence, Depth of Knowledge, Range and Balance. Findings here suggest that the HSAA Reading Elementary items are well aligned to the Reading Elementary APIs. This strongly supports the content validity of the HSAA Reading Elementary test form. Table 15: Alignment of Reading Elementary APIs to HSAA Reading Elementary Items with 5 Panelists


CAT DOK Range Balance CAT DOK Range Balance Conventions and Skills 17.8 78% 74% 0.80 YES YES YES YES

Reading Comprehension 19 79% 70% 0.75 YES YES YES YES

Literary Response and Analysis 18.4 71% 67% 0.76 YES YES YES YES

Table 16 shows alignment findings for the 6th grade Reading APIs and the 6th grade HCPSIII Reading Standards. The Depth of Knowledge level for Conventions and Skills is inadequately aligned. Essentially, alignment panelists felt that none of the APIs were at or above the Depth of Knowledge Levels of HCPSIII Standards. Further, the Coverage of Literary Response and Analysis is limited. Table 16: Alignment of Reading APIs to Reading HCPS III Standards Grade 6 with 5 Panelists




Literary Response and Analysis 100% 52% 60% YES YES NO


Similarly, 7th grade findings show that Conventions and Skills has unacceptable Depth of Knowledge level and Reading Comprehension has unacceptable breadth. Table 17: Alignment of Reading APIs to Reading HCPS III Standards Grade 7 with 5 Panelists



Reading Comprehension 100% 41% 50% YES YES NO


Eighth grade alignment findings show that the Depth of Knowledge level for Conventions and Skills is inadequate. Table 18: Alignment of Reading APIs to Reading HCPS III Standards Grade 8 with 5 Panelists





As displayed in Table 19, panelists found substantial misalignment between the 10th grade APIs and HCPSIII Reading standards. Conventions and Skills has an inadequate Depth of Knowledge level. No criteria are met for Reading Comprehension, and Literary Response and Analysis has weak Coverage. The standards-to-standards alignment at 10th grade Reading is the worst of all grades. This suggests that an examination of the APIs at this grade may be necessary. It is important to note, however, that in developing APIs for Reading Comprehension at 10th grade the state and its experts felt only one topic was appropriate for this grade and Strand. While coverage is minimal in this strand, it is so by design. Selection of appropriate content for APIs is consistent with Federal Guidance—mentioned above. While the application of the Coverage criterion indicates that it is unacceptable, it does not mean that this is a shortfall of the APIs. This notion of acceptability could also be applied to DOK as well. The alignment conducted here is designed primarily to glimpse the relationship between APIs and HCPSIII Standards. As such, it is clear that APIs across grades and subjects consistently draw from state standards. Table 19: Alignment of Reading APIs to Reading HCPS III Standards Grade 10 with 5 Panelists



Reading Comprehension 60% 0% 0% NO NO NO

Literary Response and Analysis 100% 40% 67% YES YES WEAK

Finally, Table 20 shows the alignment between HSAA Reading Secondary test items and the Reading Secondary APIs. With the exception of a weak Balance, there is good alignment between the alternate assessment Reading items and the Alternate Performance Indicators at the Secondary level.


Table 20: Alignment of Reading Secondary APIs to HSAA Reading Secondary Items with 4 Panelists


CAT DOK Range Balance CAT DOK Range Balance Conventions and Skills 13 73% 79% 0.63 YES YES YES WEAK

Reading Comprehension 17.75 88% 56% 0.74 YES YES YES YES

Literary Response and Analysis 19.25 71% 58% 0.73 YES YES YES YES

The next table summarizes alignment findings across all grades. When put together, patterns emerge. For example, practically all of the APIs’ DOK levels for Conventions and Skills do not meet the criteria. It may be that the types of cognitive tasks specified in the APIs for Conventions and Skills are not as diverse as they could be. Reading Comprehension APIs tend to have limited Coverage at grades 5, 7 and 10, and these APIs at 10th grade do not meet any criteria. Alignment of APIs in the Literary Response and Analysis strand has limited Coverage at grades 3, 6, and 10. To sum, the alignment of Reading APIs is generally good, but there are several areas of misalignment. The cognitive expectation of Conventions and Skills is limited. The breadth of coverage in Reading Comprehension and Literary Response and Analysis has some weaknesses. Finally, the 10th grade APIs are very limited in Reading Comprehension. Table 21: HSAA Alignment—Summary of Finding for Reading

APIs to HCPSIII—Standards-to-Standards Alignment Strand Grade CAT DOK Coverage

Conventions and Skills

3 YES YES YES 4 YES NO YES 5 YES NO YES 6 YES NO YES 7 YES NO YES 8 YES NO YES 10 YES NO YES

Reading Comprehension

3 YES YES YES 4 YES YES YES 5 YES YES WEAK 6 YES YES YES 7 YES YES NO 8 YES YES YES 10 NO NO NO

Literary Response and Analysis

3 YES YES NO 4 YES YES YES 5 YES YES YES 6 YES YES NO 7 YES YES YES 8 YES YES YES 10 YES YES WEAK


HSAA Items to APIs—Test-to-Standards Alignment Results

Elementary Strands CAT DOK Range Balance Conventions and Skills YES YES YES YES Reading Comprehension YES YES YES YES Literary Response and Analysis YES YES YES YES Secondary Strands Conventions and Skills YES YES YES WEAK Reading Comprehension YES YES YES YES Literary Response and Analysis YES YES YES YES

There is good alignment between HSAA Reading items and the Elementary and Secondary levels, with the minor exception of Reading Secondary Conventions and Skills Balance. The next section summarizes findings from math alignments.

Mathematics Alignment Results There are five strands in Mathematics. Table 22 presents findings for the 3rd grade standards-to-standards alignment between Mathematics APIs and Mathematics HCPSIII Standards. There is good alignment between APIs and state standards at this grade, with the exception of the Coverage of Patterns Functions, and Algebra. Table 22: Alignment of Mathematics APIs to Mathematics HCPS III Standards Grade 3 with 6 Panelists


CAT DOK Coverage CAT DOK Coverage

A - Numbers and Operations 100% 43% 100% YES YES YES

B - Measurement 100% 56% 100% YES YES YES

C - Geometry and Spatial Sense 100% 50% 75% YES YES YES

D - Patterns, Functions, and Algebra

100% 34% 50% YES YES NO

E - Data Analysis, Statistics, Probability 100% 34% 100% YES YES YES

There is weak Depth of Knowledge alignment between Geometry and Spatial Sense in the 4th grade alignment. Otherwise, there is good alignment between Mathematics APIs and state standards at this grade.


Table 23: Alignment of Mathematics APIs to Mathematics HCPS III Standards Grade 4 with 6 Panelists





C - Geometry and Spatial Sense 100% 26% 100% YES WEAK YES


100% 46% 100% YES YES YES


At 5th grade, Patterns, Functions, and Algebra has weak Depth of Knowledge. All other areas meet criteria. Table 24: Alignment of Mathematics APIs to Mathematics HCPS III Standards Grade 5 with 6 Panelists





C - Geometry and Spatial Sense 100% 57% 100% YES YES YES


100% 23% 100% YES WEAK YES


Table 25 shows the alignment between the HSAA Mathematics items and Mathematics Elementary APIs. Several areas have limited alignment. The breadth of coverage (Range or Balance or both) across all strands is limited. This suggests that broader item coverage of Elementary Mathematics APIs should be considered. Also, the strand Data Analysis, Statistics, Probability has a weak Depth of Knowledge level.


Table 25: Alignment of Mathematics Elementary APIs to HSAA Mathematics Elementary Items with 3 Panelists


CAT DOK Range Balance CAT DOK Range Balance

A - Numbers and Operations 20 95% 35% 0.71 YES YES NO YES

B - Measurement 15 92% 31% 0.64 YES YES NO WEAK

C - Geometry and Spatial Sense 19 76% 33% 0.74 YES YES NO YES


16 54% 40% 0.80 YES YES WEAK YES

E - Data Analysis, Statistics, Probability 18.33 49% 49% 0.83 YES WEAK WEAK YES

Table 26 displays the alignment between grade 6 APIs and grade 6 HCPSIII Mathematics standards. Panelists found limited alignment in three areas: Geometry and Spatial Sense, Patterns, Functions, and Algebra, and Data Analysis, Statistics, Probability. Table 26: Alignment of Mathematics APIs to Mathematics HCPS III Standards Grade 6 with 5 Panelists





C - Geometry and Spatial Sense 100% 10% 100% YES NO YES


100% 25% 50% YES WEAK NO

E - Data Analysis, Statistics, Probability 100% 35% 50% YES YES NO

At 7th grade (Table 24), Geometry and Spatial Sense show unacceptable Coverage, and Data Analysis, Statistics, Probability has a weak Depth of Knowledge level.







C - Geometry and Spatial Sense 100% 37% 25% YES YES NO


100% 42% 100% YES YES YES

E - Data Analysis, Statistics, Probability 100% 22% 100% YES WEAK YES

At 8th grade there is good alignment with the exception of Geometry and Spatial Sense, where Coverage is below criterion. Table 28: Alignment of Mathematics APIs to Mathematics HCPS III Standards Grade 8 with 3 Panelists





C - Geometry and Spatial Sense 100% 36% 50% YES YES NO


100% 71% 100% YES YES YES


Several areas at 10th grade exhibit weak alignment. Numbers and Operations has weak Coverage and Geometry and Spatial Sense and Data Analysis, Statistics, Probability have weak Depth of Knowledge levels.





A - Numbers and Operations 100% 73% 67% YES YES WEAK


C - Geometry and Spatial Sense 100% 29% 100% YES WEAK YES


100% 43% 100% YES YES YES

E - Data Analysis, Statistics, Probability 100% 25% 100% YES WEAK YES

The alignment between HSAA Mathematics Secondary items and Mathematics Secondary APIs has limited breadth of coverage, specifically in Range. That is, many Mathematics APIs are not covered by HSAA items. This may be indicative of limited items or large number of standards. In the case of Secondary Mathematics, the latter may be the prime reason for limited Range. Also, the Depth of Knowledge level for Measurement is weak. Table 30: Alignment of Mathematics Secondary APIs to HSAA Mathematics Secondary Items with 3 Panelists


CAT DOK Range Balance CAT DOK Range Balance

A - Number and Operations 16 95% 28% 0.76 YES YES NO YES

B - Measurement 8.33 47% 37% 0.84 YES WEAK NO YES

C - Geometry and Spatial Sense 17.33 61% 45% 0.79 YES YES WEAK YES


23.33 71% 14% 0.73 YES YES NO YES

E - Data Analysis, Statistics, Probability 19 54% 21% 0.82 YES YES NO YES

Table 31 summarizes Mathematics findings for both Standards-to-Standards and Test-to-Standards alignments. The strands Number Operations and Measurement have good alignment of APIs to HCPSIII standards across grades, with 10th grade being the exception. Geometry, Patterns, Functions, and Algebra, and Data Analysis, Statistics, Probability have several grades with limited Depth of Knowledge, Coverage or both.


Table 31: HSAA Alignment—Summary of Finding for Mathematics

HSAA Items to APIs—Test-to-Standards Alignment Results Grade CAT DOK Coverage

A - Numbers and Operations

3 YES YES YES 4 YES YES YES 5 YES YES YES 6 YES YES YES 7 YES YES YES 8 YES YES YES 10 YES YES WEAK

B - Measurement

3 YES YES YES 4 YES YES YES 5 YES YES YES 6 YES YES YES 7 YES YES YES 8 YES YES YES 10 YES YES YES

C - Geometry and Spatial Sense

3 YES YES YES 4 YES NO YES 5 YES YES YES 6 YES NO YES 7 YES YES NO 8 YES YES NO 10 YES NO YES


3 YES YES NO 4 YES YES YES 5 YES NO YES 6 YES NO NO 7 YES YES YES 8 YES YES YES 10 YES YES YES

E - Data Analysis, Statistics, Probability

3 YES YES YES 4 YES YES YES 5 YES YES YES 6 YES YES NO 7 YES NO YES 8 YES YES YES 10 YES NO YES


APIs to HCPSIII—Standards-to-Standards Alignment

Elementary Strands CAT DOK Range Balance A - Numbers and Operations YES YES NO YES B - Measurement YES YES NO WEAK C - Geometry and Spatial Sense YES YES NO YES D - Patterns, Functions, and Algebra YES YES WEAK YES E - Data Analysis, Statistics, Probability YES WEAK WEAK YES Secondary Strands

A - Numbers and Operations YES YES NO YES B - Measurement YES WEAK NO YES C - Geometry and Spatial Sense YES YES WEAK YES D - Patterns, Functions, and Algebra YES YES NO YES E - Data Analysis, Statistics, Probability YES YES NO YES

It is very clear from the Test-to-Standards alignment that the Range of coverage is inadequate at both the Elementary and Secondary levels. This is not surprising, however. Note the number of panelists conducting the alignment at the Secondary level for Mathematics—just three. Actually, there were six panelists, but because there were so many APIs the primary and second Mathematics alignment studies had to be split into two segments. Strands A, B and C were evaluated by three panelists, and three other panelists aligned strands D and E. One might claim that the large number of APIs are the problem, but recall that APIs are created from HCPSIII standards. It may be that too many standards/APIs are being assessed. Regardless, alignment of HSAA items to APIs at the Elementary and Secondary levels have unacceptable Range values. The Balance criterion in Measurement and the Depth of Knowledge values in Data Analysis, Statistics, Probability at the Elementary level are unacceptable as well. At the Secondary level, Measurement has weak Depth of Knowledge. At least two major findings can be seen in the Mathematics alignment. First, the alignment between APIs and HCPSIII standards in the areas of Geometry and Spatial Sense, Patterns, Functions, and Algebra, and Data Analysis, Statistics, Probability at certain grades is limited in either Depth of Knowledge or Coverage or both. At a minimum, this suggests looking at APIs to identify areas that may need changing. Regarding the alignment between HSAA Mathematics items and APIs, the breadth of coverage (Range) needs to improve to be fully aligned. Again, this may be reflective of the large number of APIs/standards. However, to fully align with current APIs, a broader spectrum of items is needed.

Standard Setting On May 10th and 11th of 2006, a standard setting workshop was conducted with 45 educators, educational administrators and parents from the state of Hawaii. The primary purpose of this workshop was to establish Alternate Achievement Standards (AAS) for the Hawaii State Alternate Assessment (HSAA). The HSAA tests students with significant cognitive disabilities in grades 3, 4, 5, 6, 7, 8, and 10. The workshop was conducted by Dr. Stephen Elliott of Vanderbilt University, Dr. H. Gary Cook of the University of Wisconsin, Dr. Selvin Chin-Chance, Ms. Leola Sueoka, and Mr. Jerald Plett of the Hawaii State Department of Education, Ms. Liz Arakaki of Harcourt Assessment, Inc. A detailed review of this standard setting process can be seen in Cook, Chin-Chance and Sueoka (2006).11

11 Cook, HG, Chin-Chance, S., Sueoka, L (2006). Summary Report: Alternate Achievement Standards Proficiency Scores for the Hawaii State Alternate Assessment. Honolulu, HI: Hawaii Department of Education


A modified bookmark procedure was used to set alternate assessment performance standards. A general description of the steps involved in this procedure is presented below:

♦ Introduction to standard setting ♦ Review all items on the rating scale ♦ Review and discuss the current proficiency descriptors for each proficiency level ♦ Reach consensus on the definition of proficient as measured by the HSAA ♦ Round 1: Individuals independently place marks in test booklets to indicate proficiency cut score ♦ Post-Round 1: Individuals at each table discuss their placements of marks for the proficiency cut

score ♦ Round 2: Table teams make a consensus decision about marks for the proficient level of

performance ♦ Post-Round 2: Feedback is provided about the mean cut scores and the likely distribution of

students at each level, then the group can discuss rationale for their ratings ♦ Round 3: Teams make final decisions about marks for each of four levels of proficiency ♦ Post-Round 3: Feedback is provided about the committee’s mean cut scores and likely impact on

student distributions as well as pre-assigned proficiency ratings given by educators who administered the HSAA.

♦ Review and revise, if necessary, the descriptions associated with each of the four levels of proficiency

The three-round procedure of bookmarking was followed for each of the content areas assessed by the HSAA at each grade level (i.e., 3rd – 8th and 10th grades for reading and mathematics). This modified Bookmark Procedure for each of the alternate assessment content areas resulted in cut scores and a refined definition of a proficient performance on the HSAA in each content area (Appendix C). The detailed result about what constitutes a “proficient performance” on the HSAA contributes information that can be integrated with other students’ results on the Hawaii State Assessment (HSA). Together the results from the HSAA and the HSA provide a summary of all students in Hawaii Public Schools who are achieving at a proficient level for purposes of the federally required Adequate Yearly Progress (AYP) calculation and report. By design, HSAA Reading forms at the Elementary and Secondary levels have 40 items and the HSAA Mathematics forms have 75 items. With a possible 3 points for each item, the maximum total number correct for HSAA Reading is 120, and for Mathematics it is 225. HSAA has two forms: Elementary and Secondary. However, standards were set for each subject and form at the grade level. Tables 32 and 33 present final cut scores for the HSAA. Table 32: Score Ranges for HSAA Proficiency Levels for Reading

Grade Proficiency Levels

Well Below Proficiency

Approaches Proficiency

Meets Proficiency

Exceeds Proficiency

Elementary Form

3 0 - 22 23 - 47 48 - 92 93 - 120

4 0 - 20 21 - 47 48 – 85 86 - 120

5 0 - 20 21 - 44 45 - 83 84 - 120

Secondary Form

6 0 - 18 19 - 35 36 - 69 70 -120

7 0 - 22 23 - 35 36 - 75 76 - 120

8 0 - 22 23 - 37 38 - 74 75 - 120

10 0 - 22 23 - 40 41 - 74 75 - 120


Table 33: Score Ranges for HSAA Proficiency Levels for Mathematics

Grade Proficiency Levels

Well Below Proficiency

Approaches Proficiency

Meets Proficiency

Exceeds Proficiency

Elementary Form

3 0 - 29 30 - 63 64 - 129 130 - 225

4 0 - 33 34 - 77 78 - 124 125 - 225

5 0 - 45 46 - 87 88 - 135 136 - 225

Secondary Form

6 0 - 43 44 - 76 77 - 123 124 - 225

7 0 - 51 52 - 89 90 - 131 132 - 225

8 0 - 56 57 - 102 103 - 145 146 - 225

10 0 - 79 80 - 123 124 - 151 152 - 225

HSAA Item and Test Analyses A total of 316 trained Hawaii teachers scored this year’s HSAA test forms. Of the 316, 166 participated as first raters; 167 were second raters, and 17 teachers served both as first and second raters. Table 34 displays summary statistics for final 2006 HSAA test scores. Also listed in this table are test reliability indices. Cronbach Alpha coefficients ranged from a low of 0.975 (Secondary Reading) to 0.986 (Secondary Mathematics). These are very acceptable reliability indices. Variances of Secondary test forms were slightly higher than Elementary forms, but score ranges for Secondary forms were smaller. Table 34: HSAA 2006 Summary Statistics by Subject and Test Form Summary Statistics

Reading Mathematics Elementary Secondary Elementary Secondary

Mean Score 45.42 40.23 69.86 75.45 N 186 182 186 182 Std. Deviation 28.35 26.23 49.35 52.90 Minimum Score 0 0 0 0 Maximum Score 120 107 201 191 Test Item Count 40 40 75 75 Cronbach Alpha 0.978 0.975 0.985 0.986


Figure 1: Histogram of Elementary and Secondary Reading Scores

120100806040200

Total Reading Score

25

20

15

10

5

0

Freq

uenc

y

Reading Elementary

120100806040200

Total Reading Score

25

20

15

10

5

0

Freq

uenc

y

Reading Secondary

Figure 1 displays histograms for both Reading forms of HSAA. The Elementary Reading form seems almost bimodal, with one group of students scoring around 10 and the other scoring close to 40. The Secondary Reading form is positively skewed. Figure 2 presents histograms for Mathematics forms of HSAA. Both Mathematics forms are positively skewed. Note the comparison in the range of scores between Elementary and Secondary forms. The Secondary Mathematics form has a much narrower range. Also, both Mathematics forms have a large number of students receiving scores of 10 or less, suggesting a floor effect for these tests. Figure 2: Histogram of Elementary and Secondary Mathematics Scores

250200150100500

Total Mathematics Scores

30

25

20

15

10

5

0

Freq

uenc

y

Mathematics Elementary

250200150100500

Total Mathematics Score

30

25

20

15

10

5

0

Freq

uenc

y

Mathematics Secondary

Tables 35 through 38 display item statistics for each HSAA form. The first column in these tables lists the test item number (e.g., r1 = reading item 1 or m1 = math item 1). The second column shows the item’s mean (highest possible score = 3), followed by the standard deviation and number of responses for that item. Following this are two statistics that identify the item’s contribution to the test. The first, “Cronbach's Alpha if Item Deleted,” identifies what reliability would be if that item were removed. If the value goes down, it indicates that that item contributes positively to the test reliability. If it goes up, this indicates that this item could potentially be removed, which would tend to improve reliability. The next column,


“Corrected Item-Total Correlation,” is a point biserial correlation. This test statistic is not relevant for graded responses (i.e., 0, 1, 2 or 3), but is used for dichotomously scored items (i.e., 0/1). The last two columns display Kappa Coefficients for 1st and 2nd raters. For example, Reading item 1 (r1) in Table 32 had 48 1st and 2nd ratings. That is, 48 students had this item as an IEP aligned item, which required a second rater to rate the item. For item r1, the agreement coefficient was 0.969, which is an adequate coefficient. The goal is to have Kappa Coefficients greater than 0.800. Table 35: HSAA Reading Elementary Item Statistics—Final Scores

Item Mean Standard Deviation N

Cronbach's Alpha if Item

Deleted Corrected Item-

Total Correlation N KAPPA

Coefficient r1 2.287 0.853 181 0.978 0.513 48 0.969 r2 2.077 0.778 181 0.977 0.513 49 0.969 r3 1.851 1.138 181 0.977 0.666 33 0.914 r4 1.265 1.089 181 0.977 0.748 37 0.923 r5 1.729 1.105 181 0.977 0.691 62 0.953 r6 1.320 1.094 181 0.977 0.744 55 0.921 r7 1.188 1.110 181 0.977 0.730 33 0.916 r8 1.475 0.981 181 0.977 0.703 48 0.772 r9 1.304 0.978 181 0.977 0.715 70 0.907

r10 1.094 0.911 181 0.977 0.765 33 0.858 r11 1.337 0.973 181 0.977 0.725 41 0.833 r12 1.265 0.975 181 0.977 0.734 33 0.806 r13 2.017 0.853 181 0.977 0.522 80 0.964 r14 1.315 1.108 181 0.977 0.731 30 0.951 r15 1.182 1.014 181 0.977 0.681 37 0.927 r16 1.182 0.997 181 0.977 0.659 36 0.922 r17 1.287 1.030 181 0.977 0.674 50 0.946 r18 0.972 0.951 181 0.977 0.753 36 0.922 r19 1.287 1.003 181 0.977 0.719 43 0.808 r20 0.939 1.006 181 0.977 0.754 35 0.847 r21 0.762 0.897 181 0.977 0.808 35 0.958 r22 0.624 0.818 181 0.977 0.719 0 . r23 0.834 0.928 181 0.977 0.757 41 0.929 r24 0.674 0.868 181 0.977 0.760 25 1.000 r25 1.000 0.955 181 0.977 0.802 33 0.912 r26 1.497 1.094 181 0.977 0.627 58 0.903 r27 0.807 0.989 181 0.977 0.679 49 0.973 r28 0.972 1.002 181 0.977 0.758 44 0.871 r29 0.536 0.827 181 0.977 0.725 27 0.873 r30 1.066 1.003 181 0.977 0.813 49 0.816 r31 0.740 0.884 181 0.977 0.812 0 . r32 0.613 0.866 181 0.977 0.722 25 1.000 r33 1.309 1.107 181 0.977 0.726 34 0.881 r34 0.923 0.951 181 0.977 0.773 32 0.862 r35 1.011 0.994 181 0.977 0.756 25 0.943 r36 1.077 0.957 181 0.977 0.717 44 0.876 r37 1.409 1.079 181 0.977 0.769 41 0.931 r38 0.691 0.915 181 0.977 0.725 28 0.893 r39 0.492 0.793 181 0.977 0.729 0 . r40 0.464 0.742 181 0.977 0.719 25 0.933


Table 36: HSAA Reading Secondary Item Statistics—Final Scores





Coefficient r1 1.0060 0.93742 166 0.974 0.735 43 1.000 r2 0.9036 0.97997 166 0.974 0.668 45 0.934 r3 1.4096 1.10685 166 0.974 0.715 55 0.903 r4 2.0000 1.13885 166 0.975 0.623 71 0.942 r5 1.4639 1.13159 166 0.974 0.726 59 1.000 r6 1.7651 0.88021 166 0.975 0.634 83 0.896 r7 1.4880 0.91279 166 0.975 0.594 87 0.838 r8 1.2048 0.84937 166 0.974 0.688 55 0.974 r9 1.2229 0.94949 166 0.974 0.682 49 1.000

r10 0.6024 0.80060 166 0.975 0.635 42 0.961 r11 0.8313 0.82857 166 0.974 0.701 51 1.000 r12 0.7169 0.80789 166 0.974 0.659 42 1.000 r13 1.4096 0.90166 166 0.975 0.621 69 0.895 r14 1.5181 0.85083 166 0.975 0.626 80 0.924 r15 1.3313 0.94316 166 0.974 0.768 58 0.976 r16 0.7892 0.86557 166 0.974 0.769 39 1.000 r17 1.1446 0.94237 166 0.974 0.821 56 0.975 r18 0.9277 0.95706 166 0.974 0.787 42 1.000 r19 1.1446 0.98636 166 0.974 0.706 48 0.911 r20 0.9639 0.92698 166 0.974 0.786 50 0.972 r21 1.0663 0.95432 166 0.974 0.741 66 1.000 r22 1.1506 0.94463 166 0.974 0.650 51 0.973 r23 0.7590 0.90239 166 0.974 0.811 41 0.962 r24 0.9398 0.88557 166 0.974 0.771 55 0.922 r25 0.8675 0.86369 166 0.974 0.792 42 0.965 r26 0.9458 0.93586 166 0.974 0.801 58 0.951 r27 1.0602 1.01324 166 0.974 0.751 77 0.928 r28 0.7048 0.91630 166 0.974 0.653 46 0.902 r29 0.5120 0.71128 166 0.974 0.719 38 0.951 r30 1.1024 0.89868 166 0.975 0.609 59 0.924 r31 0.7410 0.82338 166 0.974 0.725 43 1.000 r32 1.1627 0.93617 166 0.975 0.599 55 0.923 r33 0.3434 0.64867 166 0.975 0.651 40 0.949 r34 0.6566 0.79173 166 0.974 0.737 41 1.000 r35 0.7711 0.94485 166 0.975 0.525 50 0.937 r36 0.7349 0.87503 166 0.974 0.726 43 1.000 r37 1.1566 0.92745 166 0.975 0.582 57 0.928 r38 0.6566 0.81437 166 0.974 0.763 0 . r39 0.7229 0.87820 166 0.974 0.655 46 0.901 r40 0.5181 0.76057 166 0.974 0.693 36 1.000


Table 37: HSAA Math Elementary Item Statistics—Final Scores





Coefficient m1 0.8383 0.93343 167 0.985 0.683 35 1.000 m2 0.9701 0.95958 167 0.985 0.743 27 1.000 m3 1.1617 1.09938 167 0.985 0.749 29 0.953 m4 0.7784 0.97808 167 0.985 0.603 28 0.899 m5 1.6886 0.98729 167 0.985 0.754 91 0.918 m6 1.8443 1.13528 167 0.985 0.745 47 0.971 m7 1.4910 1.09141 167 0.985 0.790 47 0.968 m8 0.9521 1.06312 167 0.985 0.799 29 1.000 m9 0.4132 0.79339 167 0.985 0.539 22 0.784

m10 0.9281 1.04460 167 0.985 0.770 27 1.000 m11 1.2216 1.20941 167 0.985 0.741 30 0.953 m12 0.9940 1.03258 167 0.985 0.791 34 0.957 m13 0.8862 0.97819 167 0.985 0.779 25 1.000 m14 0.6467 0.87176 167 0.985 0.722 39 0.929 m15 0.8323 0.98576 167 0.985 0.737 33 0.917 m16 0.4790 0.74315 167 0.985 0.666 23 0.926 m17 0.5689 0.92149 167 0.985 0.635 30 0.877 m18 1.0838 1.04946 167 0.985 0.759 31 0.954 m19 0.2335 0.55911 167 0.985 0.567 21 0.644 m20 0.7964 1.00923 167 0.985 0.585 70 0.961 m21 1.3293 1.00866 167 0.985 0.689 53 0.923 m22 1.2874 1.13595 167 0.985 0.793 34 0.920 m23 0.9401 1.06251 167 0.985 0.767 24 0.937 m24 1.1078 1.15137 167 0.985 0.823 32 0.956 m25 1.7545 1.17436 167 0.985 0.768 34 0.879 m26 1.2994 1.06712 167 0.985 0.779 37 0.852 m27 1.7844 0.89926 167 0.985 0.530 57 0.917 m28 0.5090 0.71024 167 0.985 0.728 0 m29 0.1916 0.54810 167 0.985 0.446 21 1.000 m30 0.4491 0.72526 167 0.985 0.466 26 1.000 m31 0.4311 0.81017 167 0.985 0.612 0 m32 2.0299 1.13236 167 0.985 0.706 57 0.926 m33 2.0599 1.12850 167 0.985 0.744 71 0.913 m34 1.7066 1.14761 167 0.985 0.811 43 0.829 m35 1.4192 1.25309 167 0.985 0.663 0 m36 0.4551 0.78929 167 0.985 0.519 24 0.795 m37 1.5868 1.19358 167 0.985 0.782 32 0.870 m38 1.9641 1.09155 167 0.985 0.731 72 0.935 m39 0.9102 1.04608 167 0.985 0.586 26 0.839 m40 0.8743 1.03073 167 0.985 0.607 24 0.879 m41 0.7665 1.02350 167 0.985 0.584 23 1.000 m42 0.5988 0.87852 167 0.985 0.619 0 m43 1.5928 1.19817 167 0.985 0.823 35 0.921 m44 0.7605 0.99523 167 0.985 0.606 23 0.861 m45 0.8623 1.05805 167 0.985 0.560 27 0.894 m46 0.3832 0.66529 167 0.985 0.620 21 0.786 m47 0.4012 0.70335 167 0.985 0.636 0


Table 37: HSAA Math Elementary Item Statistics—Final Scores





Coefficient m48 0.2455 0.58564 167 0.985 0.424 25 1.000 m49 0.3593 0.59335 167 0.985 0.641 22 0.879 m50 1.3114 1.01734 167 0.985 0.626 43 0.905 m51 1.5928 1.05371 167 0.985 0.694 62 0.978 m52 1.3174 1.03604 167 0.985 0.677 56 0.903 m53 0.7964 0.93486 167 0.985 0.732 31 0.908 m54 1.2036 1.17985 167 0.985 0.765 34 0.918 m55 0.9880 0.94415 167 0.985 0.760 0 m56 0.9581 1.01408 167 0.985 0.804 27 0.899 m57 1.1198 1.05169 167 0.985 0.799 36 0.923 m58 1.1078 1.02986 167 0.985 0.794 35 0.962 m59 0.9940 1.01493 167 0.985 0.822 24 0.879 m60 0.5389 0.84132 167 0.985 0.676 25 0.868 m61 0.5449 0.85522 167 0.985 0.708 22 0.908 m62 1.3174 0.97616 167 0.985 0.672 68 0.914 m63 1.2455 0.96609 167 0.985 0.669 35 0.959 m64 1.1198 1.07436 167 0.985 0.673 44 0.848 m65 0.5569 0.82582 167 0.985 0.695 24 0.929 m66 0.4491 0.74977 167 0.985 0.602 29 0.927 m67 0.7904 0.95585 167 0.985 0.653 26 0.882 m68 1.0539 0.94587 167 0.985 0.752 34 0.920 m69 0.4731 0.79011 167 0.985 0.599 27 1.000 m70 1.1078 0.99414 167 0.985 0.606 35 0.882 m71 1.3772 1.06769 167 0.985 0.666 57 0.929 m72 0.5150 0.77503 167 0.985 0.588 27 0.865 m73 0.2934 0.56303 167 0.985 0.571 21 1.000 m74 0.2994 0.57577 167 0.985 0.589 20 0.504 m75 0.3892 0.71008 167 0.985 0.315 26 0.871

Table 38: HSAA Math Secondary Item Statistics—Final Scores





Coefficient m1 1.8851 1.04017 148 0.985 0.749 82 0.933 m2 0.8108 1.02587 148 0.986 0.642 42 0.961 m3 2.1014 1.19943 148 0.985 0.764 66 0.882 m4 1.2635 1.19159 148 0.985 0.809 44 0.968 m5 0.9865 1.08788 148 0.986 0.693 45 1.000 m6 1.4865 1.10341 148 0.985 0.788 54 0.949 m7 1.3243 1.11402 148 0.985 0.801 51 1.000 m8 1.2297 1.03078 148 0.985 0.822 43 0.968 m9 1.3041 1.14677 148 0.985 0.848 49 0.972

m10 1.1554 1.14115 148 0.985 0.838 44 1.000 m11 0.3784 0.70378 148 0.986 0.626 41 1.000 m12 0.3243 0.66205 148 0.986 0.601 40 1.000 m13 1.3446 1.19928 148 0.985 0.754 94 0.985 m14 1.2703 0.95899 148 0.986 0.737 61 0.908 m15 0.3784 0.65367 148 0.986 0.658 44 0.956







Coefficient m16 0.5068 0.76928 148 0.986 0.561 59 0.943 m17 0.4595 0.77704 148 0.986 0.499 40 1.000 m18 0.2973 0.61123 148 0.986 0.519 39 1.000 m19 0.6284 0.89807 148 0.986 0.629 50 0.968 m20 0.5946 0.85585 148 0.986 0.627 56 0.949 m21 1.3784 1.10906 148 0.986 0.647 65 0.938 m22 1.1419 1.12492 148 0.986 0.744 42 0.963 m23 1.2500 1.11193 148 0.986 0.697 53 0.888 m24 1.0405 0.99576 148 0.986 0.694 86 0.903 m25 0.7432 0.98369 148 0.986 0.681 43 0.952 m26 0.7432 0.94128 148 0.986 0.506 46 0.929 m27 0.4595 0.72260 148 0.986 0.594 43 0.949 m28 1.7230 1.10547 148 0.985 0.801 63 0.936 m29 1.6892 1.13000 148 0.985 0.818 56 0.976 m30 1.3176 0.99685 148 0.985 0.761 47 0.912 m31 0.7568 1.01433 148 0.985 0.800 44 1.000 m32 2.0270 1.07505 148 0.986 0.739 87 0.917 m33 1.0811 1.15772 148 0.985 0.776 42 1.000 m34 1.0135 1.17216 148 0.986 0.714 37 1.000 m35 0.8514 1.04566 148 0.986 0.685 37 1.000 m36 0.3378 0.62329 148 0.986 0.548 38 1.000 m37 1.5405 1.09655 148 0.985 0.826 40 1.000 m38 1.7568 1.13462 148 0.986 0.644 49 0.972 m39 1.1892 1.12698 148 0.986 0.612 61 1.000 m40 0.8446 0.98086 148 0.986 0.620 46 0.966 m41 1.6757 1.17350 148 0.985 0.759 64 0.956 m42 1.5000 1.13389 148 0.985 0.751 44 0.907 m43 1.0811 0.90737 148 0.986 0.745 52 0.944 m44 1.4595 1.09033 148 0.986 0.671 62 1.000 m45 1.0338 0.97879 148 0.986 0.711 62 0.911 m46 1.2838 1.03701 148 0.986 0.721 48 0.914 m47 1.2905 1.00511 148 0.985 0.786 53 0.974 m48 0.8581 0.98292 148 0.985 0.773 44 0.929 m49 0.9730 0.98937 148 0.985 0.796 43 0.932 m50 0.9797 0.92926 148 0.985 0.783 44 0.966 m51 1.2770 1.10547 148 0.985 0.828 65 0.915 m52 1.2838 1.10065 148 0.985 0.824 44 0.969 m53 1.4189 1.17811 148 0.985 0.768 48 0.972 m54 1.3041 1.06687 148 0.985 0.785 59 0.906 m55 1.2230 1.22791 148 0.985 0.758 52 0.946 m56 0.3108 0.72704 148 0.986 0.547 40 1.000 m57 0.3851 0.77831 148 0.986 0.574 41 0.948 m58 1.1959 1.09208 148 0.985 0.758 50 0.973 m59 0.4054 0.77229 148 0.986 0.644 40 1.000 m60 0.4865 0.75129 148 0.986 0.666 41 1.000 m61 0.9527 0.91350 148 0.986 0.553 68 0.958 m62 1.1689 1.01946 148 0.986 0.642 79 0.914 m63 0.6351 0.83421 148 0.986 0.670 45 0.826







Coefficient m64 0.6351 0.87403 148 0.986 0.653 53 0.914 m65 1.1824 0.91872 148 0.985 0.777 65 1.000 m66 1.2703 1.03408 148 0.985 0.778 58 1.000 m67 0.3176 0.59519 148 0.986 0.603 39 1.000 m68 0.3851 0.68535 148 0.986 0.551 41 1.000 m69 0.5811 0.82490 148 0.986 0.643 0 m70 0.2365 0.53910 148 0.986 0.580 41 1.000 m71 0.6554 0.93104 148 0.986 0.622 61 0.976 m72 0.3851 0.64443 148 0.986 0.560 41 1.000 m73 0.3446 0.60286 148 0.986 0.451 50 1.000 m74 0.2568 0.62945 148 0.986 0.481 36 1.000 m75 0.1351 0.38063 148 0.986 0.357 36 1.000

Table 39 presents summaries of item means, inter-item correlations and Kappa coefficients. The mean of item means is 1.147 for the Reading Elementary form of HSAA. The minimum item mean is 0.464 and the maximum item mean is 2.287. Low item mean values are more difficult items and high mean values are easier items. Inter-item correlations reflect how well items are correlated with each other. The average inter-item correlation is around 0.500. It is interesting to note that in only two forms—Reading Elementary and Mathematics Elementary—is the minimum Kappa coefficient less than 0.800. In fact, only six items have Kappas less than 0.800. They are item 8 in the Reading Elementary form and items 9, 19, 36, 46 and 74 on the Mathematics Elementary form. Again, this is a good reflection on the training of the raters. Table 39: Summaries of Item Statistics Reading Elementary Mean Min Max Item Means 1.147 0.464 2.287 Inter-item Correlation 0.524 0.191 0.871 Kappa Coefficient 0.908 0.772 1.000 Reading Secondary Mean Min Max Item Means 1.010 0.343 2.000 Inter-item Correlation 0.499 0.195 0.839 Kappa Coefficient 0.954 0.838 1.000 Mathematics Elementary Mean Min Max Item Means 0.964 0.192 2.060 Inter-item Correlation 0.465 0.068 0.934 Kappa Coefficient 0.910 0.504 1.000 Mathematics Secondary Mean Min Max Item Means 0.976 0.135 2.101 Inter-item Correlation 0.478 0.079 0.932 Kappa Coefficient 0.954 0.838 1.000

Overall, both item and test characteristics for all forms of the HSAA are very good and suggest that reliable information may be obtained from all test forms. Prior to moving on to the next session, a comment about the 3rd set of ratings of HSAA test forms is necessary. All item and test statistics described above derive from the final scores given for each student. However, Kappa Coefficients are derived from only the 1st and 2nd raters—all Hawaii educators.


For this administration of HSAA, Harcourt Assessment, Inc. back-checked as a validation process a subset of HSAA test forms. Too few students had items that were rated by a third rater to calculate meaningful Kappa values by item. However, the percent agreement rates between Rater 1 and Rater 3 were able to be calculated, and those results appear in the section entitled, “Scoring HSAA”.

Validation Studies A preliminary examination of HSAA validity was conducted while piloting the test in the fall 2005-2006 school year. Both surveys and a criterion-related validity study were conducted on the pilot test and are presented in Cook (2006)12. The criterion-related validity study explored the relationship between the HSAA pilot test, the Social Skills Rating System (Gresham and Elliott, 1990)13 and the Academic Competence Evaluation Scales (DiPerna and Elliott, 2000)14. Moderate correlations were found between these assessments, which suggested that constructs being evaluated on HSAA were related to constructs measured on these content-related, special education, clinical assessments. Also, a focus group session was conducted to streamline the test and solicit input on the importance and consequence of the test. Three additional validity studies are provided in this report. First, earlier in this report, results from alignments between the HSAA and Hawaii’s APIs and Hawaii’s APIs and state standards (HCPSIII) are presented. Second, an evaluation of the underlying constructs evaluated on HSAA is presented below. Finally, consequential validity evidence is provided in two forms: results from a teacher survey on HSAA as well as HSAA item importance ratings.

Factor Analysis of HSAA Samuel Messick’s (1989)15 seminal work on validity argues that everything is “construct validity.” That is, all methods and mechanisms used for collecting evidence to validate a test and its use relate back to that test’s underlying constructs. This report has already discussed two facets of examining the HSAA’s underlying constructs: content validations (alignment studies), and criterion-related validity. The reliability findings mentioned in the previous section also support this test’s validity. Another method of collecting construct evidence is to look at the underlying structure of the test. This is commonly done either by conducting a multitrait-multimethod analysis or some sort of factor analytic approach—most commonly factor analysis. For this report, a factor analysis of the HSAA forms is used, specifically exploratory factor analysis (EFA). One of the drawbacks in using factor analysis is there is no underlying statistical test that indicates findings are “statistically significant.” Factor analysis is a reductive, descriptive technique with no hard and fast rules. Fortunately, there are guidelines for using EFA. We adopt the following criteria when evaluating EFA results in this report:

♦ First, we will be using a principal component analysis to extract major factor components in the analyses presented here.

♦ Second, we will enforce orthogonal relationships between items when conducting EFAs. Since we are using SPSS 14.0 for Windows16 we will select the Varimax rotation method for extracting orthogonal factors.

♦ Third, once extracted, we will examine factors that have eigenvalues greater than one. ♦ Fourth, we will examine scree plots to identify factors to be retained, and ♦ Fifth, test items with factor loadings greater than 0.60 will be considered associated with factor

components. Note that this is a very conservative assumption.

12 Cook, H.G. (March 2006). Hawaii State Alternate Assessment Pilot Test Administration Technical Report. Honolulu, HI: Hawaii Department of Education. 13 Gresham, F.M and Elliott, S.N. (1990). Social Skills Rating System (SSRS). Circle Pines, MN: AGS Publishing. 14 DiPerna, J.C. and Elliott, S.N. (2000). Academic Competence Evaluation Scales (ACES). San Antonio, TX: Harcourt Assessment, Inc. 15 Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan. 16 SPSS. (Nov 2005). SPSS 14.0 for Windows. Chicago, IL: Author.


Four factor analytic studies are presented below, one for each form of the HSAA: Reading Elementary, Reading Secondary, Mathematics Elementary, and Mathematics Secondary. Summaries of major findings are provided to support interpretation. Table 40 presents eigenvalues and total variance explained from a factor analysis of the HSAA Reading Elementary test form. Four components (factors) were extracted from this analysis with eigenvalues greater than one, and they are listed in the table below. The first factor accounts for 54% of the variance between items and represents a very strong first factor. An examination of the associated scree plot suggests that four factors is a reasonable finding from this analysis. Table 40: HSAA Reading Elementary Factor Loadings

Component Eigenvalues % of Variance Cumulative % 1 21.65 54.12 54.12 2 2.38 5.96 60.08 3 1.55 3.86 63.94 4 1.37 3.42 67.36

The following table highlights item factor loadings from this analysis. Cells highlighted in gray have factor loadings greater than or equal to 0.60, which suggests that they are strongly related to that factor. Notice also that items’ strands are displayed in the second column. If HSAA items strongly associate (high factor loadings) with strands, this suggests that the underlying constructs being measured by the items and strands are related. Table 41: HSAA Reading Elementary Exploratory Factor Analysis Rotated Component Matrix

Item Strand Component Factors

1 2 3 4 r1

Con

vent

ions

and

Ski

lls

0.119 0.181 0.716 0.265 r2 0.217 0.101 0.756 0.176 r3 0.132 0.467 0.514 0.406 r4 0.345 0.368 0.233 0.693 r5 0.160 0.408 0.310 0.726 r6 0.322 0.330 0.253 0.761 r7 0.380 0.501 0.220 0.376 r8 0.276 0.681 0.249 0.165 r9 0.187 0.709 0.204 0.359 r10 0.403 0.638 0.179 0.254 r11 0.274 0.660 0.246 0.269 r12 0.306 0.743 0.193 0.154 r13

Rea

ding

Com

preh

ensi

on

0.207 0.235 0.642 0.118 r14 0.395 0.529 0.338 0.183 r15 0.326 0.616 0.178 0.193 r16 0.268 0.697 0.145 0.142 r17 0.232 0.749 0.149 0.162 r18 0.411 0.513 0.206 0.381 r19 0.356 0.626 0.149 0.258 r20 0.657 0.326 0.067 0.388 r21 0.686 0.305 0.200 0.379 r22 0.669 0.378 0.065 0.184 r23 0.662 0.357 0.055 0.342 r24 0.688 0.468 0.153 0.028


Table 41: HSAA Reading Elementary Exploratory Factor Analysis Rotated Component Matrix

Item Strand Component Factors

1 2 3 4 r25 0.604 0.559 0.149 0.149 r26

Lite

rary

Res

pons

e an

d A

naly

sis

0.297 0.426 0.540 0.039 r27 0.696 0.242 -0.023 0.337 r28 0.672 0.217 0.191 0.422 r29 0.838 0.116 0.166 0.224 r30 0.564 0.485 0.184 0.339 r31 0.659 0.348 0.294 0.274 r32 0.609 0.406 0.336 -0.019 r33 0.574 0.288 0.435 0.145 r34 0.708 0.253 0.203 0.325 r35 0.643 0.373 0.415 -0.018 r36 0.644 0.292 0.378 0.046 r37 0.513 0.376 0.338 0.325 r38 0.690 0.320 0.305 0.013 r39 0.784 0.269 0.194 0.053 r40 0.795 0.169 0.211 0.151

The following table identifies the number of items associated with each factor and strand. The first factor (encompassing 54% of the variance) is related to both Reading Comprehension and Literary Response and Analysis. The second and third factors associate most closely with Conventions and Skills and Reading Comprehension. The final factor associates with Conventions and Skills. In total, 32 HSAA Reading Elementary items associate with first four factors. Conversely, 8 items did not associate with these factors, e.g., items 7 (r7) and 14 (r14). This does not mean that these items are invalid or unimportant. It does suggest that examination of these items or examination of the scoring of these items may be in order. Table 42: Items and Strands Associated with the HSAA Reading Elementary Factor Analysis

Strands Number of Items Associated with Each Factor

Factor 1 Factor 2 Factor 3 Factor 4 Conventions and Skills 5 2 3 Reading Comprehension 6 4 1 Literary Response and Analysis 11 Tables 43 and 44 present factor analysis findings for the HSAA Mathematics Elementary test form. Notice that Table 40 has a bold line under the fourth factor. This indicates that more than four factors have eigenvalues greater than 1.0. A scree plot suggests that four factors provide the best explanation for the variance associated with Mathematics Elementary items. As an aside, the actual factor analysis uncovered 11 factors with eigenvalues greater than one. Nonetheless, inter-item relationships are best explained by these four component factors, which cumulatively explain 61% of the variability.

Table 43: HSAA Mathematics Elementary Factor Loadings Component Eigenvalues % of Variance Cumulative %

1 36.14 48.19 48.19 2 4.07 5.42 53.61 3 3.64 4.85 58.46 4 2.12 2.82 61.28

5 1.94 2.59 63.87 6 1.59 2.12 65.99 7 1.44 1.92 67.91


The first component accounts for 48% of the variability, with subsequent factors having markedly less associated variance. The next table displays factor loadings for the 75 Mathematics items. Table 44: HSAA Mathematics Elementary Exploratory Factor Analysis Rotated Component Matrix

Item Strand 1 2 3 4 m1

Num

ber O

pera

tions

0.724 0.357 0.010 0.050 m2 0.530 0.414 0.080 0.239 m3 0.349 0.462 0.015 0.247 m4 0.101 0.357 0.122 0.199 m5 0.343 0.735 0.120 0.128 m6 0.348 0.757 0.123 0.046 m7 0.442 0.751 0.069 0.066 m8 0.672 0.379 0.072 0.238 m9 0.283 0.145 0.149 0.123

m10 0.679 0.397 0.160 0.227 m11 0.544 0.430 0.148 0.289 m12 0.738 0.412 0.181 0.181 m13 0.725 0.392 0.204 0.208 m14 0.824 0.269 0.066 0.084 m15 0.702 0.301 0.098 0.153 m16 0.483 0.203 0.283 0.057 m17 0.690 0.247 0.092 -0.009 m18 0.312 0.327 0.149 0.611 m19 0.572 -0.069 0.080 0.060 m20

Mea

sure

men

t

0.388 0.471 0.014 0.087 m21 0.216 0.386 0.073 0.711 m22 0.273 0.419 0.315 0.463 m23 0.272 0.302 0.283 0.537 m24 0.480 0.333 0.226 0.443 m25 0.363 0.481 0.161 0.459 m26 0.447 0.513 0.042 0.130 m27 0.050 0.404 0.114 0.044 m28 0.661 0.254 0.006 0.152 m29 0.155 0.060 -0.085 0.181 m30 0.109 0.066 0.086 0.177 m31 0.248 0.060 0.235 0.368 m32

Geo

met

ry

0.210 0.767 0.185 0.222 m33 0.209 0.758 0.159 0.158 m34 0.192 0.721 0.216 0.247 m35 0.144 0.594 0.286 0.050 m36 0.341 0.208 0.068 0.093 m37 0.261 0.619 0.342 0.330 m38 0.217 0.823 0.179 0.136 m39 0.091 0.301 0.837 0.167 m40 0.156 0.292 0.851 0.147 m41 0.124 0.265 0.824 0.117 m42 0.317 0.246 0.200 0.275 m43 0.283 0.646 0.284 0.243


Table 44: HSAA Mathematics Elementary Exploratory Factor Analysis Rotated Component Matrix Item Strand 1 2 3 4 m44 0.140 0.308 0.276 0.117 m45 0.036 0.260 0.259 0.106 m46 0.372 0.189 0.406 -0.067 m47 0.363 0.189 0.329 0.064 m48 0.180 0.178 -0.002 0.045 m49 0.385 0.231 0.297 0.165 m50

Pat

tern

s an

d Fu

nctio

ns

0.333 0.305 0.114 0.242 m51 0.321 0.672 0.040 0.094 m52 0.374 0.590 0.221 0.197 m53 0.428 0.239 0.140 0.470 m54 0.560 0.520 0.230 0.040 m55 0.485 0.394 0.152 0.243 m56 0.522 0.339 0.225 0.251 m57 0.431 0.495 0.295 0.163 m58 0.535 0.480 0.213 0.110 m59 0.546 0.407 0.061 0.314 m60 0.822 0.168 0.059 0.157 m61 0.745 0.172 0.118 0.115 m62

Dat

a A

naly

sis,

Sta

tistic

s an

d P

roba

bilit

y

0.316 0.478 0.119 0.079 m63 0.169 0.321 0.219 0.162 m64 0.126 0.323 0.245 0.155 m65 0.402 0.170 0.125 0.323 m66 0.450 0.218 0.022 0.089 m67 0.358 0.257 0.255 0.096 m68 0.274 0.366 0.361 0.562 m69 0.483 0.108 -0.098 0.265 m70 0.114 0.252 0.173 0.209 m71 0.113 0.466 0.118 0.138 m72 0.313 0.125 0.267 0.223 m73 0.402 0.121 -0.024 0.127 m74 0.302 0.145 0.188 0.032 m75 0.108 -0.067 0.066 0.145

The first factor seems most strongly associated with Number Operations, Patterns and Functions and Measurement, but mostly with Numbers and Operations. The second factor relates most strongly with Geometry as does the third factor. The fourth factor has only two associated items and relates with Numbers and Operations and Measurement.


Table 45: Items and Strands Associated with the HSAA Mathematics Elementary Factor Analysis


Factor 1 Factor 2 Factor 3 Factor 4

Numbers and Operations 8 3 1

Measurement 1 1

Geometry 6 3

Patterns and Functions 2 1

Data Analysis, Statistics and Probability

Several interesting findings can be seen in this table. First, notice that, in total, only 26 of the 75 HSAA items strongly associate with the first four components. The strand Data Analysis, Statistics and Probability (Data Analysis) has no items associated with the first four factors, and Measurement has only two associated items. It is not surprising that Data Analysis has no associated items. The number of strands Mathematics has exceeds the number of primary factors. Findings suggest that some items may not be contributing to the lion’s share of the variance of this test. It is possible that fewer, more targeted items could be used to evaluate these strands. Further investigation is suggested, specifically examining items that do not strongly associate with these factors and potentially culling items to make this test form more efficient. Table 46: HSAA Reading Secondary Factor Loadings

Component Eigenvalues % of Variance Cumulative % 1 20.65 51.63 51.63 2 2.54 6.34 57.97 3 1.53 3.83 61.80

4 1.30 3.26 65.05 The above table shows the factors associated with the HSAA Reading Secondary factor analysis. Again, note that there is a bold line below the third component. There were four factors that had eigenvalues greater than 1.0; however, the scree plot suggested only three factors be included. With three factors, 62% of the variance is explained. Table 47 presents the rotated component matrix for this form of HSAA.


Table 47: HSAA Reading Secondary Exploratory Factor Analysis Rotated Component Matrix Item Strand 1 2 3 r1

Con

vent

ions

and

Ski

lls

0.379 0.232 0.666 r2 0.170 0.125 0.762 r3 0.229 0.130 0.830 r4 0.173 0.029 0.722 r5 0.201 0.144 0.789 r6 0.206 0.129 0.363 r7 0.117 0.243 0.272 r8 0.260 0.205 0.144 r9 0.387 0.087 0.218 r10 0.527 0.041 0.297 r11 0.116 0.244 0.247 r12 0.248 0.150 0.232 r13

Rea

ding

Com

preh

ensi

on

0.102 0.150 0.114 r14 0.122 0.363 0.215 r15 0.299 0.264 0.403 r16 0.364 0.212 0.364 r17 0.371 0.226 0.442 r18 0.374 0.412 0.327 r19 0.431 0.149 0.253 r20 0.342 0.422 0.247 r21 0.226 0.260 0.254 r22 0.258 0.813 0.161 r23 0.533 0.370 0.198 r24 0.491 0.417 0.237 r25 0.338 0.360 0.385 r26

Lite

rary

Res

pons

e an

d A

naly

sis

0.443 0.331 0.353 r27 0.400 0.313 0.218 r28 0.593 0.188 0.332 r29 0.692 0.206 0.162 r30 0.189 0.808 0.154 r31 0.554 0.421 0.140 r32 0.249 0.755 0.038 r33 0.548 0.345 0.179 r34 0.631 0.370 0.225 r35 0.671 0.089 0.131 r36 0.577 0.277 0.386 r37 0.170 0.778 0.052 r38 0.538 0.469 0.169 r39 0.613 0.421 0.104 r40 0.530 0.479 0.197


Table 48: Items and Strands Associated with the HSAA Reading Elementary Factor Analysis


Factor 1 Factor 2 Factor 3 Conventions and Skills 5 Reading Comprehension 1 Literary Response and Analysis 4 3 Table 48 summarizes the strands and number of items associated with the three main factors. Factor 1 strongly relates to Literary Response and Analysis. Factor 2 also has a strong Literary Response and Analysis component, and one item from Reading Comprehension is also associated with this factor. Factor 3 loads on Conventions and Skills. Note, however, that only 13 of the 40 items strongly load on the first three factors. Table 49 presents the final factor analysis. In this analysis, many factors had eigenvalues greater than 1.0. Four factors seem to best describe the variance, and these four factors account for 61% of the variance. Table 49: HSAA Mathematics Secondary Factor Loadings

Component Eigenvalues % of Variance Cumulative % 1 37.09 49.46 49.46 2 4.37 5.82 55.28 3 2.62 3.50 58.78 4 2.04 2.72 61.49

5 1.93 2.57 64.06 6 1.58 2.11 66.17 7 1.51 2.01 68.18

Table 50: HSAA Mathematics Secondary Exploratory Factor Analysis Rotated Component Matrix Strand 1 2 3 4 m1

Num

bers

and

Ope

ratio

ns

0.760 0.164 0.255 0.038 m2 0.451 0.460 0.188 0.154 m3 0.805 0.150 0.187 0.147 m4 0.666 0.355 0.101 0.217 m5 0.494 0.182 0.473 0.037 m6 0.739 0.251 0.239 0.066 m7 0.725 0.352 0.237 0.070 m8 0.661 0.327 0.224 0.167 m9 0.716 0.274 0.226 0.206 m10 0.674 0.310 0.199 0.186 m11 0.266 0.756 0.122 0.044 m12 0.254 0.766 0.106 0.023 m13 0.735 0.240 0.084 0.216 m14 0.557 0.183 0.263 0.261 m15 0.357 0.426 0.113 0.051 m16

Mea

sure

men

t 0.282 0.679 -0.158 0.301 m17 0.201 0.726 -0.036 0.298 m18 0.162 0.756 0.160 -0.040 m19 0.236 0.648 0.235 0.086 m20 0.244 0.562 0.334 0.091



0.351 0.162 0.724 0.130 m22 0.457 0.203 0.642 0.150 m23 0.479 0.043 0.470 0.242 m24 0.636 0.048 0.105 0.073 m25 0.291 0.157 0.422 0.263 m26 0.230 0.258 0.141 0.109 m27 0.322 0.376 0.141 0.047 m28

Geo

met

ry

0.679 0.198 0.258 0.265 m29 0.711 0.171 0.334 0.221 m30 0.687 0.142 0.265 0.137 m31 0.471 0.454 0.270 0.107 m32 0.764 0.106 0.166 0.158 m33 0.503 0.147 0.121 0.539 m34 0.431 0.084 0.172 0.709 m35 0.436 0.048 0.189 0.705 m36 0.282 0.392 -0.007 0.070 m37 0.679 0.171 0.276 0.222 m38 0.616 0.148 0.178 0.333 m39 0.332 0.122 0.514 0.456 m40 0.491 0.210 0.063 0.634 m41 0.803 0.310 -0.025 0.218 m42 0.761 0.336 -0.094 0.171 m43

Pat

tern

s an

d Fu

nctio

ns

0.650 0.152 0.040 0.141 m44 0.747 0.062 -0.016 -0.014 m45 0.579 0.168 0.133 0.082 m46 0.567 0.082 0.131 0.328 m47 0.650 0.113 0.281 0.076 m48 0.460 0.388 0.196 0.207 m49 0.610 0.190 0.227 0.228 m50 0.539 0.313 0.457 -0.039 m51 0.710 0.197 0.272 0.265 m52 0.606 0.236 0.404 0.207 m53 0.664 0.370 0.226 -0.041 m54 0.665 0.161 0.181 0.323 m55 0.697 0.256 0.020 0.230 m56 0.235 0.553 0.136 -0.157 m57 0.319 0.259 0.127 0.137 m58 0.637 0.279 0.191 0.217 m59 0.240 0.571 0.157 -0.034 m60 0.306 0.427 0.093 0.182 m61

Dat

a A

naly

sis,

S

tatis

tics

and

Pro

babi

lity

0.190 0.188 0.293 0.454 m62 0.381 0.266 0.375 0.338 m63 0.353 0.391 0.190 0.124 m64 0.315 0.525 0.331 0.127 m65 0.554 0.365 0.407 0.148 m66 0.575 0.232 0.519 0.103



Dat

a A

naly

sis,

S

tatis

tics

and

Pro

babi

lity

0.145 0.592 0.167 0.202 m68 0.195 0.325 0.095 0.022 m69 0.309 0.457 0.165 0.317 m70 0.167 0.539 0.075 0.146 m71 0.306 0.246 0.133 0.194 m72 0.276 0.171 0.093 0.106 m73 0.131 0.047 0.133 0.107 m74 0.133 0.233 0.179 0.189 m75 0.108 0.134 0.077 0.099

Table 51: Items and Strands Associated with the HSAA Mathematics Secondary Factor Analysis


Factor 1 Factor 2 Factor 3 Factor 4

Numbers and Operations 9 2

Measurement 1 4 2

Geometry 8 3

Patterns and Functions 10

Data Analysis, Statistics and Probability

Finally, the first factor, which accounts for almost 50% of the variance, is most strongly represented by Patterns and Functions, Numbers and Operations, and Geometry. The second factor associates with Measurement and Numbers and Operations. The third factor associates with just Measurement and the last factor with Geometry. In total, 39 items strongly associate with the first four factors. As with Elementary Mathematics, Data Analysis has no strongly associating items represented in the first four factors. When examining factor analyses from all four HSAA test forms, we see some important findings. First, test items clearly seem to pattern with specific HSAA strands. This suggests, albeit imperfectly, that the underlying constructs expected to be assessed by the HSAA are being assessed in some fashion. Second, several items, especially in the area of Mathematics, do not seem to be strongly associated with identified factors. These items may be assessing different constructs or different facets of the given constructs. It could be that teacher scoring is systematically contributing to the relatively small number of items associating with identified factors. Nonetheless, findings indicate that many of the constructs designed to be assessed by the HSAA do in fact seem to be being assessed. Certainly, refinement of items would strengthen these constructs. That is the recommendation made here; that is, to examine items’ characteristics and factor analytic results with the intent of identifying items not performing as efficiently as expected.

HSAA Teacher Survey This section presents findings from a survey given to teachers during the administration of the HSAA. Evidence provided here supports consequential validity issues. Of the 316 teachers who participated in the HSAA, approximately 190 completed this survey.


Survey response categories ranged from 1-Strongly Disagree to 5-Strongly Agree. Low mean survey values indicate more negative responses while high values are positive. The highest rater survey prompt was Item 5. Respondents seem to most strongly agree that the HSAA was well aligned to the state’s academic standards. The alignment study presented earlier in this report supports this view. The lowest rated item on this survey is Item 3. One of the prevailing issues with the HSAA from its inception is the items and resources needed to administer the assessment. Clearly, respondents feel that insufficient time is given to administer the assessment. The next lowest rating was Item 1. Here we see that respondents also felt there were issues with the ease of administration. Table 52: HSAA Teacher Survey Summary Statistics Item Survey Prompt Mean Median SD N Min Max

1 The HSAA was easy to use. 2.80 3 1.24 190 1 5

2 The HSAA appears to facilitate the participation in the state’s assessment system of students who historically would have been left out.

3.73 4 1.30 186 1 5

3 The amount of time needed to administer the HSAA is reasonable. 2.33 2 1.35 193 1 5

4 The results of the HSAA would be useful to me and others who make instructional plans for students with significant disabilities.

3.66 4 1.29 192 1 5

5 The HSAA items seem well aligned with the state’s general education academic standards. 4.16 4 0.99 192 1 5

6 The HSAA items seem well aligned with the curriculum and instruction typically provided to students with disabilities.

3.33 4 1.30 190 1 5

7 By conducting an alternate assessment, I learned more about Hawaii’s academic standards and statewide assessment system.

3.75 4 1.20 191 1 5

8 The results of the HSAA that I conducted appeared to be statistically sound. 3.92 4 1.04 189 1 5

9 The student results appeared to be an accurate representation of the student’s skills that were measured.

3.99 4 1.16 193 1 5

10 The Tasks/ Evidence Collection Forms are helpful. 3.97 4 1.13 192 1 5

11 The scores and performance level information resulting from the HSAA appear to be meaningful.

3.49 4 1.30 191 1 5

12 The results of the HSAA would be useful for developing IEP strands, standards and topics. 3.85 4 1.29 190 1 5

Many positive findings are seen in this survey. Teachers indicated that participating in the process was helpful (Item 10). They felt that the process was statistically sound and an accurate representation of their students’ skills (Items 8 and 9). And teachers felt that results from the HSAA would be useful in developing future IEP goals (Item 12). Table 53 categorizes teachers’ written responses into six categories. The most prevalent suggestions were about the time needed to administer this assessment. Some had other suggestions to improve the process, and some were frustrated with the process or felt that the process was inappropriate for their students.


Table 53: Open Ended Response Categories for HSAA Teacher Survey Open Ended Response Categories Number of Responses

Frustrated with process 23 Positive comment 12 Comment about time needed 47 Suggestion other than time 33 Process improved over previous year 4 Process not appropriate for students with severe disabilities 15 Some common themes seen from results of this survey are as follows: 1) it is time consuming, 2) it is a complex process, 3) it does provide good information, 4) it aligns with state standards, and 5) it positively supports the process of assessing students with special needs.

HSAA Parent Survey This section presents findings from a survey given to parents after the test administration. Evidence provided here supports consequential validity issues. Of the 368 students who participated in the HSAA, only 29 completed this survey. Survey response categories ranged from 1-Strongly Disagree to 5-Strongly Agree. Low mean survey values indicate more negative responses while high values are positive. The respondents generally believed that the time spent by teachers conducting the alternate assessment was important to their child. In general, the parents also felt that it was good that all students in the state were able to participate in an assessment that focused on achievement areas such as reading and mathematics and they were pleased to know that the assessment was aligned with the state’s academic standards. Table 54: HSAA Parent Survey Summary Statistics Item Survey Prompt Mean Median SD N Min Max

1 I found the alternate assessment to be useful. 3.62 4 1.35 29 1 5

2 I was pleased to know that the assessment was aligned with the state’s academic standards.

3.90 4 0.86 29 3 5

3

I think it is good that all students in the state participate in an assessment that focuses on their achievement in basic areas such as reading and math.

4.21 5 1.05 29 1 5

4 I was confident in the results about my child’s functioning that teachers provided me. 3.79 4 1.26 29 1 5

5 I believe the time spent by teachers conducting an alternate assessment is important to their teaching of my child.

4.38 5 0.82 29 2 5

Parents also had an opportunity to make open comments. However, only eight parents chose to make any comments. The comments varied widely, and there were not any comments that generally conveyed the same message. Therefore, the open comments are not summarized here as they were for the teacher’s comments.


Item Importance Ratings Tables 55 through 59 display the item importance ratings for each item on the HSAA. A value of 0 means that that item is not important and a value of 3 is very important. Thus, higher item mean values indicate more importance from raters’ perspectives. In each table the top 5 rated items are colored in yellow and the bottom 5 are colored in pink. When looking at item revision possibilities, these tables provide another evidence of teachers’ perceptions of item importance. One interesting note is that many of the lowest rated items are at the end of each test form. Conceptually, Data Analysis (the last set of items in Mathematics) and Literary Response and Analysis (the last set of items in Reading) do seem more challenging. It could also be that since these items are at the end of the rating form, scoring fatigue by raters may be setting in. It may be interesting to move strands around in the rating form to see if importance ratings change. Table 55: HSAA Reading Elementary Items Importance Ratings

Item # MEAN STD MIN MAX N 1 2.172 0.859 0 3 186 2 2.156 0.853 0 3 186 3 1.672 1.068 0 3 186 4 1.677 1.077 0 3 186 5 1.941 1.081 0 3 186 6 1.855 1.146 0 3 186 7 1.366 0.973 0 3 186 8 1.78 0.93 0 3 186 9 1.839 0.995 0 3 186 10 1.71 1.004 0 3 186 11 1.753 0.977 0 3 186 12 1.543 0.907 0 3 186 13 1.962 0.853 0 3 186 14 1.263 0.919 0 3 186 15 1.301 0.922 0 3 186 16 1.435 0.986 0 3 186 17 1.71 1.004 0 3 186 18 1.349 0.965 0 3 186 19 1.629 0.905 0 3 186 20 1.28 0.868 0 3 186 21 1.183 0.924 0 3 186 22 1.075 0.828 0 3 186 23 1.349 0.993 0 3 186 24 1.285 0.912 0 3 186 25 1.489 0.954 0 3 186 26 1.398 0.914 0 3 186 27 1.247 0.96 0 3 186 28 1.081 0.894 0 3 186 29 0.978 0.936 0 3 186 30 1.242 0.953 0 3 186 31 0.952 0.853 0 3 186 32 1.038 0.897 0 3 186 33 1.333 0.916 0 3 186 34 0.995 0.848 0 3 186 35 1.414 0.933 0 3 186 36 1.419 0.91 0 3 186 37 1.317 0.931 0 3 186


Table 55: HSAA Reading Elementary Items Importance Ratings Item # MEAN STD MIN MAX N

38 0.952 0.902 0 3 186 39 0.78 0.792 0 3 186 40 0.887 0.902 0 3 186

Grand Total 2.172 1.146 0 3 186 Table 56: HSAA Reading Secondary Item Importance Ratings

Item # MEAN STD MIN MAX N 1 1.258 0.943 0 3 182 2 1.192 0.976 0 3 182 3 1.588 0.997 0 3 182 4 1.907 1.06 0 3 182 5 1.703 1.102 0 3 182 6 1.89 0.91 0 3 182 7 1.791 0.929 0 3 182 8 1.725 0.958 0 3 182 9 1.319 0.945 0 3 182 10 0.857 0.815 0 3 182 11 1.203 0.939 0 3 182 12 1.033 0.904 0 3 182 13 1.67 0.97 0 3 182 14 1.736 0.92 0 3 182 15 1.324 0.841 0 3 182 16 1.066 0.902 0 3 182 17 1.214 0.894 0 3 182 18 1.115 0.906 0 3 182 19 1.165 0.914 0 3 182 20 1.192 0.893 0 3 182 21 1.231 0.893 0 3 182 22 1.258 0.895 0 3 182 23 1.011 0.794 0 3 182 24 1.225 0.945 0 3 182 25 1.022 0.834 0 3 182 26 1.06 0.849 0 3 182 27 1.159 0.899 0 3 182 28 0.912 0.856 0 3 182 29 0.676 0.743 0 3 182 30 1.242 0.908 0 3 182 31 0.967 0.779 0 3 182 32 1.423 0.906 0 3 182 33 0.56 0.643 0 2 182 34 0.813 0.735 0 3 182 35 0.912 0.93 0 3 182 36 0.703 0.772 0 3 182 37 1.291 0.915 0 3 182 38 0.945 0.852 0 3 182 39 0.929 0.854 0 3 182 40 0.676 0.743 0 3 182

Grand Total 1.907 1.102 0 3 182


Table 57: HSAA Mathematics Elementary Items Importance Ratings

Item # MEAN STD MIN MAX N 1 1.355 1.087 0 3 186 2 1.382 1.002 0 3 186 3 1.237 0.911 0 3 186 4 1.022 0.894 0 3 186 5 1.93 1.003 0 3 186 6 1.78 1.075 0 3 186 7 1.758 1.115 0 3 186 8 1.199 0.905 0 3 186 9 0.876 0.852 0 3 186 10 1.28 0.968 0 3 186 11 1.532 1.066 0 3 186 12 1.366 1.016 0 3 186 13 1.36 1.016 0 3 186 14 1.188 1.072 0 3 186 15 1.14 0.931 0 3 186 16 1.022 0.851 0 3 186 17 1.118 0.996 0 3 186 18 1.484 1.004 0 3 186 19 0.785 0.856 0 3 186 20 1.769 1.108 0 3 186 21 1.683 0.925 0 3 186 22 1.468 0.982 0 3 186 23 1.22 0.958 0 3 186 24 1.237 1.007 0 3 186 25 2.016 0.978 0 3 186 26 1.747 1.037 0 3 186 27 2.124 0.806 0 3 186 28 0.968 0.906 0 3 186 29 0.5 0.676 0 2 186 30 1.21 0.95 0 3 186 31 0.817 0.85 0 3 186 32 1.28 0.963 0 3 186 33 1.452 0.913 0 3 186 34 1.269 0.896 0 3 186 35 0.978 0.825 0 3 186 36 0.694 0.719 0 2 186 37 1.366 0.962 0 3 186 38 1.548 1.061 0 3 186 39 0.785 0.849 0 3 186 40 0.753 0.853 0 3 186 41 0.57 0.77 0 3 186 42 0.828 0.833 0 3 186 43 1.435 0.975 0 3 186 44 0.769 0.761 0 3 186 45 0.823 0.809 0 3 186 46 0.5 0.651 0 2 186


Table 57: HSAA Mathematics Elementary Items Importance Ratings Item # MEAN STD MIN MAX N

47 0.5 0.668 0 2 186 48 0.484 0.722 0 3 186 49 0.688 0.742 0 3 186 50 1.312 0.906 0 3 186 51 1.355 0.908 0 3 186 52 1.253 0.873 0 3 186 53 1.349 0.919 0 3 186 54 1.376 1.002 0 3 186 55 1.538 0.976 0 3 186 56 1.22 0.912 0 3 186 57 1.484 0.965 0 3 186 58 1.22 0.97 0 3 186 59 1.312 0.953 0 3 186 60 1.07 1.003 0 3 186 61 0.995 0.95 0 3 186 62 2.091 1.059 0 3 186 63 1.403 0.909 0 3 186 64 1.28 0.951 0 3 186 65 0.839 0.836 0 3 186 66 0.742 0.837 0 3 186 67 1.134 0.98 0 3 186 68 1.484 0.89 0 3 186 69 0.726 0.867 0 3 186 70 1.199 0.881 0 3 186 71 1.344 0.894 0 3 186 72 1.14 0.925 0 3 186 73 0.892 0.825 0 3 186 74 0.839 0.775 0 3 186 75 0.817 0.811 0 3 186

Grand Total 2.124 1.115 0 3 186 Table 58: HSAA Mathematics Secondary Items Importance Ratings

Item # MEAN STD MIN MAX N 1 1.956 0.957 0 3 182 2 1.044 0.915 0 3 182 3 2.022 1.035 0 3 182 4 1.242 1.023 0 3 182 5 1.269 1.019 0 3 182 6 1.802 1.016 0 3 182 7 1.692 0.994 0 3 182 8 1.401 0.985 0 3 182 9 1.39 0.926 0 3 182 10 1.341 0.901 0 3 182 11 0.764 0.79 0 3 182 12 0.758 0.791 0 3 182 13 1.665 1.104 0 3 182 14 1.456 0.902 0 3 182 15 0.846 0.806 0 3 182


Table 58: HSAA Mathematics Secondary Items Importance Ratings Item # MEAN STD MIN MAX N

16 0.582 0.706 0 2 182 17 0.637 0.736 0 2 182 18 0.473 0.679 0 2 182 19 0.769 0.774 0 3 182 20 1 0.958 0 3 182 21 1.192 0.923 0 3 182 22 0.989 0.821 0 3 182 23 1.154 0.82 0 3 182 24 1.654 1.065 0 3 182 25 1.11 0.922 0 3 182 26 0.824 0.774 0 3 182 27 0.742 0.709 0 3 182 28 1.247 0.873 0 3 182 29 1.198 0.831 0 3 182 30 1.165 0.804 0 3 182

Table 59: HSAA Mathematics Secondary Items Importance Ratings

Item # MEAN STD MIN MAX N 31 0.566 0.676 0 3 182 32 1.555 0.937 0 3 182 33 0.791 0.801 0 3 182 34 0.593 0.681 0 3 182 35 0.516 0.601 0 2 182 36 0.484 0.663 0 2 182 37 1.099 0.78 0 3 182 38 1.17 0.827 0 3 182 39 1.033 0.898 0 3 182 40 0.764 0.761 0 3 182 41 0.984 0.739 0 3 182 42 0.852 0.717 0 3 182 43 1.258 0.844 0 3 182 44 1.275 0.842 0 3 182 45 1.154 0.846 0 3 182 46 1.357 1.002 0 3 182 47 1.555 0.907 0 3 182 48 0.989 0.886 0 3 182 49 1.066 0.883 0 3 182 50 1.33 0.941 0 3 182 51 1.22 0.938 0 3 182 52 1.379 0.876 0 3 182 53 1.11 0.904 0 3 182 54 1.407 0.922 0 3 182 55 1.357 1.087 0 3 182 56 0.692 0.81 0 3 182 57 0.874 0.992 0 3 182 58 1.379 1.074 0 3 182 59 0.527 0.687 0 2 182 60 0.714 0.777 0 3 182


Table 59: HSAA Mathematics Secondary Items Importance Ratings Item # MEAN STD MIN MAX N

61 1.132 0.919 0 3 182 62 1.28 0.982 0 3 182 63 0.835 0.804 0 3 182 64 0.83 0.84 0 3 182 65 1.489 0.956 0 3 182 66 1.253 0.899 0 3 182 67 0.5 0.671 0 2 182 68 0.648 0.763 0 3 182 69 0.879 0.865 0 3 182 70 0.61 0.741 0 3 182 71 0.813 0.813 0 3 182 72 0.775 0.8 0 3 182 73 0.791 0.848 0 3 182 74 0.533 0.636 0 2 182 75 0.489 0.592 0 2 182

Grand Total 2.022 1.104 0 3 182

IEP Alignment Ratings Tables 60-63 indicate the number and percentage of students who had IEP goals that were aligned to each item. Items that had at least 10% of the students’ IEP goals aligned to that item are highlighted in yellow. Table 60: HSAA Reading Elementary IEP Alignment

item COUNT Percent 01 29 15.6% 02 29 15.6% 03 8 4.3% 04 16 8.6% 05 43 23.1% 06 35 18.8% 07 10 5.4% 08 36 19.4% 09 55 29.6% 10 11 5.9% 11 19 10.2% 12 20 10.8% 13 41 22.0% 14 10 5.4% 15 13 7.0% 16 17 9.1% 17 23 12.4% 18 17 9.1% 19 24 12.9% 20 9 4.8% 21 13 7.0% 22 1 0.5% 23 18 9.7% 24 2 1.1%


Table 60: HSAA Reading Elementary IEP Alignmentitem COUNT Percent 25 14 7.5% 26 15 8.1% 27 13 7.0% 28 15 8.1% 29 3 1.6% 30 23 12.4% 31 3 1.6% 32 4 2.2% 33 11 5.9% 34 4 2.2% 35 4 2.2% 36 11 5.9% 37 13 7.0% 38 4 2.2% 39 1 0.5% 40 3 1.6%

Table 61: HSAA Reading Secondary IEP Alignment

item COUNT Percent 01 11 6.0% 02 6 3.3% 03 22 12.1% 04 35 19.2% 05 33 18.1% 06 63 34.6% 07 78 42.9% 08 28 15.4% 09 25 13.7% 10 6 3.3% 11 12 6.6% 12 12 6.6% 13 47 25.8% 14 56 30.8% 15 22 12.1% 16 5 2.7% 17 23 12.6% 18 7 3.8% 19 15 8.2% 20 19 10.4% 21 31 17.0% 22 18 9.9% 23 9 4.9% 24 28 15.4% 25 10 5.5% 26 26 14.3% 27 45 24.7%


Table 61: HSAA Reading Secondary IEP Alignmentitem COUNT Percent 28 9 4.9% 29 6 3.3% 30 17 9.3% 31 7 3.8% 32 27 14.8% 33 3 1.6% 34 3 1.6% 35 16 8.8% 36 5 2.7% 37 17 9.3% 38 10 5.5% 39 10 5.5% 40 1 0.5%

Table 62: HSAA Mathematics Elementary IEP Alignment

item COUNT Percent 01 14 7.5% 02 6 3.2% 03 7 3.8% 04 3 1.6% 05 75 40.3% 06 25 13.4% 07 37 19.9% 08 4 2.2% 09 1 0.5% 10 6 3.2% 11 5 2.7% 12 15 8.1% 13 8 4.3% 14 17 9.1% 15 14 7.5% 16 2 1.1% 17 11 5.9% 18 9 4.8% 19 0 0.0% 20 47 25.3% 21 15 8.1% 22 9 4.8% 23 3 1.6% 24 8 4.3% 25 4 2.2% 26 16 8.6% 27 19 10.2% 28 6 3.2% 29 0 0.0% 30 5 2.7%


Table 62: HSAA Mathematics Elementary IEP Alignmentitem COUNT Percent 31 1 0.5% 32 17 9.1% 33 22 11.8% 34 16 8.6% 35 5 2.7% 36 3 1.6% 37 8 4.3% 38 36 19.4% 39 10 5.4% 40 8 4.3% 41 7 3.8% 42 4 2.2% 43 6 3.2% 44 2 1.1% 45 3 1.6% 46 2 1.1% 47 1 0.5% 48 0 0.0% 49 1 0.5% 50 12 6.5% 51 20 10.8% 52 16 8.6% 53 7 3.8% 54 8 4.3% 55 7 3.8% 56 6 3.2% 57 23 12.4% 58 7 3.8% 59 7 3.8% 60 5 2.7% 61 3 1.6% 62 41 22.0% 63 12 6.5% 64 17 9.1% 65 5 2.7% 66 5 2.7% 67 5 2.7% 68 12 6.5% 69 4 2.2% 70 17 9.1% 71 29 15.6% 72 3 1.6% 73 1 0.5% 74 2 1.1% 75 5 2.7%


Table 63: HSAA Mathematics Secondary IEP Alignment

item COUNT Percent 01 55 30.2% 02 2 1.1% 03 38 20.9% 04 11 6.0% 05 7 3.8% 06 31 17.0% 07 12 6.6% 08 17 9.3% 09 14 7.7% 10 7 3.8% 11 2 1.1% 12 2 1.1% 13 68 37.4% 14 12 6.6% 15 6 3.3% 16 8 4.4% 17 7 3.8% 18 2 1.1% 19 8 4.4% 20 15 8.2% 21 9 4.9% 22 3 1.6% 23 4 2.2% 24 50 27.5% 25 3 1.6% 26 7 3.8% 27 3 1.6% 28 17 9.3% 29 13 7.1% 30 7 3.8% 31 3 1.6% 32 19 10.4% 33 0 0.0% 34 0 0.0% 35 0 0.0% 36 4 2.2% 37 6 3.3% 38 8 4.4% 39 6 3.3% 40 4 2.2% 41 10 5.5% 42 6 3.3% 43 13 7.1% 44 15 8.2% 45 9 4.9% 46 13 7.1%


Table 63: HSAA Mathematics Secondary IEP Alignmentitem COUNT Percent 47 14 7.7% 48 6 3.3% 49 6 3.3% 50 4 2.2% 51 16 8.8% 52 7 3.8% 53 10 5.5% 54 17 9.3% 55 17 9.3% 56 0 0.0% 57 5 2.7% 58 16 8.8% 59 1 0.5% 60 0 0.0% 61 22 12.1% 62 27 14.8% 63 7 3.8% 64 5 2.7% 65 15 8.2% 66 11 6.0% 67 0 0.0% 68 5 2.7% 69 4 2.2% 70 1 0.5% 71 11 6.0% 72 4 2.2% 73 7 3.8% 74 0 0.0% 75 0 0.0%


Appendix A: Alignment Materials

Expanded Depth of Knowledge by Subject Descriptors for Special Education Standards/Assessments

Webb DOK Levels Level 1—Recall of Information

Level 2—Basic Reasoning

Level 3—Complex Reasoning

Level 4—Extended Reasoning

SPED DOK Stages Stage 1-Respond Stage 2-Reproduce Stage 3-Recall Stage 4 Stage 5 Stage 6

Engl

ish

Lang

uage

Art

s

Requires the ability to respond to, indicate, or acknowledge text or discourse related features. Example: ♦ Points to the

letters/words/ pictures on a page

♦ Points to letters or words

♦ Acknowledges a discourse interaction with an interlocutor

♦ Responds to a conversation

♦ Acknowledges someone signing

♦ Attends to text

Requires the ability to copy, replicate, repeat, re-enact, mirror, or match text or discourse related features. Example: ♦ Copies letters ♦ Reproduces

letters, text or words either verbally or through writing/signing

♦ Matches sound/sound

♦ Matches letter/letter

♦ Matches words ♦ Re-enacts a story

or interaction either verbally or through text (also drawing)

♦ Matches picture/picture

♦ Matches symbol/symbol

Requires the ability to recite or recall facts or information. Involves the ability to distinguish between text-based or discourse features. Example: ♦ Identifies pictures

of objects (animate or inanimate) though verbal cues or text-based cues

♦ Identifies details in text

♦ Identifies correct spelling or meaning of words

♦ Identifies letters ♦ Identifies sounds

Requires processing beyond recall and observation. Requires both comprehension and subsequent processing of text. Involves ordering, classifying text as well as identifying patterns, relationships and main points. Examples: ♦ Uses context to

identify unfamiliar words

♦ Predicts logical outcome

♦ Identifies and summarizes main points

♦ Associates/identifies letters with sounds

♦ Identifies figurative language

♦ Uses dictionary ♦ Indicates what

comes next in a story

Requires students to go beyond text. Requires students to explain, generalize and connect ideas. Involves inferencing, prediction, elaboration and summary. Requires students to support positions using prior knowledge and to manipulate themes across passages. Examples: ♦ Determines effect

of author’s purpose on text elements

♦ Summarizes information from multiple sources

♦ Critically analyzes literature/text

♦ Expresses an opinion about text, citing evidence to support reasoning

Requires extended higher order processing. Typically requires extended time to complete task, but time spent not on repetitive tasks. Involves taking information from one text/passage and applying this information to a new task. May require generating hypotheses and performing complex analyses and connections among texts. Examples: ♦ Analyzes and

synthesizes information from multiple sources

♦ Examines and explains alternative perspectives across sources

♦ Describes and illustrates common themes across a variety of texts

♦ Creates compositions that synthesize, analyze, and evaluate


Expanded Depth of Knowledge by Subject Descriptors for Special Education Standards/Assessments

Webb DOK Levels Level 1—Recall of Information

Level 2—Basic Reasoning

Level 3—Complex Reasoning

Level 4—Extended Reasoning

SPED DOK Stages Stage 1-Respond Stage 2-Reproduce Stage 3-Recall Stage 4 Stage 5 Stage 6

Mat

hem

atic

s

Requires the ability to respond to, indicate or acknowledge mathematical features. Example: ♦ Points to a number ♦ Attends to

someone counting ♦ Indicates a

measuring device, i.e., ruler, measuring cup, scale

♦ Points to common shapes

Requires the ability to copy, replicate, repeat, re-enact, mirror, or match mathematical features. Example: ♦ Copies numbers

from 1-10 ♦ Counts from 1-10

with prompting ♦ Reproduces a

pattern, shape or figure, e.g., circle, square, triangle, rectangle

♦ Matches even numbers/even numbers

♦ Matches odd numbers/odd numbers

♦ Matches similar shapes together

Requires students to recall or observe facts, definitions, terms. Involves simple one-step procedures. Involves computing simple algorithms (e.g., sum, quotient). Examples: ♦ Recalls or

recognizes a fact, term or property

♦ Conducts simple computation using manipulatives

♦ Represents in words, pictures or symbols in a math object or relationship

♦ Performs routine procedure like measuring

♦ Points to a number and the physical representation of that number (e.g., 2, two balls)

♦ Identifies common shapes and figures

♦ Identifies measuring devices, i.e., ruler, measuring cup, scale

Requires students to make decisions of how to approach a problem. Requires students to compare, classify, organize, estimate or order data. Typically involves two-step procedures. Examples: ♦ Specifies and

explains relationships between facts, terms, properties or operations

♦ Selects procedure according to criteria and performs it

♦ Solves routine multiple-step problems

♦ Recognizes the pattern in a series of objects/numbers

♦ Organizes fractional items from big to small

♦ Compares different attributes of data (tells which has more/less/same)

Requires reasoning, planning or use of evidence to solve problem or algorithm. May involve activity with more than one possible answer. Requires conjecture or restructuring of problems. Involves drawing conclusions from observations, citing evidence and developing logical arguments for concepts. Uses concepts to solve non-routine problems. Examples: ♦ Analyzes

similarities and differences between procedures

♦ Formulates original problem given situation

♦ Formulates mathematical model for complex situation

Requires complex reasoning, planning, developing and thinking. Typically requires extended time to complete problem, but time spent not on repet-itive tasks. Requires students to make several connections and apply one app-roach among many to solve the problem. Involves complex restructuring of data, establishing and evaluating criteria to solve problems. Examples: ♦ Applies math-

ematical model to illuminate a problem, situation

♦ Conducts a project that specifies a problem, ident-ifies solution paths, solves the problem, and reports results

♦ Designs a mathematical model to inform and solve a practical or abstract situation


Depth of Knowledge by Subject Descriptors—HI API and HCPS III Alignment

Subject Depth of Knowledge Levels

Level 1 Level 2 Level 3 Level 4

Engl

ish

Lang

uage

Art

s

Requires students to recall, observe, question or represent facts, simple skills or abilities. Requires only surface understanding of text; often verbatim recall. Examples: ♦ Support ideas by reference to

details in text ♦ Use dictionary to find meaning ♦ Identify figurative language in

passage ♦ Identify correct spelling or

meaning of words

Requires processing beyond recall and observation. Requires both comprehension and subsequent processing of text. Involves ordering, classifying text as well as identifying patterns, relationships and main points. Examples: ♦ Use context to identify

unfamiliar words ♦ Predict logical outcome ♦ Identify and summarize main

points ♦ Apply knowledge of

conventions of standard American English

♦ Compose accurate summaries

Requires students to go beyond text. Requires students to explain, generalize and connect ideas. Involves inferencing, prediction, elaboration and summary. Requires students to support positions using prior knowledge and to manipulate themes across passages. Examples: ♦ Determine effect of author’s

purpose on text elements ♦ Summarize information from

multiple sources ♦ Critically analyze literature ♦ Compose focused,

organized, coherent, purposeful prose

Requires extended higher order processing. Typically requires extended time to complete task, but time spent not on repetitive tasks. Involves taking information from one text/passage and applying this information to a new task. May require generating hypotheses and performing complex analyses and connections among texts. Examples: ♦ Analyze and synthesize

information from multiple sources ♦ Examine and explain alternative

perspectives across sources ♦ Describe and illustrate common

themes across a variety of texts ♦ Create compositions that

synthesize, analyze, and evaluate

Mat

hem

atic

s

Requires students to recall or observe facts, definitions, terms. Involves simple one-step procedures. Involves computing simple algorithms (e.g., sum, quotient). Examples: ♦ Recall or recognize a fact, term

or property ♦ Represent in words, pictures or

symbols in a math object or relationship

♦ Perform routine procedure like measuring

Requires students to make decisions of how to approach a problem. Requires students to compare, classify, organize, estimate or order data. Typically involves two-step procedures. Examples: ♦ Specify and explain

relationships between facts, terms, properties or operations

♦ Select procedure according to criteria and perform it

♦ Solve routine multiple-step problems

Requires reasoning, planning or use of evidence to solve problem or algorithm. May involve activity with more than one possible answer. Requires conjecture or restructuring of problems. Involves drawing conclusions from observations, citing evidence and developing logical arguments for concepts. Uses concepts to solve non-routine problems. Examples: ♦ Analyze similarities and

differences between procedures

♦ Formulate original problem given situation

♦ Formulate mathematical model for complex situation

Requires complex reasoning, planning, developing and thinking. Typically requires extended time to complete problem, but time spent not on repetitive tasks. Requires students to make several connections and apply one approach among many to solve the problem. Involves complex restructuring of data, establishing and evaluating criteria to solve problems. Examples: ♦ Apply mathematical model to

illuminate a problem, situation ♦ Conduct a project that specifies a

problem, identifies solution paths, solves the problem, and reports results

♦ Design a mathematical model to inform and solve a practical or abstract situation

ENGLISH LANGUAGE ARTS EXAMPLES SPECIAL EDUCATION ASSESSMENT ALIGNMENT

DEPTH OF KNOWLEDGE LEVELS AND STAGES Level 1—Stage 1: Respond to Discourse Materials Level 1—Stage 1 requires students to display the ability to respond to, indicate, or acknowledge text or discourse related features Examples of Standards 3rd Grade Reading Objective: Respond to a symbol system (graphic or braille). 7th Grade Literature Objectives: Acknowledge potential communication partner, listen attentively. General Reading Standard: Understand that written words represent people, objects, or actions. These standards represent Stage 1 because students are tasked to respond to symbols, words or communication, and nothing more. Examples of Items (APIs) Student demonstrates the ability to attend to pictures/symbols/objects pertinent to a story. Student displays attention to people, surroundings or materials. Student attends while teacher reads. These items display Stage 1 characteristics because students are asked to reply or respond to texts or individuals. Level 1—Stage 2: Reproduce Discourse Related Materials Level 1—Stage 2 requires students to display the ability to copy, replicate, repeat, re-enact, mirror, or match text or discourse related features. Examples of Standards 3rd Grade Reading Objective: Discriminate similarities and/or differences. 8th Grade Reading Objective: Follow a schedule. General Reading Standard: Demonstrate understanding that written words represent people, objects, or actions. These standards reflect Stage 2 since students reproduce or follow directive discourse activities.


Examples of Items (APIs) Student can match pictures that depict emotions such as happy, sad, or angry. Student matches pictures with printed words to actual objects. Student matches letter to letter and reproduces sounds of letters. These items are Stage 2 because students are doing more than just acknowledging discourse; they are matching or reproducing discourse activities. Level 1—Stage 3: Recall Information about Discourse Related Materials Level 1—Stage 3 requires the ability to recite or recall facts or information. Involves the ability to distinguish between text-based or discourse features. Examples of Standards 7th Grade Literature Objective: Follow a story sequence. General Reading Standard: Respond to literal questions (Who?, What?, Where?, When?) about reading material. General Reading Standard: Sequence events from a story. These standards are Stage 3 because students recall or recite events of discourse which they have heard or read. Examples of Items (APIs) Student demonstrates understanding of new words or recalls basic ideas in passages via speech, writing, signs, or assistive device. Student can retell information taken from printed materials. Student can answer who, what, and where questions about a story. The above items ask students to recall discourse and/or discourse related materials to which they have been exposed. Level 2—Stage 4: Basic Reasoning Level 2—Stage 4 requires processing beyond recall and observation. Requires both comprehension and subsequent processing of text. Involves ordering, classifying text as well as identifying patterns, relationships and main points.


Examples of Standards Edit final copies for correct use of language, spelling, punctuation, and capitalization. Understand main idea and purpose implied by text. These standards expand beyond recall. Students must move beyond the discourse (text) to which they are exposed and do something with the information they have received. Examples of Items (APIs) Student identifies the main idea or message in the text. The student must not only read the text but understand and summarize the text for the main idea or message. Here is a paragraph from a student’s report about Maya Angelou. The paragraph has six mistakes in grammar, punctuation, and capitalization. Draw a line through each part that has a mistake, and write the correction above it. I Know Why the Caged Bird Sings is the first of five volumes in Maya Angelou’s autobiography. This book is about Mayas childhood in the South during the 1930s. Her autobiography also cover her life as an adult in San Francisco, where she had many extraordinary experiences. She work as a streetcar conductor and trained to become a singer and a dancer. She toured europe and North America in the musical Porgy and Bess. The end of her autobiography deals with Angelou’s career as a civil rights activist, and exploring the relationship between African Americans and Africa. Which of these best describes what the passage is about? a. how a young man started a successful business after coming to America b. why blue jeans are superior to pants made by other manufacturers c. how clothing styles were affected by the work of the gold miners d. why immigrants came to America to improve their lives These items have students move beyond the text (discourse). In the above examples, students read, process and do something with what they’ve read. In the first example, students must edit or correct mistakes. In the second example, student must not only read the text but understand and summarize the text. Level 3—Stage 5: Complex Reasoning Level 3—Stage 5 requires students to go beyond text. Requires students to explain, generalize and connect ideas. Involves inferencing, prediction, elaboration and summary. Requires students to support positions using prior knowledge and to manipulate themes across passages. Examples of Standards Evaluate the relative accuracy and usefulness of information from different sources.


Understand the relationships between and among elements of literature, including characters, plot, setting, tone, figurative language, point of view, and theme. These standards illustrate Level 3—Stage 5. Students must understand a variety of kinds of texts and text features, make inferences across passages and demonstrate the ability to evaluate information according to various criteria. Examples of Items (APIs) Student predicts what will happen in a story and cite evidence to support the inference using both prior knowledge and information from the text. A series of more complex tasks are being elicited for this task. The student is required to consider the text as a whole and use proper knowledge and evidence from the text to make an inference that allows a probable prediction of the outcome. Which of these conclusions is best supported by information from the passage? a. If a candidate meets the personal and educational qualifications and is in fair physical shape, his or

her chances of becoming an agent are very good. b. Compared with other law enforcement agencies in the country, the F.B.I. has a low success rate for

tracking down and apprehending suspected offenders. c. The job of an agent is not for everyone; it takes someone with special training who is not afraid of

danger and doesn’t mind being socially isolated at times. d. The life of a federal investigator is not as interesting as most people think; agents spend most of their

time working at desks. The author’s style is characterized by similes like these found in text: “bitter as a penny,” “flapped the jacket like a bird’s wings,” “my forearms stuck out like the necks of

turtles.” Which of these best explains the effect of these similes on the author’s writing? a. They create a formal tone. b. They create a soft sound. c. They create unsentimental images. d. They create a flowery, ornamental style. Although texts are not provided, it is clear that more complex tasks are being elicited. In the first item, students are required to consider an article as a whole, reflecting on emphasis, author’s purpose, tone, etc. The second item requires students to think abstractly about connotative dimensions of language to reach a conclusion about the figures of speech habitually chosen by an author. Both items are examples of Level 3—Stage 5 tasks.


Level 4—Stage 6: Extended Reasoning Level 4—Stage 6 requires extended higher order processing. Typically requires extended time to complete task, but time spent not on repetitive tasks. Involves taking information from one text/passage and applying this information to a new task. May require generating hypotheses and performing complex analyses and connections among texts. Examples of Standards Locate, gather, analyze and evaluate written information for the purpose of drafting a reasoned report that supports and appropriately illustrates inferences and conclusions drawn from research. Generate and edit media work as appropriate to audience and purpose. Both tasks require students to perform a variety of higher order tasks. The first standard is an extended activity requiring students to gather, read, and evaluate materials; select and record information according to a specific purpose; and generate an original composition shaped for a specific audience and purpose. The second standard requires students to perform a variety of higher order tasks similar to those above; in addition, this task requires students to manage the necessary technology required to create a media piece. Examples of Items You will become a storyteller. You will research and write the story of someone who has immigrated to the United States and/or migrated within the United States. You will get a role card from your native country, and you will become that person. The role cards feature many countries and many time periods: gender and age are mixed. For example:

Moira Adair, 50, arriving from Northern Ireland in 1980. Your husband was killed in an IRA bombing. You are a computer expert and have family in Minneapolis. Sean Dolan, 21, arriving from Ireland in 1853. You are alone but you have a relative in New York. You are an apprentice stone mason.

Students must produce an original map showing their home country as it was when they left. They describe the culture (social, economic, political, dominant religious affiliation, educational system, legal system), including the dominant values, customs, and traditions of the culture. Further, they note specific problems in their homeland, explaining why people emigrate to America at that time. The trip to America is the bridge to researching settlement in a specific area or community; this is where imagination takes over for a time, although students will also need to maintain accuracy. The next major research involves the assimilation process in America. Additionally, students need to research the contributions of their ethnic group to America. To guide them through this project, students receive a packet of materials that includes everything from graphic organizers to specific prompts. The project culminates in an Ellis Island simulation and a “feast” for which students research and prepare food, music, and dance from their assigned homeland. This extended activity includes several assignments that would clearly represent Level 4—Stage 6 reasoning.


MATHEMATICS EXAMPLES SPECIAL EDUCATION ASSESSMENT ALIGNMENT

DEPTH OF KNOWLEDGE LEVELS AND STAGES Level 1—Stage 1: Respond to Mathematical Features Level 1—Stage 1 requires the ability to respond to, indicate or acknowledge mathematical features. Examples of Standards General Mathematics Standard: Point to a number versus other graphics (select numbers from other symbols such as letters, shapes, pictures lines). General Mathematics Standard: Interact with objects related to mathematical activities. These standards represent Stage 1 because students are tasked to respond to math symbols or visuals and nothing more. Examples of Items (APIs) Student attends to counting task. Student responds to math ideas. These items display Level 1—Stage 1 characteristics because students are asked to reply or respond to math concepts. Level 1—Stage 2: Reproduce Mathematical Features Level 1—Stage 2 requires the ability to copy, replicate, repeat, re-enact, mirror, or match mathematical features. Examples of Standards 3rd Grade Mathematics Standard: Match three-dimensional shapes and/or manipulatives. General Mathematics Standard: Match numerals to pictures representing the same number of objects. General Mathematics Standard: Match numeral to numeral. These standards reflect Level 1—Stage 2 since students reproduce or match requested math-related activities. Examples of Items (APIs) Student copies numbers accurately in a variety of contexts. Student accurately sorts basic shapes into identified groups. Student matches objects following a pattern.


These items are Stage 2 because students are doing more than just acknowledging math concepts; they are matching or reproducing them. Level 1—Stage 3: Recall Information about Mathematical Features Level 1—Stage 3 requires students to recall or observe facts, definitions, terms. Involves simple one-step procedures. Involves computing simple algorithms (e.g., sum, quotient). Examples of Standards General Mathematics Standard: Determine area and perimeter using concrete materials. 8th Grade Mathematics Standard: Tell time with some type of time-keeping device. General Mathematics Standard: Read and label three digit numbers. These standards are Stage 3 because students recall or recite math concepts which they have learned. Examples of Items (APIs) Student locates a pattern in order to solve a problem. Student measures using feet and yards. Student uses a calculator or concrete objects to add and subtract. The above items ask students to recall mathematical features to which they have been exposed. Level 2—Stage 4: Basic Reasoning Level 2—Stage 4 requires students to make decisions of how to approach a problem. Requires students to compare, classify, organize, estimate or order data. Typically involves two-step procedures. Examples of Standards 7th Grade Standard: Construct two-dimensional patterns for three-dimensional models, such as cylinders and cones. This objective is an example of Level 2—Stage 4. Although recognizing and drawing a two-dimensional pattern or a regular cylinder may be a level 1, building a representation of a three-dimensional model would not be routine. It would require at least two steps: first recognizing the shape, then drawing a two-dimensional object to reflect the shape in three-dimensions.


Examples of Items (APIs) Student gathers, sorts, and graphs data into categories to solve a problem (how many have… which has more… etc.). The item requires students to think about what data to use and how to categorize it to find the correct answers to the problem/questions. A car odometer registered 41,256.9 miles when a highway sign warned of a detour 1,200 feet ahead. What will the odometer read when the car reaches the detour? (5280 feet = 1 mile) (a) 42,456.9 (b) 41,279.9 (c) 41,261.3 (d) 41,259.2 (e) 41,257.1 If each of the counting numbers from 1 though 10 is multiplied by 13, how many of the resulting numbers will be even? (a) One (b) Four (c) Five (d) Six (e) Ten Both of these items require students to move beyond recall. The first item requires students to think about what operation(s) to use and how to apply it/them to find the correct mileage. The second requires students to use properties of multiplication and odd and even numbers. Although for some students this may be routine, the item requires students to consider more than one concept. Level 3—Stage 5: Complex Reasoning Level 3—Stage 5 requires reasoning, planning or use of evidence to solve problems or algorithms. May involve activity with more than one possible answer. Requires conjecture or restructuring of problems. Involves drawing conclusions from observations, citing evidence and developing logical arguments for concepts. Uses concepts to solve non-routine problems. Examples of Standards 8th Grade Standard: Solve two-step linear equations and inequalities in one variable over the rational numbers, interpret the solution or solutions in the context from which they arose, and verify the reasonableness of results. The expectation expressed here is that students will not only solve a two-step linear equation, but will also interpret the solution and verify the results. This will require students to do some reasoning in order to interpret the solution and could be fairly complex depending on the context. If students were only required to solve linear equations and verify solutions, then the expectation would be Level 2—Stage 4. Examples of Items (APIs) Student constructs a series of shapes using various tangram puzzle pieces. The item requires students to plan to solve the problem. This activity has more than one


right answer and requires restructuring of the problem since the student must determine which shapes will be constructed. This question refers to shapes N, P and Q. In Mr. Bell’s classes, the students voted for their favorite shape for a symbol. Here are the results.

Class 1 Class 2 Class 3 Shape N 9 14 11 Shape P 1 9 17 Shape Q 22 7 2

Using the information in the chart, Mr. Bell must select one of the shapes to be the symbol. Which one should he select and why? The shape Mr. Bell should select: ________. Explain: Sam can purchase his lunch at school. Each day he wants to have juice that costs 50¢, a sandwich that costs 90¢, and fruit that costs 35¢. His mother has only $1.00 bills. What is the least number of $1.00 bills that his mother should give him so he will have enough money to buy lunch for 5 days? The first item is for 8th graders, and the second is for 4th graders. Both items reflect more than a two-step process. To successfully solve the first problem, students must consider different ways that votes could be counted (total votes or votes by class). The student must then give a rationale for what response is selected. The student is required to reason and consider alternate solutions. Likewise, to successfully respond to the second item, 4th graders need to reason how they would develop a solution and develop a method for this solution. Both items reflect Level 3—Stage 5 activities. Level 4—Stage 6: Extended Reasoning Level 4—Stage 6 requires complex reasoning, planning, developing and thinking. Typically requires extended time to complete problem, but time spent not on repetitive tasks. Requires students to make several connections and apply one approach among many to solve the problem. Involves complex restructuring of data, establishing and evaluating criteria to solve problems. Examples of Standards 8th Grade Standard (from NEAP Math Framework): Design a statistical experiment to study a problem and communicate the outcomes. This standard requires students to plan statistical experiments. Students must define a problem (research question) and develop a procedure for solving it. This involves identifying the correct statistical model, applying the model to data, and communicating the outcome of the selected model. Students must interpret findings and make reasonable and rationed inferences from obtained data. This represents complex, multi-step reasoning and reflects a Level 4—Stage 6 task.


Examples of Items This question requires you to show your work and explain your reasoning. You may use drawings, words, and numbers in your explanation. Your answer should be clear enough so that another person could read it and understand your thinking. It is important that you show all of your work.

The data in the table above has been correctly represented by both graphs shown below. Which graph would be best to help convince others that the Metro Rail Company made a lot more money from ticket sales in March than in October? Explain your reasons for making this selection. Why might people who thought that there was little difference between October and March ticket sales consider the graph you chose to be misleading?

Month Daily Ridership

October 14,000 November 14,100 December 14,100 January 14,200 February 14,300

March14,600


This task requires students to relate a table of numbers to two graphs of the same information displayed differently. Students must then decide which of the two graphs will convince others of increased sales. Students are required to explain their answer. Following this, students are asked to take the opposite position and argue why the graph they chose is misleading. The task requires extended time to complete, involves complex restructuring of data, and requires students to oppose views of the same information. This activity is at a Level 4—Stage 6.


Appendix B: Web Alignment Tool Notes and Source of Challenge Output Reading APIs to Reading HCPSIII Standards

Grade API Comment

4

The student: Communicates an opinion about text/picture/media presentation that depicts the author’s language/style (choice of words, setting, or purpose).

This one doesn't exactly match either standard selected, but includes aspects of each.

5

The student: Sequences two events related to cause and effect in text/pictures or other media presentation.

I don't think that this item directly correlates to one of the standards, but is related to these three.

8

The student:Draws conclusions about the quality of information in the text/picture/media by the identified author.

Isn't this quite a far fetched API? Hmmm, I see that Benchmark 8.2.3 relates to author's credentials. I wonder how regular ed students determine reliability of author, author's credentials? Interesting benchmark

10

The student:Identifies a similar historical, cultural, or political personality or situation to what is represented in a text/picture/media presentation.

May be level 2 or 3 because identifying a similar historical, ..... will require student to draw upon prior knowledge, connect ideas.

HSAA Reading Elementary Items to Reading Elementary APIs

Item# Comments

2 If you define the reader as an "object," then this association holds. Otherwise, uncodeable.

16 Harder to choose DOK when task asks to both identify and use.

22 I believe that I am at odds with my group in asserting that a comparison requires a judgement.

35 simply communicate, not elaborate or explain why.


Mathematics APIs to Mathematics HCPSIII Standards Grade API Comment

3 The student:Demonstrates one-to-one correspondence. Same as item #5

4 The student:Demonstrates the concept of multiplication and division.

multiplication and division are 2 step processes

5

he student:Selects/measures/compares using standard measurement tools (e.g., ruler, tape measure, or yardstick).

added component of being able to compare

The student:Traces/follows the path to a location from a simple map/drawing and replicates map.

replicates

7

The student:Matches the number of items needed to solve a problem. not worded very clear

The student:Uses a method to measure the perimeter/area of a common shape. do they mean and sovle it?

The student:Predicts the likelihood of a given event and performs a simple experiment to test his/her prediction.

Student makes a personal decision on a particular event and its outcome. Student collects data and proves the accuracy of the experiment.

8 The student:Identifies a nonlinear line. awkward wording ... how can a "line" also be "nonlinear"?

10

The student:Identifies a nonlinear line. awkward wording ... a "line" can never be "nonlinear"

The student:Finds the solution of two lines, given the graphs. awkward wording

The student:Compares and answers questions regarding two data sets.

The standard seems appropriate but the benchmark is not in that it requires one to design a study.


HSAA Mathematics Secondary Items to Mathematics Secondary APIs

Item# Comment

11 The primary standard indicated is a bit of a stretch; it depends on the context of the question being asked. This could be linked to an Algebra item or a geometry item.

12 Again, the primary standard indicated is a bit of strectch. This item might be more appropriately linked to a "number and operations" API.





Appendix C: Standard Setting Proficiency (Performance) Level Descriptors (PLDs) Note: PLDs for Reading were determined by the expert panel to be the same for grades 3, 4, & 5 and for grades 6, 7, & 8. Grade 10 Reading PLDs were unique. All Math PLDs are specific to each grade level.


Overall Elementary Grade 3 Reading Proficiency Continuum

Well Below Proficiency (AAS)

Approaches Proficiency (AAS)

Meets Proficiency (AAS)

Exceeds Proficiency (AAS)

• Student may be able to attend

to reading instruction and may demonstrate knowledge of one or two or no basic reading concepts only with significant, extensive, and intensive prompting and support.

• Student currently exhibits

none or only one or two of the reading entry-level skills based on the Alternate Performance Indicators at an Emerging level.

• This level of functioning

occurs in a highly structured setting.

• Student attends to and

demonstrates some understanding of basic reading concepts and participates in activities with extensive support.

• Student responds or performs

several basic reading skills based on the Alternate Performance Indicators typically at the Emerging level with some at the Progressing level.


occurs in a limited number of settings.

• Student demonstrates an

understanding of basic reading concepts when provided moderate support.

• Student demonstrates the

basic reading skills based on the Alternate Performance Indicators typically at the Progressing level with some at the Mastered level.


occurs in two or more settings.

• Student demonstrates a

consistent understanding of the basic reading concepts with minimal or no support.

• Student demonstrates many

of the reading skills based on the Alternate Performance Indicators at the Progressing and Mastered levels.


occurs in multiple settings.

For example, the student: • Demonstrates very limited or

no understanding of the most basic reading concepts and skills.

• May attend to reader sporadically and only with significant, extensive, and intensive prompting and support.

For example, the student: • Matches a few words/symbols

graphics that have similar parts.

• Recognizes a few letters/words/symbols in the environment or from simple text with significant support.

• Matches words/pictures/symbols to a few basic concepts.

• Matches a few functional symbols with very basic concepts.

• Identifies the first or last event.

• Attends and responds to texts that are read to him/her.

• Locates basic information in text/story/pictures with significant support.

• Express a like or dislike of text with significant support.

For example, the student: • Matches words/symbols


• Identifies most letters & and some sounds.

• Matches words/pictures/symbols to basic concepts.

• Identifies functional and new sight vocabulary symbols/words with moderate support.

• Identifies the first and last event.

• Follows sequential steps from text/media/pictures.

• Locates basic information in text/story/pictures.

• Identifies the main idea events/theme of story/picture/media presentation.

• Expresses likes/dislikes. • Begins to express

thoughts/feelings about text media/pictures.

For example, the student: • Identifies words/symbols that

have similar parts. • Identifies all letters and basic

sounds. • Identifies/uses an expanded

sight vocabulary and phonological skills with minimal adult support.

• Responds to basic “wh” questions to demonstrate a clear understanding of texts/ media that he or she reads or that have been read to him/her.

• Identifies cause-effect events from text/media/pictures.

• Makes connections between information in text and life experiences.

• Interprets text and expresses an opinion and feelings about text.

0 ------------------------- 22 23 -------------------------47 48 ------------------------- 92 93 ------------------------- 120






























































0 ------------------------- 20 21 ------------------------ 47 48 ------------------------- 85 86 ------------------------- 120






























































0 ------------------------- 20 21 ------------------------- 44 45 ------------------------- 83 84 ------------------------- 120


Overall Secondary Grade 6 Reading Proficiency Continuum
































For example, the student: • Recognizes some letters/

words/symbols in the environment or from simple text.



• Identifies some basic information in text/pictures with significant support.

• Uses pictures/symbols to make predictions

• Responds to simple literal questions with significant support.

• Expresses a like or dislike of text.

For example, the student: • Identifies most to all letters

and basic sounds. • Reads most basic/functional

sight words in text with moderate adult help.

• Demonstrates basic understanding of texts/media that he/she reads or that have been read to him/her.

• Identifies basic information in text.

• Responds to basic “wh” questions.

• Begins to interpret text and expresses an opinion/feeling about text/pictures.

For example, the student: • Uses word study skills

(compound words/roots/affixes) to help read new words.

• Reads sight words and uses higher level phonological skills with minimal adult support.

• Consistently demonstrates a complete understanding of texts/media that he or she reads or that have been read to him/her.


• Infers ideas from text. • Expresses opinions about text

based on evidence.

0 ------------------------- 18 19 ------------------------- 35 36 ------------------------- 69 70 ------------------------- 120























































based on evidence.

0 ------------------------- 22 23 ------------------------- 35 36 ------------------------- 75 76 ------------------------- 120























































based on evidence.

0 ------------------------- 22 23 ------------------------- 37 38 ------------------------- 74 75 ------------------------- 120






































• Follows simple sequential steps from text/pictures.


• Uses pictures/symbols to make predictions.







• Follows multiple sequential steps from text/pictures.

• Locates information from appropriate source.



• Interprets text and expresses an opinion/feeling about text/pictures.

For example, the student: • Uses word study (compound

words/roots/affixes) to help read new words.



• Locates critical information in text/pictures and uses information to complete a task.


• Infers ideas from text. • Expresses an opinion about

text based on evidence.

0 ------------------------- 22 23 ------------------------- 40 41 ------------------------- 74 75 ------------------------- 120


Overall Elementary Grade 3 Mathematics Proficiency Continuum





• Student may be able to

attend to math instruction and may demonstrate knowledge of one or two or no basic math concepts only with significant, extensive, and intensive prompting and support.


none or only one or two of the math entry-level skills based on the Alternate Performance Indicators at an Emerging level.




demonstrates some understanding of basic math concepts and participates in activities with extensive support.


several basic math skills based on the Alternate Performance Indicators typically at the Emerging level with some at the Progressing level.




understanding of basic math concepts when provided moderate support.


basic math skills based on the Alternate Performance Indicators typically at the Progressing level with some at the Mastered level.




consistent understanding of the basic math concepts with minimal or no support.

• Student demonstrates many of

the math skills based on the Alternate Performance Indicators at the Progressing and Mastered levels.

• This level of functioning occurs

in multiple settings.

For example, the student: • Demonstrates very limited

or no understanding of most numerical and basic math concepts and skills.

• May attend to at most one or two of the math entry-level skills sporadically and only with significant, extensive, and intensive prompting and support.

For example, the student: • Matches one item to the

numeral. • Groups items in pairs with

significant support. • Recognizes symbolic

representations of time and clock.

• Identifies “more” or “less” using manipulatives.

• Sorts objects by color and shape with significant support.

• Indicates whether two shapes are similar or different.

• Copies a pattern with support. • Provides essential data about

himself/herself with significant support.

For example, the student: • Demonstrates one to one

correspondence. • Recalls and identifies

numbers. • Matches one item to numeral

one. • Identifies longer or shorter. • Tells time by the hour (1-5) • Identifies “more” or “less.” • Identifies basic shapes. • Matches 2 dimensional

shapes. • Sorts objects by color, shape,

and size. • Reproduces patterns • Sequence or order objects by

their color, shape, and size. • Provides essential data about

himself/herself.

For example, the student: • Matches one to nine items to

the correct numeral. • Tells time to the hour and half

hour. • Indicates understanding of

comparison words: more/fewer/same/none/larger/ smaller/less/most.

• Identifies which rectangular/square container holds more volume and which holds less.

• Matches the corresponding three-dimensional shapes with the two-dimensional shapes.

• Identifies rotated shapes. • Identifies missing elements in

a picture or sequence of numbers or symbols.

• Collects and sorts data by attributes.

0 ------------------------ 29 30 ------------------------ 63 64 ------------------------ 129 130 ------------------------ 225


























consistent understanding of basic math concepts with minimal or no support.








For example, the student: • Demonstrates one-to-one

correspondence in counting. • Matches one item to numeral

one. • Compares and identifies the

longer object or identifies more or less.

• Matches a circle to a circle and triangle to a triangle.

• Copies a simple pattern with significant support.

• Sorts objects based on a single attribute (e.g. shape, color).

• Identifies first object. • Provides data about

himself/herself with significant support.

For example, the student: • Groups items in pairs. • Matches one to nine items to

the correct numeral. • Identifies more, less, same,

none, larger, or smaller based on two items.

• Matches two-dimensional shapes.

• Identifies basic shapes (circle, triangle, square, rectangle) and indicates whether two shapes are same or different.

• Reproduces a repeating event.

• Copies/extends a simple pattern.

• Identifies first, next, and last. • Provides essential data about

himself or herself. • Recognizes different attributes

of objects/data.

For example, the student: • Orders factional items by from

smallest to biggest. • Estimates the answer simple

addition and subtraction number sentences.

• Compares quantities of objects that indicate more, less, same, larger or smaller.

• Identifies basic three dimensional shapes (sphere, cube, cylinder).

• Identifies missing elements in a sequence of numbers or symbols.

• Displays data using concrete representations.

• Recognizes and sorts data into categories based on attributes.

0 ------------------------ 33 34 ------------------------ 77 78 ------------------------ 124 125 ------------------------ 225


































For example, the student: • Matches 1 to 5 items to the

correct numeral with significant support.

• Indicates equal number of groupings with significant support.

• Indicates symbolic representations of time (by the hour, day/night), calendar, and clock.

• Identifies which area is bigger or smaller.

• Matches corresponding shapes.

• Identifies shapes that have been slid.

• Copies a pattern. • Provides essential data about

himself/herself.

For example, the student: • Matches one to twelve items

to the correct numeral. • Identifies equal and total

number of items in a set, and then adds one more item.

• Tells time by the hour. • Indicates understanding of

comparison words: more, fewer, same, none, larger, smaller, less, or more.

• Identifies rotated shapes. • Indicates which of two

objects or shapes is larger. • Completes a number chart

(1-10) with the missing number of objects.

• Identifies the missing number, symbol, or letter in a familiar sequence.

• Indicates answers to survey questions.

For example, the student: • Combines equal groups of two,

three, or four manipulatives and is able to count total.

• Groups different size rectangles, squares, and triangles into correct shape groups.

• Tells time to the hour and half hour.

• Groups different-sized rectangles, squares, and triangles into correct shape groups.

• Identifies basic shapes that rotated, flipped and slid.

• Recognizes patterns in a series of objects or numbers.

• Completes a missing number in a simple series of numbers.

• Collects data by asking questions of peers or adults.

0 ------------------------ 45 46 ------------------------ 87 88 ------------------------ 135 136 ------------------------ 225


Overall Secondary Grade 6 Mathematics Proficiency Continuum
































For example, the student: • Identifies which value on a number line is larger or smaller. • Performs simple addition

problem using objects with significant support.

• Uses measurement tools with significant support.

• Identifies time by the hour with significant support.

• Recognizes the concept of fast or slow.

• Differentiates between objects by size, color, and shape.

• Copies a simple pattern. • Demonstrates understanding

of “more” or “less.” • Begins to collect data with

significant adult support.

For example, the student: • Indicates values on a number

line of a number (0-19). • Solves a simple additional

problem. • Selects appropriate

measurement tool for measuring length or volume with moderate support.

• Matches a drawing or model to the actual object.

• Identifies concepts for time to the half hour.

• Identifies rates of movement, fast or slow.

• Identifies flipped shapes. • Connects dots to create a

basic shape. • Recognizes the patterns in a

series of objects or numbers. • Identifies smaller of two

presented quantities in a data set.

For example, the student: • Performs 4 basic calculations

(+, -, x, ÷) consistently with minimal support.

• Uses measurement tools with minimal or no support.

• Consistently recognizes, labels, and manipulates shapes.

• Extends a growing number pattern with minimal or no support.

• Communicates relationships between categories of data and formulates questions concerning the data sets.

• Predicts probability of an event.

0 ------------------------ 43 44 ------------------------ 76 77 ------------------------ 123 124 ------------------------ 225


































For example, the student: • Matches fractional pieces

with significant support. • Solves simple addition

problems with significant support.

• Demonstrates basic understanding of more or less in terms of numbers, time, volume, and length.

• Matches two-dimensional shape with significant support.

• Copies a simple pattern in a series of objects or numbers.

• Identifies that objects represents number of items.

• Identifies smaller of two presented quantities using concrete representations.

• Begins to collect data with significant adult support.

For example, the student: • Orders fractional items and

indicates which one is larger or smaller.

• Solves simple addition and subtraction problems.

• Generalizes more/less in terms of quantity, time, volume, and length.

• Identifies concepts of time, minutes and hours.

• Identifies rotated shapes. • Matches a three-dimensional

object to a two-dimensional picture or graphic of the object.

• Draws/creates two-dimensional shapes.

• Predicts the next number to occur in a simple pattern set.

• Identifies a straight line. • Identifies smaller of two

presented quantities in a data set.

For example, the student: • Adds fractional items with

minimal support. • Estimates answers to math

problems. • Performs 4 basic calculations


• Solves math problems in terms of quantity, time, volume, and length.

• Identifies concepts of time, minutes, seconds, and hours.

• Identifies rotated, flipped and slid shapes.

• Creates three dimensional shapes.

• Creates a simple pattern set. • Identifies the mode (most

frequently occurring score) of a small set of data.

0 ------------------------ 51 52 ------------------------ 89 90 ------------------------ 131 132 ------------------------ 225


































For example, the student: • Performs simple addition or

subtraction problems using objects with significant support.

• Identifies time by the hour and minutes with significant support




of “more” or “less”. • Begins to collect data with

significant adult support.

For example, the student: • Solves simple addition and

subtraction problems. • Adds sets of 2, 3, 4, and 5

objects. • Identifies rates of movement,

fast or slow. • Identifies concepts of time,

seconds, minutes and hours. • Identifies differences in

numbers or symbols. • Identifies symbols +, -, =. • Completes a missing number

in a simple series of numbers. • Uses manipulatives/assistive

technology to solve numerical problems.

• Indicates understandings of comparison words: more/fewer/same/none/larger/smaller/less/most.



• Adds and subtracts fractional pieces.

• Indicates area and perimeter of common shapes.

• Consistently recognizes, labels, and manipulates shapes.

• Extends a growing number pattern with minimal or no support.

• Communicates relationships between categories of data and formulates questions concerning the data sets.


0 ------------------------ 56 57 ------------------------ 102 103 ------------------------ 145 146 ------------------------ 225


































For example, the student: • Performs simple addition or

subtraction problems using objects and with support.




of more/less/none. • Begins to collect data.

For example, the student: • Shows proportional

relationship of 1:1. • Solves simple addition and

subtraction problems. • Adds sets of 2, 3, 4, and 5

objects. • Identifies rotated shapes. • Identifies rates of movement,

fast or slow. • Identifies symbols +, -, =. • Completes a missing number

in a simple series of numbers. • Indicates understandings of

comparison words: more/fewer/same/none/larger/smaller/less/most.

• Collects and records data by asking questions.



• Adds and subtracts fractional pieces.

• Estimates answers to math problems.

• Indicates area and perimeter of common shapes.

• Consistently recognizes and manipulates shapes.

• Indicates the line of symmetry. • Extends a growing number

pattern. • Communicates relationships

between categories of data and formulates questions concerning the data sets.


0 ------------------------ 79 80 ------------------------ 123 124 ------------------------ 151 152 ------------------------ 225

Date post:	02-Jul-2018
Category:	Documents
Upload:	phungngoc
View:	214 times
Download:	0 times

Report prepared by H. Gary Cook, Ph.D. Wisconsin Center ... Year 1 Technical... · Hawaii State...

Documents