EOC - Subscore Reporting Study · 2017-02-27 · This study explores one norm-referenced method...

Examples of Subscore Reporting for the Missouri EOC Assessments:

Study Results and Recommendation

Presented by Questar Assessment, Inc.

March 10, 2017

5550 Upper 147th Street West

Apple Valley, MN 55124 (952) 997-2700

www.questarai.com

http://www.questarai.com/

Examples of Subscore Reporting for the MO EOC Assessments i

Table of Contents

1. Questions for TAC ..................................................................................................... 1

2. Introduction ................................................................................................................ 1

2.1. Purpose of the Document .............................................................................. 1 2.2. Background .................................................................................................... 2 2.3. Assessment Overview ................................................................................... 2

3. Methods and Results ................................................................................................. 4

3.1. Classification Based on Percentile Rank ....................................................... 4

3.1.1. Method ............................................................................................... 4 3.1.2. Percentile Rank Results..................................................................... 5

3.2. Classification Based on SEM CIs .................................................................. 7 3.2.1. Method ............................................................................................... 7 3.2.2. SEM CI Results ................................................................................. 8

3.3. Classification Based on IRT CSEM and Theta Cut Scores .......................... 10

3.3.1. Method ............................................................................................. 10 3.3.2. IRT CSEM and Theta Results .......................................................... 11

4. Comparison of the Four Methods............................................................................. 14

5. Questar’s Recommendation .................................................................................... 17

6. References .............................................................................................................. 18

Appendix A: Percentile Rank Results ............................................................................ 19

Appendix B: SEM CIs .................................................................................................... 24

Appendix C: IRT Results ............................................................................................... 29

Appendix D: Subscore Classification Charts ................................................................. 35

List of Tables

Table 2.1. Test Composition and Characteristics ............................................................ 3

Table 2.2. Correlation Coefficients Between Total Test and Strands—Algebra I ............ 3

Table 2.3. Correlation Coefficients Between Total Test and Strands—English II ............ 3

Table 2.4. Correlation Coefficients Between Total Test and Strands—Biology ............... 4

Table 3.1. Classification Based on Percentile Rank ........................................................ 5

Table 3.2. Classification Based on SEM CIs ................................................................... 8

Table 3.3. Classification Based on IRT CSEM .............................................................. 11

Table 3.4. Classification Based on Theta Cut Scores ................................................... 12

Table 4.1. Comparison of Score Ranges for all Four Methods ...................................... 15

Examples of Subscore Reporting for the MO EOC Assessments 1

1. Questions for TAC

1. Does the TAC feel that Questar’s recommendation for reporting subscores is reasonable?

2. Regarding the multiple stakeholders interested in ISRs (e.g., educators, students, and parents), does the TAC believe they want normative- or criterion-referenced subscore interpretations? One can be above or below a standard and still have a relative strength or weakness.

3. Does one method stand out as clearly preferable to the TAC? 4. Would the TAC like any variants of the approaches to be considered? For

example, other percentile rank groupings could be used, such as making the middle band bigger (1–24, 25–75, and 76–99).

5. What criteria would the TAC recommend for reporting subscore data (e.g., minimum item or points)?

2. Introduction

2.1. Purpose of the Document

As a follow-up from the subscore reporting paper Questar presented at the December 2016 Technical Advisory Committee (TAC) meeting titled “Scaling and Reporting Subscores from the New MO EOC Assessments,” this document presents the results of a study and a recommendation to the Missouri Department of Elementary and Secondary Education (DESE) and the TAC for reporting student subscores for the Missouri End-of-Course (MO EOC) Assessments. In this study, data from the Spring 2016 administration of the Algebra I, English II, and Biology assessments were used to create student subscore profiles (i.e., what the subscore results show for students in each achievement level) based on the following four methods of classifying students for being on track to meet their learning expectations:

1. Classification based on percentile rank 2. Classification based on standard error of measurement (SEM) confidence

intervals (CIs) 3. Classification based on item response theory (IRT) conditional standard error of

measurement (CSEM) 4. Classification based on theta cut scores

Subscores are not currently reported on the individual student report (ISR), but DESE is considering the possibility. Subscores are of interest to state assessment stakeholders because of their potential for diagnostic, remedial, and instructional information. Educators and parents are often interested in more detailed information about students’ knowledge, skills, strengths, and areas for improvement. Like any scale, the subscore scale should facilitate proper score interpretations while minimizing misinterpretations and unwarranted inferences. To this end, the approach taken in this study was to report a limited number of ordered score categories for subscores (e.g., Low, Middle, and High; Below Target, On Target, and Above Target). This helps prevent subscores from being over-interpreted by students, parents, or teachers given the concern of subscore unreliability.


2.2. Background

Subscores (also known as strand scores, reporting category scores, or claim scores) indicate the performance of a student, school, or district on a related set of items that are typically defined in the content specifications. Subscore reporting warrants careful consideration of several issues, including the following:

Subscores can have limited reliability and wide SEMs.

Subscores are highly correlated with the total test score and therefore often provide limited information beyond the total test scores (i.e., the total test score often provides more information about a student's skills in the subscore area than the subscore itself).

Stakeholders often desire that subscores be reported despite their known limitations.

When reporting subscores, the pros and cons of various approaches (e.g., scores types) are often unfamiliar to stakeholders.

Subscores are often seen as a means of obtaining actionable information from test results, and school personnel and parents can have different interests regarding how subscores are used. For example, in some cases criterion-referenced interpretations are of interest, whereas in other cases norm-referenced interpretations are of interest. This study explores one norm-referenced method (i.e., percentile rank) and three criterion-referenced methods (i.e., SEM, IRT CSEM, and theta cut scores) for classifying subscore performance. 2.3. Assessment Overview

There are nine EOC assessments, and four of them (Algebra I, English II, Biology, and Government) are required for graduation. For this study, the Algebra I, English II, and Biology assessments were used to represent the MO EOC assessments because they offer different scenarios for which to study subscore reporting. For example, as shown in Table 2.1, the number of strands varies from two to four across Algebra I, English II, and Biology, with the number of points per strand ranging from 3–35. While some strands are multiple-choice (MC) only, the Writing strand and the Scientific Inquiry strand are entirely constructed response (CR). Only the Algebra strand has both MC and CR items. Table 2.1 also presents the mean, standard deviation (SD), reliability, and SEM for each test and subscore. The test-level reliability is 0.85 or higher across the three content areas, and the reliability for the strands is roughly 0.70 or higher, with the exceptions of the Number and Quantity and the Statistics and Probability strands that have lower reliability and the fewest number of points.


Table 2.1. Test Composition and Characteristics

Test Composition

#Items Test Characteristics

Test/Subscore MC CR Total #Points Mean SD Reliability SEM

Algebra I 40 4 44 50 22.78 8.99 0.87 3.24

Number and Quantity 3 -- 3 3 1.19 0.96 0.37 0.76

Algebra 17 4 21 27 10.70 5.05 0.76 2.47

Functions 15 -- 15 15 7.74 3.08 0.69 1.72

Statistics and Probability 5 -- 5 5 3.15 1.36 0.51 0.95

English II 35 3 38 45 31.73 6.58 0.85 2.51

Reading 35 -- 35 35 23.96 5.77 0.83 2.38

Writing -- 3 3 10 7.81 1.30 0.81 0.57

Biology 35 10 45 55 34.63 10.15 0.89 3.35

Characteristics and Interactions 22 -- 22 22 13.39 4.45 0.79 2.04

Changes in Ecosystems 13 -- 13 13 9.21 2.68 0.72 1.42

Scientific Inquiry -- 10 10 20 12.07 4.34 0.76 2.13

Tables 2.2, 2.3, and 2.4 show the correlation coefficients between the total test and strands for Algebra I, English II, and Biology, respectively. The tables report the Pearson correlation coefficients below the diagonal. Because the correlation coefficients are affected by the limited number of items measuring each strand, the correlation coefficient between two strands may be artificially low because of measurement error. The correlation coefficients were observed to be in the moderate to strong range (0.30 and above), indicating the expected relationships between the clusters. Table 2.2. Correlation Coefficients Between Total Test and Strands—Algebra I

Algebra I

Number and Quantity Algebra Functions

Statistics and Probability

Algebra I 1.00

Number and Quantity 0.55 1.00

Algebra 0.95 0.46 1.00

Functions 0.88 0.41 0.72 1.00

Statistics and Probability 0.71 0.33 0.58 0.58 1.00

Table 2.3. Correlation Coefficients Between Total Test and Strands—English II

English II Reading Writing

English II 1.00

Reading 0.98 1.00

Writing 0.65 0.51 1.00


Table 2.4. Correlation Coefficients Between Total Test and Strands—Biology

Biology

Characteristics and Interactions

Changes in Ecosystems

Scientific Inquiry

Biology 1.00

Characteristics and Interactions

0.90 1.00

Changes in Ecosystems 0.83 0.68 1.00

Scientific Inquiry 0.89 0.67 0.63 1.00

3. Methods and Results

3.1. Classification Based on Percentile Rank

3.1.1. Method

The percentile rank method is a normative approach to reporting subscores that ranks students based on their strand scores. For example, students scoring well compared to their peers would have percentile ranks in the 80s and 90s, students in the middle would have percentile ranks around 50, and students scoring poorly would have percentile ranks at the lower end. The percentile ranks reflect general strengths and weaknesses in comparison to other students in the state (e.g., a percentile rank of 75 suggests a strength compared to other students who took the same assessment, whereas a percentile rank of 25 suggests a weakness). One advantage is that this approach provides subscore information that is easily interpreted by students, parents, and teachers. The percentile rank calculations for this study were performed using raw scores rather than Rasch theta scores. However, the results would be the same because there is a one-to-one correspondence between the two sets of scores. The percentile rank corresponding to a particular raw score is interpreted as the percentage of students in the norm group who scored at or below the score of interest. The formula for percentile rank is as follows (Crocker & Algina, 1986):

𝑃 =𝑐𝑓𝑙 + 0.5(𝑓𝑖)

𝑁 × 100%

where 𝑐𝑓𝑙 is the cumulative frequency for all scores lower than the score of interest, 𝑓𝑖 is the frequency of scores in the interval of interest, and N is the number in the sample. For each strand, the percentile ranks were categorized into one of three categories, representing low, middle, and high percentile rank:

Category 1 = percentile ranks 1–33

Category 2 = percentile ranks 34–66

Category 3 = percentile ranks 67–99 Due to the discrete nature of the scale, the percent of students in each category may deviate from the one-third of students within each category.


3.1.2. Percentile Rank Results

Appendix A provides the percentile rank results (i.e., the raw score-to-percentile rank conversion tables). Appendix D presents the results graphically. The categories for percentile rank are Low, Middle, and High, whereas the other three methods categorize student performance on subscores as Below Target, On Target, and Above Target. Each figure in the appendix illustrates the percent of students classified as Below Target, On Target, and Above Target based on the total test achievement level (i.e., Below Basic, Basic, Proficient, and Advanced). The results for the subscore classification methods are placed side-by-side to facilitate comparisons. Table 3.1 presents the percentile rank classifications by student performance on the total test (i.e., Below Basic, Basic, Proficient, and Advanced). The student counts and percentages of students classified in the low, middle, and high percentile rank groupings are provided for each total test achievement level. Table 3.1. Classification Based on Percentile Rank

Strand

Total Test Achievement

Level

Classification

Total N-Count

Low Middle High

N-Count % N-Count % N-Count % Total %

Algebra I

Number and Quantity

Below Basic 4,234 53 2,975 38 710 9 7,919 100

Basic 5,081 43 5,005 42 1,855 16 11,941 100

Proficient 7,323 25 11,879 40 10,480 35 29,682 100

Advanced 416 4 2,088 18 8,886 78 11,390 100

Total 17,054 28 21,947 36 21,931 36 60,932 100

Algebra

Below Basic 7,785 98 146 2 -- -- 7,931 100

Basic 8,395 70 3,543 30 3 -- 11,941 100

Proficient 2,613 9 17,509 59 9,560 32 29,682 100

Advanced -- -- 59 1 11,331 99 11,390 100

Total 18,793 31 21,257 35 20,894 34 60,944 100

Functions

Below Basic 7,725 98 194 2 -- -- 7,919 100

Basic 9,154 77 2,755 23 32 -- 11,941 100

Proficient 5,565 19 16,347 55 7,770 26 29,682 100

Advanced 5 -- 982 9 10,403 91 11,390 100

Total 22,449 37 20,278 33 18,205 30 60,932 100


Below Basic 6,603 83 1,105 14 211 3 7,919 100

Basic 7,233 61 3,293 28 1,415 12 11,941 100

Proficient 6,009 20 8,487 29 15,186 51 29,682 100

Advanced 126 1 901 8 10,363 91 11,390 100

Total 19,971 33 13,786 23 27,175 45 60,932 100


Strand


Level

Classification

Total N-Count

Low Middle High


English II

Reading

Below Basic 1,995 100 -- -- -- -- 1,995 100

Basic 10,005 99 63 1 -- -- 10,068 100

Proficient 8,757 22 21,966 56 8,803 22 39,526 100

Advanced -- -- -- -- 11,149 100 11,149 100

Total 20,757 33 22,029 35 19,952 32 62,738 100

Writing

Below Basic 1,773 91 165 8 4 -- 1,942 100

Basic 5,142 52 4,661 47 169 2 9,972 100

Proficient 5,098 13 30,262 77 4,127 10 39,487 100

Advanced 125 1 6,079 55 4,945 44 11,149 100

Total 12,138 19 41,167 66 9,245 15 62,550 100

Biology

Characteristics and

Interactions

Below Basic 3,513 100 3 -- -- -- 3,516 100

Basic 13,662 84 2,537 16 69 -- 16,268 100

Proficient 4,651 16 15,204 53 8,765 31 28,620 100

Advanced -- -- 567 4 12,464 96 13,031 100

Total 21,826 36 18,311 30 21,298 35 61,435 100


Below Basic 3,502 100 14 -- -- -- 3,516 100

Basic 12,461 77 3,199 20 608 4 16,268 100

Proficient 5,496 19 11,752 41 11,372 40 28,620 100

Advanced 96 1 1,596 12 11,339 87 13,031 100

Total 21,555 35 16,561 27 23,319 38 61,435 100

Scientific Inquiry

Below Basic 3,450 100 7 -- 1 -- 3,458 100

Basic 12,366 76 3,687 23 156 1 16,209 100

Proficient 3,846 13 16,431 57 8,338 29 28,615 100

Advanced 5 -- 1,585 12 11,441 88 13,031 100

Total 19,667 32 21,710 35 19,936 33 61,313 100

The general trend observed was more consistent classification of students within an achievement level given more items in the strand. The results for the Biology strands and the Algebra and Functions strands show this pattern. Not surprisingly, students in the Below Basic category on the full test were primarily classified in the low group and students in the Advanced category on the full test were primarily classified in the high group based on their performance on the strands. The subscore profiles for students in the Basic and Proficient levels were more variable. Students in the Basic group were mostly classified as low, but there were some students in the middle and high percentile rank groups. Students in the Proficient group were mostly classified in the middle and high percentile rank groups, but some students were in the low group.


A less consistent pattern was observed for strands with fewer items. The results for the Algebra and Statistics and Probability strands (3 and 5 points, respectively) show that there were students in the low, middle, and high levels across all achievement levels. The results suggest that classifying student performance on the subscores using the percentile rank method may work best for strands with higher numbers of points for determining students’ strengths and weaknesses. For strands with a small number of points, the middle percentile rank group falls on a single raw score point (i.e., the Number and Quantity, Statistics and Probability, and Writing strands). 3.2. Classification Based on SEM CIs

3.2.1. Method

Another method that uses raw scores to interpret subscore results is to create an error band (i.e., confidence interval) around each raw score. No test provides a perfect measure of a student’s ability because all tests have a known SEM, which represents the amount of variability that can be expected in a student’s test score because of the inherent imprecision of the test. For example, if the student was tested again with a new test of comparable difficulty, he or she would likely obtain a slightly different score. The expected range for this new score is provided as a standard error and gives an indication of the margin of error for the score. The SEM is defined as the standard deviation of the distribution of observed scores for students with identical true scores. The standard deviation is a measure of the dispersion of the observed scores. For the normal distribution, about 32 percent of observations are more than one standard deviation above or below the mean. The SEM formula is as follows:

𝑆𝐸𝑀 = 𝜎𝑋√1 − 𝛼 This formula indicates that the value of the SEM depends on both the reliability coefficient and the standard deviation of test scores. SEMs allow statements regarding the overall precision of test scores. SEMs help place “reasonable limits” (Gulliksen, 1950) around observed scores through construction of an approximate CI. These intervals are constructed by taking the observed scores, X, and adding and subtracting a multiplicative factor of the SEM. For example, students with a given true score will have observed scores that fall between +/-1 SEM about two-thirds of the time. CIs were constructed around each raw score point in the strand using a CI of 1 SEM. The criterion for which to judge student performance on each strand was the mean performance of the “Just Proficient” students, which are students who obtained a score of 200 on the full test. The “Just Proficient” student means were based on 2,440 students for Algebra I, 1,873 students for English II, and 1,859 students for Biology. If the “Just Proficient” mean fell within the CI for a given raw score, students were considered on target for meeting their learning expectations. If the CI for a given raw score did not include the “Just Proficient” mean, student performance was determined


to be below target or above target depending on the location on the scale. Student’s strengths and weaknesses were identified with reference to the “Just Proficient” group. 3.2.2. SEM CI Results

Appendix B presents the CIs for each raw score and this method’s grouping for student results. Appendix D presents the results graphically. Each figure in the appendix illustrates the percent of students classified as Below Target, On Target, and Above Target. Table 3.2 presents the SEM CIs based on student performance on the total test (i.e., Below Basic, Basic, Proficient, and Advanced), including the student counts and percentages of students classified as Below Target, On Target, and Above Target. Table 3.2. Classification Based on SEM CIs

Strand


Level

Classification

Total N-Count

Below Target On Target Above Target


Algebra I

Number and Quantity

Below Basic 4,234 53 2,975 38 710 9 7,919 100

Basic 5,081 43 5,005 42 1,855 16 11,941 100

Proficient 7,323 25 11,879 40 10,480 35 29,682 100

Advanced 416 4 2,088 18 8,886 78 11,390 100

Total 17,054 28 21,947 36 21,931 36 60,932 100

Algebra

Below Basic 6,320 80 1,611 20 -- -- 7,931 100

Basic 3,069 26 8,717 73 155 1 11,941 100

Proficient 322 1 12,260 41 17,100 58 29,682 100

Advanced -- -- -- -- 11,390 100 11,390 100

Total 9,711 16 22,588 37 28,645 47 60,944 100

Functions

Below Basic 5,769 73 2,146 27 4 -- 7,919 100

Basic 3,288 28 8,409 70 244 2 11,941 100

Proficient 685 2 15,685 53 13,312 45 29,682 100

Advanced 1 0 261 2 11,128 98 11,390 100

Total 9,743 16 26,501 43 24,688 41 60,932 100


Below Basic 3,935 50 3,773 48 211 3 7,919 100

Basic 2,794 23 7,732 65 1,415 12 11,941 100

Proficient 1,401 5 13,095 44 15,186 51 29,682 100

Advanced 14 -- 1,013 9 10,363 91 11,390 100

Total 8,144 13 25,613 42 27,175 45 60,932 100


Strand


Level

Classification

Total N-Count



English II

Reading

Below Basic 1,995 100 -- -- -- -- 1,995 100

Basic 7,266 72 2,689 27 113 1 10,068 100

Proficient 16 -- 5,778 15 33,732 85 39,526 100

Advanced -- -- -- -- 11,149 100 11,149 100

Total 9,277 15 8,467 13 44,994 72 62,738 100

Writing

Below Basic 1,619 83 319 16 4 -- 1,942 100

Basic 3,199 32 6,604 66 169 2 9,972 100

Proficient 1,999 5 33,361 84 4,127 10 39,487 100

Advanced 18 -- 6,186 55 4,945 44 11,149 100

Total 6,835 11 46,470 74 9,245 15 62,550 100

Biology

Characteristics and

Interactions

Below Basic 3,392 96 124 4 -- -- 3,516 100

Basic 8,927 55 6,775 42 566 3 16,268 100

Proficient 1,071 4 10,507 37 17,042 60 28,620 100

Advanced -- -- 33 -- 12,998 100 13,031 100

Total 13,390 22 17,439 28 30,606 50 61,435 100


Below Basic 3,455 98 61 2 -- -- 3,516 100

Basic 9,595 59 6,065 37 608 4 16,268 100

Proficient 2,300 8 14,948 52 11,372 40 28,620 100

Advanced 12 -- 1,680 13 11,339 87 13,031 100

Total 15,362 25 22,754 37 23,319 38 61,435 100

Scientific Inquiry

Below Basic 3,441 100 16 -- 1 -- 3,458 100

Basic 8,114 50 7,692 47 403 2 16,209 100

Proficient 903 3 14,679 51 13,033 46 28,615 100

Advanced -- -- 583 4 12,448 96 13,031 100

Total 12,458 20 22,970 37 25,885 42 61,313 100

Similar to the percentile rank results, the SEM CI results show that the students in the Below Basic achievement level were primarily classified as Below Target, and students in the Advanced achievement level were primarily classified as Above Target based on their performance on the strands. The Writing strand is an exception that shows that Advanced students were classified as On Target more often than Above Target (55% vs 44%). The Writing strand consists of only CR items for a total of 10 points. The subscore profiles for students in the Basic and Proficient achievement levels showed classification in all three groups, with the exception of Reading where students in the Proficient group were classified as Above Target and On Target based on their performance on the Reading strand.


Generally, the results show that the On Target range includes lower scores and spans a broader range of scores compared to the percentile rank method. For example, the percentile rank method for the Scientific Inquiry strand defined the middle group between 11 and 14 raw score points, whereas the SEM CI method defined the On Target group as 9 to 13 points. 3.3. Classification Based on IRT CSEM and Theta Cut Scores

3.3.1. Method

Another method of gauging student performance on subscores is to generate raw-to-scale score (RSS) conversion tables for each subscore. Two sets of criteria were applied to determine the range of scores that identify student strengths and weaknesses:

1. The first procedure was to calculate CIs around the scale score using CSEM and to apply the Proficient cut score from the full test (scale score of 200). Student subscores that fell within the CI band that included a scale score of 200 were considered On Target. Student subscores that fell below that level were deemed to be areas in need of improvement. Student subscores that fell above that level were deemed to be areas of strength.

2. The second procedure applied to the RSS tables was to use the Proficient and

Advanced theta cut scores. Theta values below the Proficient cut score defined the Below Target range, theta values between the Proficient and Advanced cut scores defined the On Target range, and theta values above the Advanced cut score defined the Above Target range.

Winsteps was run to generate the RSS tables for each subscore using the following procedure:

1. Perform an anchor run using the items in the subscore and the existing Rasch parameters.

2. Generate RSS tables for each subscore by transforming the thetas to the scale score metric using transformation constants for the full test.

3. Apply the business rules for adjusting the cut scores and setting the lowest obtainable scale score (LOSS) and the highest obtainable scale score (HOSS):

a. The raw score cut for Proficient was selected as the lowest raw score

associated with a rounded scale score of 200. b. If there was no raw score associated with a rounded scale score of 200,

the raw score with the highest scale score below 200 was selected as the cut score and assigned a scale score of 200. For example, if two consecutive raw scores were associated with rounded scale scores of 198 and 201, the scale score of 198 was moved up to 200. The same procedure was followed if there was no scale score of 225 for Advanced.

c. Scale scores below 100 were rounded up to 100. d. Scale scores above 250 were rounded down to 250. e. For each assessment, the scale score was set to 250 for a perfect raw score.


3.3.2. IRT CSEM and Theta Results

Appendix C presents the results for the IRT methods and includes the RSS tables with the subscore groupings. The “CSEM CI Group” column shows the subscore classification using the first IRT procedure, and the “Theta Cut Group” column shows the subscore classification using the second IRT procedure. Appendix D presents the CSEM and theta results graphically. Tables 3.3 and 3.4 present the classification based on IRT CSEM and theta cut scores, respectively, including the student counts and percentages of students classified as Below Target, On Target, and Above Target. The results for the Writing strand are not presented because the Winsteps analyses resulted in non-convergence. When the convergence criteria were adjusted, the results were not reasonable. This is likely due to the strand consisting of a small number of CR items. Table 3.3. Classification Based on IRT CSEM

Strand


Level

Classification

Total N-Count



Algebra I

Number and Quantity

Below Basic -- -- 7,209 91 710 9 7,919 100

Basic -- -- 10,086 84 1,855 16 11,941 100

Proficient -- -- 19,202 65 10,480 35 29,682 100

Advanced -- -- 2,504 22 8,886 78 11,390 100

Total -- -- 39,001 64 21,931 36 60,932 100

Algebra

Below Basic 6,320 80 1,611 20 -- -- 7,931 100

Basic 3,069 26 8,717 73 155 1 11,941 100

Proficient 322 1 12,260 41 17,100 58 29,682 100

Advanced -- -- -- -- 11,390 100 11,390 100

Total 9,711 16 22,588 37 28,645 47 60,944 100

Functions

Below Basic 5,769 73 2,146 27 4 -- 7,919 100

Basic 3,288 28 8,409 70 244 2 11,941 100

Proficient 685 2 15,685 53 13,312 45 29,682 100

Advanced 1 -- 261 2 11,128 98 11,390 100

Total 9,743 16 26,501 43 24,688 41 60,932 100


Below Basic 3,935 50 2,668 34 1,316 17 7,919 100

Basic 2,794 23 4,439 37 4,708 39 11,941 100

Proficient 1,401 5 4,608 16 23,673 80 29,682 100

Advanced 14 -- 112 1 11,264 99 11,390 100

Total 8,144 13 11,827 19 40,961 67 60,932 100


Strand


Level

Classification

Total N-Count



English II

Reading

Below Basic 1,995 100 -- -- -- -- 1,995 100

Basic 7,266 72 2,761 27 41 -- 10,068 100

Proficient 16 -- 12,150 31 27,360 69 39,526 100

Advanced -- -- -- -- 11,149 100 11,149 100

Total 9,277 15 14,911 24 38,550 61 62,738 100

Writing

Below Basic -- -- -- -- -- -- -- --

Basic -- -- -- -- -- -- -- --

Proficient -- -- -- -- -- -- -- --

Advanced -- -- -- -- -- -- -- --

Total -- -- -- -- -- -- -- --

Biology

Characteristics and

Interactions

Below Basic 3,487 99 29 1 -- -- 3,516 100

Basic 11,585 71 4,485 28 198 1 16,268 100

Proficient 2,407 8 13,313 47 12,900 45 28,620 100

Advanced -- -- 164 1 12,867 99 13,031 100

Total 17,479 28 17,991 29 25,965 42 61,435 100


Below Basic 3,455 98 61 2 -- -- 3,516 100

Basic 9,595 59 6,065 37 608 4 16,268 100

Proficient 2,300 8 14,948 52 11,372 40 28,620 100

Advanced 12 -- 1,680 13 11,339 87 13,031 100

Total 15,362 25 22,754 37 23,319 38 61,435 100

Scientific Inquiry

Below Basic 3,448 100 9 -- 1 -- 3,458 100

Basic 10,384 64 5,669 35 156 1 16,209 100

Proficient 1,986 7 18,291 64 8,338 29 28,615 100

Advanced -- -- 1,590 12 11,441 88 13,031 100

Total 15,818 26 25,559 42 19,936 33 61,313 100

Table 3.4. Classification Based on Theta Cut Scores

Strand


Level

Classification

Total N-Count



Algebra I

Number and Quantity

Below Basic 4,234 53 2,975 38 710 9 7,919 100

Basic 5,081 43 5,005 42 1,855 16 11,941 100

Proficient 7,323 25 11,879 40 10,480 35 29,682 100

Advanced 416 4 2,088 18 8,886 78 11,390 100

Total 17,054 28 21,947 36 21,931 36 60,932 100


Strand


Level

Classification

Total N-Count



Algebra

Below Basic 7,900 100 31 -- -- -- 7,931 100

Basic 10,321 86 1,620 14 -- -- 11,941 100

Proficient 5,238 18 22,663 76 1,781 6 29,682 100

Advanced -- -- 1,733 15 9,657 85 11,390 100

Total 23,459 38 26,047 43 11,438 19 60,944 100

Functions

Below Basic 7,725 98 194 2 -- -- 7,919 100

Basic 9,154 77 2,781 23 6 -- 11,941 100

Proficient 5,565 19 20,451 69 3,666 12 29,682 100

Advanced 5 -- 2,539 22 8,846 78 11,390 100

Total 22,449 37 25,965 43 12,518 21 60,932 100


Below Basic 6,603 83 1,105 14 211 3 7,919 100

Basic 7,233 61 3,293 28 1,415 12 11,941 100

Proficient 6,009 20 8,487 29 15,186 51 29,682 100

Advanced 126 1 901 8 10,363 91 11,390 100

Total 19,971 33 13,786 23 27,175 45 60,932 100

English II

Reading

Below Basic 1,995 100 -- -- -- -- 1,995 100

Basic 9,837 98 231 2 -- -- 10,068 100

Proficient 3,285 8 36,021 91 220 1 39,526 100

Advanced -- -- 1,813 16 9,336 84 11,149 100

Total 15,117 24 38,065 61 9,556 15 62,738 100

Writing

Below Basic -- -- -- -- -- -- -- --

Basic -- -- -- -- -- -- -- --

Proficient -- -- -- -- -- -- -- --

Advanced -- -- -- -- -- -- -- --

Total -- -- -- -- -- -- -- --

Biology

Characteristics and

Interactions

Below Basic 3,516 100 -- -- -- -- 3,516 100

Basic 14,970 92 1,288 8 10 -- 16,268 100

Proficient 7,799 27 18,159 63 2,662 9 28,620 100

Advanced 1 -- 2,913 22 10,117 78 13,031 100

Total 26,286 43 22,360 36 12,789 21 61,435 100


Below Basic 3,516 100 -- -- -- -- 3,516 100

Basic 14,496 89 1,632 10 140 1 16,268 100

Proficient 10,802 38 12,630 44 5,188 18 28,620 100

Advanced 442 3 4,365 33 8,224 63 13,031 100

Total 29,256 48 18,627 30 13,552 22 61,435 100


Strand


Level

Classification

Total N-Count



Scientific Inquiry

Below Basic 3,456 100 2 -- -- -- 3,458 100

Basic 15,163 94 1,037 6 9 -- 16,209 100

Proficient 10,773 38 15,790 55 2,052 7 28,615 100

Advanced 178 1 5,615 43 7,238 56 13,031 100

Total 29,570 48 22,444 37 9,299 15 61,313 100

An unexpected finding for the Number and Quantity strand was no raw scores were assigned Below Target based on the CSEM method due to the small number of points. That is, a score of 0 was assigned as On Target. Comparing the two IRT methods, the theta cut score procedure required more raw score points to be categorized as On Target or Above Target. By definition, the CSEM CI spans scores lower than the Proficient cut score. The result is a lower proportion of students in the Above Target range and a higher proportion of students in the Below Target for the theta cut score method compared to the CSEM method. To illustrate the effect, the On Target level spans 18–23 points, whereas by applying the theta cut score to the conversion table, the On Target score range was defined as 21–29. 4. Comparison of the Four Methods

Table 4.1 presents a comparison of the defined score ranges for Below Target, On Target, and Above Target. The SEM and IRT CSEM results are similar in that the range of scores is the same or differs by one point across the strands. These two methods have cut scores that are lower than the percentile rank and theta cut score methods. The theta cut score method tended to set the most stringent criteria for classifying student performance on subscores, followed by the percentile rank method.


Table 4.1. Comparison of Score Ranges for all Four Methods

Percentile Rank SEM IRT CSEM Theta Cut Score

Strand Low Middle High Below Target

On Target

Above Target

Below Target

On Target

Above Target

Below Target

On Target

Above Target

Algebra I

Number and Quantity 0 1 2–3 0 1 2–3 -- 0–1 2–3 0 1 2–3

Algebra I 0–7 8–12 13–37 0–5 6–10 11–27 0–5 6–10 11–27 0–8 9–15 16–27

Functions 0–6 7–9 10–15 0–4 5–8 9–15 0–4 5–8 9–15 0–6 7–10 11–15

Statistics and Probability 0–2 3 4–5 0–1 2–3 4–5 0–1 2 3–5 0–2 3 4–5

English II

Reading 0–22 23–27 28–35 0–17 18–21 22–35 0–17 18–23 24–35 0–20 21–29 30–35

Writing 0–7 8 9–10 0–6 7–8 9–10 -- -- -- -- -- --

Biology

Characteristics and Interactions 0–11 12–15 16–22 0–9 10–13 14–22 0–10 11–14 15–22 0–12 13–17 18–22

Changes in Ecosystems 0–8 9–10 11–13 0–7 8–10 11–13 0–7 8–10 11–13 0–9 10–11 12–13

Scientific Inquiry 0–10 11–14 15–20 0–8 9–13 14–20 0–9 10–14 15–20 0–12 13–16 17–20


The following summarizes advantages and limitations of each method.

Percentile Rank: The percentile rank results proved to be moderate and within the range of the results produced by the other methods. The results placed students into the Low, Middle, and High categories with more equitable numbers that the other methods. The percentile rank method may be used with raw scores or Rasch ability estimates. If chosen, this approach will accomplish three things:

o Identify relative strengths and weaknesses of students, even when

students are in the highest and lowest achievement levels. o Allow more immediate reporting of results (if pre-equating is ever used to

develop RSS scoring tables) when the previous year’s performance distribution is used.

o Mitigate against the temptation to use achievement level names to describe subscore performance because subscore category assignments are not based on the total test’s cut scores.1

One challenge with the percentile rank method is that the subscore categories have a normative interpretation while the test scores have a criterion-referenced interpretation. The subscore information would have to be accompanied by a report interpretation guide to mitigate confusion.

SEM CI: The approach of using the “Just Proficient” student mean as a benchmark to determine student strengths and weaknesses has been used for other state student reports (e.g., California). There is a concern that users will interpret the “Just Proficient” mean as being the official standard when this approach is simply meant to provide some additional interpretive scaffolding for score users. Another concern with this approach is that it requires results from all students to calculate the “Just Proficient” mean. If more immediate score reporting is needed, this method is probably not reasonable. Additionally, the results of this study showed that this method tended to produce lower standards than the percentile rank and IRT theta cut score results. However, there are variations to this method, such as using the Proficient students rather than the “Just Proficient” students to calculate the mean subscore values.

IRT CSEM and Theta: Considering the IRT methods, the scale score metric typically produces more general, interpretable, and equitable results. Transforming raw scores to scale scores removes the effects of differing total points possible and differing item difficulty when students take the same test items. The two IRT methods placed the raw scores onto the same scale as the full test and the cut score(s) were applied to those RSS tables. However, one caution with this criterion-referenced approach is that the performance standards are set on the full test rather than at the subscore level. In this case, does a

1 Although not based on total test cut scores, it would be prudent to use a different number of subscore categories than the overall test achievement levels to avoid any confusion that might arise from that. A good rule of thumb might be to use a number of subscore categories that is one less than the number of achievement levels.


subscore value greater than or equal to 200 indicate that performance in that area also meets the performance standard? Likely, that inference is debatable. Because it is important that scales try to minimize unintended interpretations, there is good reason to avoid this approach. Other reasons to be cautious about this approach include the following:

o Non-convergence may be encountered, highlighting a practical challenge

in implementing this method for subscores with few items. This was the case for the Writing data.

o Unreasonable results may be encountered. One finding of the study was all scores were allocated to the On Target and Above Target categories, which was the case for the Number and Quantity strand. This highlights a weakness using the CSEM method with subscores of few points and unreliable measures, although additional business rules may be implemented (e.g., low scores are assigned the bottom category, and high scores are assigned the top category).

5. Questar’s Recommendation

The benefit of providing subscores is realized to the extent that the information supplements that of the full test. The subscore data should provide teachers, educators, and students with actionable information based on where students are regarding their learning expectations and peers. Although each method in this study is a legitimate way to provide feedback on subscore performance, Questar recommends providing percentile rank information on the subscores as the percentile rank method produced moderate and reasonable results. Because this approach is norm-referenced, Questar recommends naming the categories as Low, Middle, or High to convey that the groupings are relative to their peers. With the common concern regarding the low reliability of many subscores, especially subscores based on a small number of possible points, Questar also recommends providing subscore information when there is a minimum of 8 score points. Subscores with few items should be combined with another subscore, based on the judgment of content experts, to meet the 8-point minimum. The recommended procedure for subscore reporting involves providing ordered categorical information. Reporting at the classroom, school, and district levels will be based on the number and percent of students within each category. The results will be of similar format to the achievement level results for the MO EOC Assessments. Some caveats for the study include the following:

If the ISRs are released before all students are scored, the methods may need to be modified. For example, if raw scores from the current administration are not available, scale scores from the prior year may be used as a proxy.

Appropriate business rules should be defined and implemented for subscore reporting, where necessary.


6. References

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart and Winston, Inc.

Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.

Appendix A: Percentile Rank Results



Table A.1. Percentile Rank Results—Algebra I, Number and Quantity

Raw Score N-Count Cumulative

N-Count Percentile

Rank Percentile

Rank Group

0 17,054 17,054 28.0 Low

1 21,947 39,001 46.0 Middle

2 15,387 54,388 76.6 High

3 6,544 60,932 94.6 High

Table A.2. Percentile Rank Results—Algebra I, Algebra


N-Count Percentile

Rank Percentile

Rank Group

0 53 53 .1 Low

1 243 296 .3 Low

2 800 1,096 1.1 Low

3 1,819 2,915 3.3 Low

4 2,833 5,748 7.1 Low

5 3,963 9,711 12.7 Low

6 4,497 14,208 19.6 Low

7 4,585 18,793 27.1 Low

8 4,666 23,459 34.7 Middle

9 4,469 27,928 42.2 Middle

10 4,371 32,299 49.4 Middle

11 4,020 36,319 56.3 Middle

12 3,731 40,050 62.7 Middle

13 3,439 43,489 68.5 High

14 3,152 46,641 73.9 High

15 2,865 49,506 78.9 High

16 2,462 51,968 83.3 High

17 2,151 54,119 87.0 High

18 1,895 56,014 90.4 High

19 1,419 57,433 93.1 High

20 1,155 58,588 95.2 High

21 857 59,445 96.8 High

22 605 60,050 98.0 High

23 428 60,478 98.9 High

24 254 60,732 99.4 High

25 142 60,874 99.8 High

26 61 60,935 99.9 High

27 9 60,944 100.0 High



Table A.3. Percentile Rank Results—Algebra I, Functions


N-Count Percentile

Rank Percentile

Rank Group

0 91 91 .1 Low

1 525 616 .6 Low

2 1,519 2,135 2.3 Low

3 3,049 5,184 6.0 Low

4 4,559 9,743 12.2 Low

5 5,955 15,698 20.9 Low

6 6,751 22,449 31.3 Low

7 6,918 29,367 42.5 Middle

8 6,877 36,244 53.8 Middle

9 6,483 42,727 64.8 Middle

10 5,687 48,414 74.8 High

11 4,719 53,133 83.3 High

12 3,584 56,717 90.1 High

13 2,423 59,140 95.1 High

14 1,273 60,413 98.1 High

15 519 60,932 99.6 High

Table A.4. Percentile Rank Results—Algebra I, Statistics and Probability


N-Count Percentile

Rank Percentile

Rank Group

0 1,533 1,533 2.5 Low

1 6,611 8,144 7.9 Low

2 11,827 19,971 23.1 Low

3 13,786 33,757 44.1 Middle

4 15,626 49,383 68.2 High

5 11,549 60,932 90.5 High

Table A.5. Percentile Rank Results—English II, Reading


N-Count Percentile

Rank Percentile

Rank Group

0 4 4 .0 Low

1 0 4 .0 Low

2 4 8 .0 Low

3 12 20 .0 Low

4 23 43 .1 Low

5 60 103 .1 Low

6 120 223 .3 Low

7 217 440 .5 Low




N-Count Percentile

Rank Percentile

Rank Group

8 343 783 1.0 Low

9 455 1,238 1.6 Low

10 627 1,865 2.5 Low

11 690 2,555 3.5 Low

12 827 3,382 4.7 Low

13 875 4,257 6.1 Low

14 1,004 5,261 7.6 Low

15 1,217 6,478 9.4 Low

16 1,274 7,752 11.3 Low

17 1,525 9,277 13.6 Low

18 1,690 10,967 16.1 Low

19 1,898 12,865 19.0 Low

20 2,252 15,117 22.3 Low

21 2,627 17,744 26.2 Low

22 3,013 20,757 30.7 Low

23 3,431 24,188 35.8 Middle

24 3,969 28,157 41.7 Middle

25 4,521 32,678 48.5 Middle

26 4,918 37,596 56.0 Middle

27 5,190 42,786 64.1 Middle

28 5,372 48,158 72.5 High

29 5,024 53,182 80.8 High

30 4,151 57,333 88.1 High

31 2,961 60,294 93.7 High

32 1,642 61,936 97.4 High

33 630 62,566 99.2 High

34 157 62,723 99.9 High

35 15 62,738 100.0 High

Table A.6. Percentile Rank Results—English II, Writing


N-Count Percentile

Rank Percentile

Rank Group

0 340 340 .5 Low

1 0 340 .5 Low

2 163 503 .7 Low

3 534 1,037 1.2 Low

4 402 1,439 2.0 Low

5 2,019 3,458 3.9 Low

6 3,377 6,835 8.2 Low




N-Count Percentile

Rank Percentile

Rank Group

7 5,303 12,138 15.2 Low

8 41,167 53,305 52.3 Middle

9 4,382 57,687 88.7 High

10 4,863 62,550 96.1 High

Table A.7. Percentile Rank Results—Biology, Characteristics and Interactions


N-Count Percentile

Rank Percentile

Rank Group

0 2 2 .0 Low

1 26 28 .0 Low

2 74 102 .1 Low

3 248 350 .4 Low

4 617 967 1.1 Low

5 1,104 2,071 2.5 Low

6 1,798 3,869 4.8 Low

7 2,548 6,417 8.4 Low

8 3,266 9,683 13.1 Low

9 3,707 13,390 18.8 Low

10 4,089 17,479 25.1 Low

11 4,347 21,826 32.0 Low

12 4,460 26,286 39.2 Middle

13 4,543 30,829 46.5 Middle

14 4,641 35,470 54.0 Middle

15 4,667 40,137 61.5 Middle

16 4,426 44,563 68.9 High

17 4,083 48,646 75.9 High

18 3,730 52,376 82.2 High

19 3,315 55,691 88.0 High

20 2,687 58,378 92.8 High

21 2,067 60,445 96.7 High

22 990 61,435 99.2 High

Table A.8. Percentile Rank Results—Biology, Changes in Ecosystems


N-Count Percentile

Rank Percentile

Rank Group

0 50 50 .1 Low

1 206 256 .2 Low

2 619 875 .9 Low

3 1,155 2,030 2.4 Low

4 1,922 3,952 4.9 Low




N-Count Percentile

Rank Percentile

Rank Group

5 2,718 6,670 8.6 Low

6 3,707 10,377 13.9 Low

7 4,985 15,362 20.9 Low

8 6,193 21,555 30.0 Low

9 7,701 29,256 41.4 Middle

10 8,860 38,116 54.8 Middle

11 9,767 47,883 70.0 High

12 8,697 56,580 85.0 High

13 4,855 61,435 96.0 High

Table A.9. Percentile Rank Results—Biology, Scientific Inquiry


N-Count Percentile

Rank Percentile

Rank Group

0 372 372 .6 Low

1 619 991 1.1 Low

2 811 1,802 2.3 Low

3 1,030 2,832 3.8 Low

4 1,263 4,095 5.6 Low

5 1,498 5,593 7.9 Low

6 1,822 7,415 10.6 Low

7 2,276 9,691 13.9 Low

8 2,767 12,458 18.1 Low

9 3,360 15,818 23.1 Low

10 3,849 19,667 28.9 Low

11 4,616 24,283 35.8 Middle

12 5,287 29,570 43.9 Middle

13 5,858 35,428 53.0 Middle

14 5,949 41,377 62.6 Middle

15 5,682 47,059 72.1 High

16 4,955 52,014 80.8 High

17 3,997 56,011 88.1 High

18 2,971 58,982 93.8 High

19 1,712 60,694 97.6 High

20 619 61,313 99.5 High

Appendix B: SEM CIs


Appendix B: SEM CIs

Table B.1. SEM CIs—Algebra I, Number and Quantity

Raw Score CI lower CI upper SEM CI Group

0 (0.76) 0.76 Below Target

1 0.24 1.76 On Target

2 1.24 2.76 Above Target


Note: The Just Proficient mean is 0.87.

Table B.2. SEM CIs—Algebra I, Algebra





3 0.53 5.47 Below Target



6 3.53 8.47 On Target

7 4.53 9.47 On Target

8 5.53 10.47 On Target

9 6.53 11.47 On Target

10 7.53 12.47 On Target



















Appendix B: SEM CIs


Table B.3. SEM CIs—Algebra I, Functions







5 3.28 6.72 On Target

6 4.28 7.72 On Target

7 5.28 8.72 On Target

8 6.28 9.72 On Target









Table B.4. SEM CIs—Algebra I, Statistics and Probability




2 1.05 2.95 On Target

3 2.05 3.95 On Target




Table B.5. SEM CIs—English II, Reading










Appendix B: SEM CIs













18 15.62 20.38 On Target

19 16.62 21.38 On Target

20 17.62 22.38 On Target

21 18.62 23.38 On Target
















Table B.6. SEM CIs—English II, Writing









7 6.43 7.57 On Target

Appendix B: SEM CIs



8 7.43 8.57 On Target




Table B.7. SEM CIs—Biology, Characteristics and Interactions












10 7.96 12.04 On Target

11 8.96 13.04 On Target

12 9.96 14.04 On Target

13 10.96 15.04 On Target











Table B.8. SEM CIs—Biology, Changes in Ecosystems







Appendix B: SEM CIs






8 6.58 9.42 On Target

9 7.58 10.42 On Target

10 8.58 11.42 On Target





Table B.9. SEM CIs—Biology, Scientific Inquiry











9 6.87 11.13 On Target

10 7.87 12.13 On Target

11 8.87 13.13 On Target

12 9.87 14.13 On Target

13 10.87 15.13 On Target









Appendix C: IRT Results



Table C.1. IRT Results—Algebra I, Number and Quantity

Raw Score Theta SE

Scale Score CSEM

Achievement Level CI lower CI upper

CSEM CI Group

Theta Cut Group

0 -0.9417 1.9388 200 42 Proficient 158 242 On Target Below Target

1 0.6005 1.2463 225 27 Advanced 198 250 On Target On Target

2 2.0373 1.2440 234 27 Advanced 207 250 Above Target Above Target


Note: The theta cut values for Proficient and Advanced are 0.46 and 1.63, respectively.

Table C.2. IRT Results—Algebra I, Algebra

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group

0 -3.7151 1.8796 110 40 Below Basic 100 150 Below Target Below Target






6 -0.0380 0.4924 189 11 Basic 178 200 On Target Below Target

7 0.1891 0.4619 194 10 Basic 184 204 On Target Below Target

8 0.3914 0.4385 200 9 Proficient 191 209 On Target Below Target

9 0.5755 0.4204 202 9 Proficient 193 211 On Target On Target


11 0.9075 0.3969 210 9 Proficient 201 219 Above Target On Target




15 1.5117 0.3883 225 8 Advanced 217 233 Above Target On Target
















Table C.3. IRT Results—Algebra I, Functions

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group






5 -0.0378 0.5794 189 12 Basic 177 201 On Target Below Target












Table C.4. IRT Results—Algebra I, Statistics and Probability

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group



2 -0.2233 0.9557 200 20 Proficient 180 220 On Target Below Target





Table C.5. IRT Results—English II, Reading

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group










Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group






12 -0.7078 0.3825 182 6 Basic 176 188 Below Target Below Target

























Table C.6. IRT Results—English II, Writing

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group








Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group






10 0.0165 0.4466 190 9 Basic 181 199 Below Target Below Target














Table C.7. IRT Results—Biology, Characteristics and Interactions

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group











10 0.0165 0.4466 190 9 Basic 181 199 Below Target Below Target








Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group









Table C.8. IRT Results—Biology, Changes in Ecosystems

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group
















Table C.9. IRT Results—Biology, Scientific Inquiry

Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group












Raw Score Theta SE

Scale Score CSEM


CSEM CI Group

Theta Cut Group














Appendix D: Subscore Classification Charts



Figure D.1. Subscore Classifications by Achievement Level—Algebra I, Number and Quantity

Figure D.2. Subscore Classifications by Achievement Level—Algebra I, Algebra

0

10

20

30

40

50

60

70

80

90

100

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

Below Basic Basic Proficient Advanced

Pe

rce

nt

Subscore Classifications: Algebra INumber and Quantity

Above Target

On Target

Below Target

0

10

20

30

40

50

60

70

80

90

100

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: Algebra IAlgebra

Above Target

On Target

Below Target



Figure D.3. Subscore Classifications by Achievement Level—Algebra I, Functions

Figure D.4. Subscore Classifications by Achievement Level—Algebra I, Statistics and Probability

0

10

20

30

40

50

60

70

80

90

100P

R

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: Algebra IFunctions

Above Target

On Target

Below Target

0

10

20

30

40

50

60

70

80

90

100

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: Algebra IStatistics and Probability

Above Target

On Target

Below Target



Figure D.5. Subscore Classifications by Achievement Level—English II, Reading

Figure D.6. Subscore Classifications by Achievement Level—English II, Writing

0

10

20

30

40

50

60

70

80

90

100P

R

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: English IIReading

Above Target

On Target

Below Target

0

10

20

30

40

50

60

70

80

90

100

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: English IIWriting

Above Target

On Target

Below Target



Figure D.7. Subscore Classifications by Achievement Level—Biology, Characteristics and Interactions

Figure D.8. Subscore Classifications by Achievement Level—Biology, Changes in Ecosystems

0

10

20

30

40

50

60

70

80

90

100P

R

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: BiologyCharacteristics and Interactions

Above Target

On Target

Below Target

0

10

20

30

40

50

60

70

80

90

100

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: BiologyChanges in Ecosystems

Above Target

On Target

Below Target



Figure D.9. Subscore Classifications by Achievement Level—Biology, Scientific Inquiry

0

10

20

30

40

50

60

70

80

90

100P

R

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta

PR

SE

M

CS

EM

Th

eta


Pe

rce

nt

Subscore Classifications: BiologyScientific Inquiry

Above Target

On Target

Below Target

Date post:	02-Jun-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

EOC - Subscore Reporting Study · 2017-02-27 · This study explores one norm-referenced method...

Documents