Using the Many-FacetedRasch Model to Evaluate
Standard Setting Judgments:An IllustrationWith the Advanced Placement
Environmental Science Exam
Pamela K. Kaliski, Stefanie A. Wind,
George Engelhard Jr., Deanna L.Morgan,
Barbara S. Plake, and Rosemary. Reshetar
psychology and applied statistics
http://ncaase.com/about/bio?id=4
http://des.emory.edu/home/people/faculty/Engelhard.html
Contents
2. Multiple Yes-No (MYN) method
1. Many-Faceted Rasch Model
4. Results and Conclusion
3. Instrument
22
introduction
Standard setting ‘‘. . . standard setting refers to the process of establishing one or
more cut scores on examinations. The cut scores divide the distribution of examinees’ test performances in two or more categories’’ Cizek and Bunch (2007)
criteria The criteria for evaluating panelist judgments Procedural validity : implementation issues and
documentation Internal validity : interpanelist and intrapanelist consistency External validity : comparisons with other methods
33
Many-Faceted Rasch Model
n: panelist k: a standard setting modified Angoff rating i :item ; j : round n is the judged severity for panelist n, i is the average judged item difficulty for item i, j is the judged average performance level for round j jk is the cut score, or threshold coefficient, from round j for
standard setting ratings of k. ( rating k relative to k − 1)
44
Rating quality indices
Rating quality indices(a) panelist severity/leniency measures:
separation statistics/chi-square statistic(b) model–data fit : Outfit MSE
(c) the creation of a visual display for comparing panelist judgments on the latent variable
55
Multiple Yes-No (MYN) method
MYN requires panelists to consider the borderline examinee at each cut score and to identify at which level the borderline examinee would be able to answer each item correctly.
panelists considered each item and decided whether or not the borderline examinee in
each category would be able to identify the correct answer
66
PLDs
77
Would a borderline-1/2 student be able to answer this item correctly? If yes, then the panelist would circle the 1/2 cut score on the rating form and move on to the next item.
If no, then the panelists would consider the next question about the same item: Would a borderline-2/3 student be able to answer this item correctly?
If yes, the 2/3 cut score would be circled for that item and the panelist would move on to the next item. If no, the panelist would consider the next question about the same item: Would a borderline-3/4 student be able to answer this item correctly?
If yes, the 3/4 cut score would be circled for that item and the panelist would move on to the next item.
If no, then the panelists would consider the next question about the same item: Would a borderline-4/5 student be able to answer this item correctly? If yes, the 4/5 cut score would be circled for that item and the panelist would move on to the next item. If no, then the panelist would consider the final question about the same item: Would the above borderline-5 student be able to answer this item correctly?
If yes (which is likely given that all other possible borderline students have
been considered), then the Above 5 score would be circled for that item.
88
99
instuments
The Advanced Placement (AP) program (Advanced Placement Environmental Science (APES) examination) is composed of 34 courses and corresponding examinations in 22 subject areas.
Data used in this study come from the 2011 administration of the APES exam and the standard setting for this examination.
100 MC items and four CR itemsData used in this study are the ratings that resulted
from two rounds of item-level judgments provided by the 15 APES panelists
1010
Research Purpose
the MFR model is used to evaluate the quality of judgments on MC items provided by panelists who participated in a modified Angoff standard setting that used the MYN method for MC items, the 2011 APES exam.
panelist characteristics (gender and level of teaching) are incorporated into the MFR model to determine whether or not these are explanatory variables that account for differences in panelist ratings
1111
Results and conclusion
1212
Results and conclusion P397
1313
P398
1414
MSE[0.6-1.5]
P400
1515
P401
1616
1717
P402
1818
P404
P405
1919
P405
2020
Future study
additional explanatory variablesadditional statistical modelsMFR model+other modified Angoff procedures,
or Bookmark proceduresoverall contribution of each facetCR questions
2121
Thus the interaction between theta and omega should be considered in Equation 1
different rating scale structure and a random effect approach
Through the PC power, let panelists use computer to do standard setting. We can record the time spent
transform cut score of each category to the expected scores
2222