eelloo.nl

A basket caseUsing baskets and verification in unproctored testing

Jozien SilvesterLiesbeth van LeeuwenJelle Geertsma

WAOP Presentation 25 november 2016

eelloo.nl

Unproctored testing

• Stakes are high in selection

• Possibility of cheating

• Proctoring not always possible or efficient

• Benefits for candidates of unproctored testing: time and place of choice

eelloo.nl

Unproctored testing

• Validity proctored /unproctored equal if:

• Informing candidates about optimal test conditions

• Taking measures to prevent fraud

• Items that measure reasoning (fluid)

• Time limit

• Basket method

eelloo.nl

Basket method

• Large item bank

• Items of equal difficulty are placed in a basket

• Item is randomly picked from basket

• Same number of baskets for every candidate

• Four subtests: Figure series, Analogies, Diagrams, Number series

eelloo.nl

Benefits of basket method

• Easy calculation

• Easy monitoring, maintaining, extension

• Easy retesting (system remembers which items were already administered)

• Prevention of item leakage

eelloo.nl

Validity

• High correlation with proctored extended test (.75)

• Test-retest correlation (.77)

• About 8% scores lower than expected, possible cheaters

eelloo.nl

Basket monitoring

eelloo.nl

Basket monitoring

• Some items with equal difficulty are more equal than others

Basket Item Difficulty (n) Difference Average

1017 Hcyf106b 0.66 (287) 0.10 0.68

Hcyf106c 0.65 (266)

HCYF01b 0.74 (294)

1018 HCYF112 0.62 (409) 0.02 0.63

HCYF112b 0.64 (438)

eelloo.nl

Difference in p-value

• Even if items are carefully balanced, there will still bedifferences, although they may be small

• For most candidates this will average

• Some will have all the ‘easy’ or all the ‘difficult’ items

• Effect on reliability and score

• Comparison of correction methods on two measures

eelloo.nl

Score calculation improvement - reliabilities (split half)

Education level N CTT IRT1pl IRT2pl Elo Glicko

MBO (Vocational) 743 0.85 0.80 0.81 0.79 0.86

HBO (Bachelor) 1397 0.82 0.82 0.83 0.79 0.85

WO (Master) 706 0.80 0.76 0.77 0.74 0.81

• Glicko has best performance

• CTT has good results as well

eelloo.nl

Effects Glicko on z-score

Item difficulty group

regular scoreGlicko score

z-sc

ore

eelloo.nl

Concluding remarks

• Valid unproctored test possible with basket method

• Glicko shows slight improvement in score calculation, but CTT also works well