George O. Wesolowsky Professor Emeritus of Management Science De Groote School of Business

G.O. Wesolowsky

Statistical Detection of Cheating on Multiple Choice Exams:

Software, Implementation, and Controversy

George O. Wesolowsky

Professor Emeritus of Management Science

De Groote School of Business

McMaster University, Hamilton. Ontario,

G.O. Wesolowsky

Outline of this Presentation

• Introduction: Cheating on multiple choice tests

• How I got into this.• Outline of statistical detection

methodology• Practical capabilities of SCheck• Common attitudes to detection and

prevention• Recommendations

G.O. Wesolowsky

Ideal* Writing Conditions

* I have seen more than 30% cheating under such conditions

G.O. Wesolowsky

Less than Ideal Writing Conditionshttp://math.berkeley.edu/~ribet/113/

G.O. Wesolowsky

Prevalence of MC Tests and Exams

• 30% ? of marks in UG classes given through MC

• At McGill (20000 undergrads) in the Fall Semester of 2002:

Finals: 83 courses,15072 students

Midterms: 70+ courses, 14000 students

G.O. Wesolowsky

How They Do It: Copying Sampler

G.O. Wesolowsky

How They Do It: Types of Cheating not Resulting in Similar Responses

Usually not vulnerable to statistical detection

I am an impostor

G.O. Wesolowsky

A guide to cheating during tests and examinationsFrom Wikibooks, the open-content textbooks collection

http://en.wikibooks.org/wiki/A_guide_to_cheating_during_tests_and_examinations

Contents[hide]1 Preamble

1.1 A few definitions to consider 1.2 The rewards/dangers of cheating

1.2.1 Rationales of cheating 1.2.2 Rationales of prosecuting cheaters 1.2.3 Possible Penalties

2 General notes 3 Techniques

3.1 Copying from a person 3.1.1 Application of codes

3.2 Copying from a pre-written source 3.2.1 Directly from textbook/notes 3.2.2 Cheat Sheet

3.3 Precautions 3.4 Copying from a planted source 3.5 Locating Cheating Material on the Web 3.6 Test previewing

G.O. Wesolowsky

Some Statistics plagiarized text (CAI)CAI Research Conducted By Don McCabe (Released In June, 2005) is typical of many studies: As part of CAI’s Assessment Project, almost 50,000 undergraduates on more than 60 campuses have participated in a nationwide survey of academic integrity since the fall of 2002. The results were disturbing, provocative, and challenging.

On most campuses, 70% of students admitted to some cheating. Close to one-quarter of the participating students admitted to serious test cheating in the past year and half admitted to one or more instances of serious cheating on written assignments.

Faculty are reluctant to take action against suspected cheaters. In Assessment Project surveys involving almost 10,000 faculty, 44% of those who were aware of student cheating in their course in the last three years, have never reported a student for cheating to the appropriate campus authority. Students suggest that cheating is higher in courses where it is well known that faculty members are likely to ignore cheating.

G.O. Wesolowsky

One Method of Cheating Detection

G.O. Wesolowsky

Questionable Statistical Detection

It is not infrequent that instructors, when confronted by a suspected cheating situation, invent their own methodology on the spot. This is usually what I call ‘outlier methodology’. The basis is some way of using the number of wrong answers that two students have in common.

It could be simply a count of such 'wrong matches', a proportion, a run length, a ratio with other counts, or a multivariate plot of such variables. The idea is to look for outliers and attribute them to cheating.

G.O. Wesolowsky

Bonnie and Clyde engaged in suspicious behavior. A comparison of responses revealed:

“Bonnie and Clyde are surprisingly similar; 23 matches out of 23 wrong.”

.....C......B...........C........BD............D..DB.A..ABD.B..A..CB...B.........ABAC..A..C

.....C......B...........C........BD............D..DB.A..ABD.B..A..CB...B.........ABAC..A..C

http://www.astro.washington.edu/fraser/multiple-choice-cheating.html

Example

. = correct Both chose C, which is wrong

G.O. Wesolowsky

But then:

“Holmes and Watson are surprisingly similar; 8 matches out of 11 wrong.”

..........................D......D....................B.....D.A.......BBA.........BD.B.....

CD..BCC....BB...DC..CA..D.EC.....D......AD...B.DDAB...B..A..A.AA....CDB.AA.C......BDDBE.B..

The instructor wrote a program:

“My script just returns any match that has a high percentage of matching errors (and sufficient errors to convince you that some thing's up!) “

Intuitive override: “ I had found by chance (Bonnie and Clyde), but what about the rest? It's very unlikely that Holmes and Watson were cheating, but I think it's likely that the others were”.

This instructor then concluded that statistical detection is not really reliable. Bad statistical detection often discredits the good.

G.O. Wesolowsky

Aside: A Better “quickie” Index

• A better but not good simple index is the Harpp-Hogan index, which is the number of wrong matches divided by the number of differences. One is supposed to be suspicious when it is > 1. For Holmes and Watson this works out to 8/32.

G.O. Wesolowsky

Problems with “Simple” Indices or combinations thereof

• The value of the indices can depend in an unknown way on class size, number of questions, number of choices, etc.

• They use very little information. Capability of students and difficulty of questions are often not incorporated

• The risk of “false accusations” is not predictable• Many combinations of indices and plots are

possible, and they may point in different directions.

G.O. Wesolowsky

How I Got Into This

• Request from an administrator

• Two students were suspected in another course, how many exactly similar answers did they have in my course?

• Probability tree diagrams

• Checked the literature

Wesolowsky G.O. (2000) "Detecting Excessive Similarity in Answers on Multiple Choice Exams", Journal of Applied Statistics, Vol. 27, 909-921.

G.O. Wesolowsky

Probability of a match by students j and k on

question I = sum of match

probabilities

Question i

pji

1 - pji

1 - pki

1 - pki

1 - pki

1 - pki

pki

w1i

w1i

w2iw2i

w3i

w4i

w3i

w4i

match

match

match

match

match

Probability correct

Cond. probability wrong

G.O. Wesolowsky

Assumptions

• The probability that a student gets an answer right depends on the ability of the student and the difficulty of the question

• The probability of a match on wrong questions depends on the ‘popularity’ of wrong answers

• Independencies as implicit in the diagram

G.O. Wesolowsky

But how to we estimate wli and pji ?

G.O. Wesolowsky

depends on two things

ˆ 1ja Proportion of class that answered correctly on question i

Below average student

Above average student

ˆ jip

0 1

1

ˆ jip

ˆ 1ja

ˆ 1ja

ir

G.O. Wesolowsky

Finding

cj = proportion of questions answered correctly by student j

ˆ ˆ1/ˆ (1 (1 ) )j ja a

ji ip r

ˆ ja1

ˆq

jii

j

pc

q

Find by solving

ˆ ja

G.O. WesolowskyP value for each pair of students = probability of the observed number

of matches or more

Question 1 Question j Question q

MM M

Compound Binomial Distribution because the probability of a match is different on each question

G.O. Wesolowsky

** pair = 2 78 ** Harpp-Hogan stat = #wr.mat/#diff = 19.00##################################################################Zb = 7.891 'equivalent' z from the BVP modelSignificance of Zb on a pre-selected pair = 1.5E-15Significance bound (Bonferroni) on program selected pairs = 1.3E-11 #matches = 33 | 34 (mu,s)=( 11.410, 2.689)prop. right for 2 = 0.441 prop. right for 78 = 0.412 Quest. range = [ 1 34 ] #students = 132 ----------------------------------------------------------------.d.abccd.e .e.abedb.. ...da..b.. ea.e --------------------------------------------------------------- .d.abccdee .e.abedb.. ...da..b.. ea.e ----------------------------------------------------------------estimated match probabilities:0.423 0.357 0.360 0.324 0.367 0.377 0.376 0.232 0.285 0.316 0.237 0.236 0.369 0.283 0.249 0.423 0.254 0.321 0.255 0.483 0.696 0.310 0.371 0.238 0.345 0.536 0.258 0.211 0.460 0.290 0.224 0.326 0.388 0.231


Pvalue =P(Number of matches 33) 1.5 15E

Zb is found by solving P(Z Zb) =1.5E-15

Example of SCheck Output

G.O. Wesolowsky

Data Dredging

• The number of student pairs examined is n(n-1)/2.

• For 693 students this is 239,778 pairs

CompBinZ

Frequency

6.85.13.41.70.0-1.7-3.4-5.1

12000

10000

8000

6000

4000

2000

0

Histogram of CompBinZ

CompBinZ7.25.43.61.80.0-1.8-3.6

Dotplot of CompBinZ

Each symbol represents up to 490 observations.

suspicious

G.O. Wesolowsky

“Unusual” Z’s Depend on Class Size

Class size No. of pairs P(Zmax >3) P(Zmax >4) P(Zmax >5) P(Zmax >6)

2 1 0.00135 3.167E-5 2.8665E-7 9.8659E-10

100 4950 0.99875 .145104 .0014179 4.8836E-6

400 79800 1.00000 .920134 .0226151 7.87297E-5

1000 499500 1.00000 1.00000 .1334040 .0004928

5000 12497500 1.00000 1.00000 .9721919 .0122542

G.O. Wesolowsky


15 11(132)(132 1)1.5(10 ) 1.3(10 )

2

Multiply the Pvalue by n(n-1)/2

A similarity this unusual will occur at most 1.3 times, on the average, per 100 billion classes.

Important!

• The significance (probability that a similarity that high will occur for an innocent pair) is different for a pair that is pre-selected by, say, suspicious behavior, from that of a pair that was selected purely by the program. In other words, the former case does not need as high a level of similarity evidence. Scheck, therefore, allows pre-selected pairs to be forced into the analysis

G.O. Wesolowsky

G.O. Wesolowsky

Features of SCheck

• Up to 30000 students• Up to 200 questions• Up to 27 choices, numbers or

letters• True or false or multiple choice

in any combination• Select a contiguous block of

questions• Option for pre-selected

student pairs• Option for similarity scores for

all students• Options for removing

student identification from input and output files

• New: Two Type I methods for setting cutoffs

• Adjustment for “speed tests”• Interactive or stored option choice• Batch processing of multiple files• Optional Excel grades output• Files with all components necessary

for verification of calculations• Diagnostic graph• Optional fine tuning (T parameter)• Compact and intuitive question

diagnostics • Utility programs (format translators)

Developed from experience with large scale testing, research into cheating psychology, tribunal cases, different data formats, etc.

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

This box and the previous one allow selection of a block of questions. Useful if,say, some questions only gather information.

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

This forces suspect pairs into the output for analysis.

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

•Vertical red line indicates similarity cutoff. Position depends on class size

•Straightness = normality

•Slope indicates stdev of Z’s

•Innocent class is symmetrical within the lines

G.O. Wesolowsky

G.O. Wesolowsky

Forced pairs in NAM file

G.O. Wesolowsky

Forced pairs in OUT file

G.O. Wesolowsky

Diagnostics on questions

G.O. Wesolowsky

It’s Cheating time

G.O. Wesolowsky

Detected Pairs

Summary of significances of identified pairs--------------------------------------- pair Z A Priori Bonferroni Signif Signif. ----------------------------------------- 2, 78 7.891 1.5E-15 1.3E-11 2, 97 7.428 5.5E-14 4.8E-10 36, 69 6.253 2.0E-10 1.7E-6 36, 70 4.755 9.9E-7 8.5E-3 36, 72 5.514 1.8E-8 1.5E-4 60, 119 4.931 4.1E-7 3.5E-3 69, 70 6.527 3.3E-11 2.9E-7 69, 72 5.474 2.2E-8 1.9E-4 70, 72 6.527 3.3E-11 2.9E-7 78, 97 7.067 7.9E-13 6.9E-9----------------------------------------

All pairs were found in adjacent seating

G.O. Wesolowsky

36

69

70

72

2

78

97

60

119

132 students

teamwork

G.O. Wesolowsky

Have you ever seen anything like it? (Contact at a testing agency)

G.O. Wesolowsky

16, 39 6.45316, 41 5.45316, 42 6.09016, 44 5.05116, 46 5.08916, 50 5.07416, 65 5.83739, 41 7.38539, 42 8.17839, 43 6.19639, 44 6.06139, 46 6.91639, 50 6.87839, 57 5.66239, 64 5.89639, 65 7.88739, 69 5.51541, 42 6.95841, 43 5.51541, 44 5.043

41, 46 6.19641, 50 5.81141, 57 5.01141, 64 4.90741, 65 6.74542, 43 5.78642, 44 5.61442, 46 7.23642, 50 7.19042, 57 5.29342, 64 6.18342, 65 7.45842, 69 5.78643, 46 5.38643, 64 5.11843, 65 5.66044, 46 5.17644, 50 5.08344, 64 4.94144, 65 5.590

46, 50 6.01346, 57 4.93346, 64 5.78646, 65 6.33746, 69 5.05549, 65 4.96450, 57 4.90050, 64 5.73750, 65 6.31457, 65 5.10664, 65 6.02165, 69 5.66082, 87 8.45682, 109 7.75082, 110 8.94586, 92 6.38187, 109 7.37587, 110 8.45691, 93 5.89691, 100 7.385

91, 107 7.38593, 100 6.59293, 107 6.592100,107 8.716109,110 7.750113,117 5.328113,119 6.013113,124 5.386113,137 5.515117,119 7.062118,138 8.415120,122 6.319120,124 5.257120,137 7.054120,141 5.345122,137 5.664122,141 7.419124,137 5.185

Pair Z Pair Z Pair Z Pair Z

G.O. Wesolowsky

82

87

109

110

91

93

100

107

109

110

118

138

16

39

41

42

43

44

46

49

50

57

64

65

69

86

92

113

117

119

120

122

124

137

141

35 suspects, 79 pairs, 200 students

Really enthusiastic teamwork!

G.O. Wesolowsky

SCheck Version 8a7. #### 03-05-2009 11:21:34 Bonferroni program selected significance bound is 0.01_____________________________________________________________

** pair = 48 188 ** Harpp-Hogan stat = #wr.mat/#diff = 2.12##################################################################Zb = 5.211 'equivalent' z from the BVP modelSignificance of Zb on a pre-selected pair = 9.4E-8Approximate significance of program selected pair = 3.7E-5Signif. bound (Bonferroni) on program selected pairs = 5.6E-3 #matches = 42 | 50 (mu,s)=( 24.832, 3.318)prop. right for 48 = 0.540 prop. right for 188 = 0.580 Quest. range = [ 1 50 ] NRT = 1.00 #students = 348 ----------------------------------------------------------------.b.b...aa. a.ba..baa. ..cb..b.c. .c.e.aad.e .....b.a.b --------------------------------------------------------------- .b.b...aa. a.ba..baa. ..cbb.b.c. .c...d..ce .....b...a ----------------------------------------------------------------estimated match probabilities:0.502 0.542 0.530 0.616 0.558 0.679 0.571 0.499 0.566 0.747 0.568 0.613 0.499 0.536 0.635 0.552 0.652 0.577 0.639 0.609 1.000 0.597 0.253 0.434 0.278 0.555 0.383 0.405 0.524 0.422 0.674 0.297 0.289 0.231 0.429 0.225 0.211 0.661 0.453 0.270 0.825 0.219 0.585 0.705 0.433 0.257 0.562 0.217 0.451 0.295

A NEW OPTION

G.O. Wesolowsky

• Early studies for economics department

• For individual instructors

• Large assessment organizations

• National education departments

History of Applications

“This is the only instance where separate examination venues have shown up and I have used your analysis on probably about 30,000 candidates.”

G.O. Wesolowsky

“ The test scores of more than 50,000 students show evidence of cheating. Some of those students were the innocent victims of others copying their answers. But experts say most were likely either deliberately copying answers or had their answer sheets doctored by school staff. “

Dallas Morning News Study on Cheating on the TAKS test (June 2007), an Application of SCheck

“Officials at the Texas Education Agency have consistently argued that statistical analysis can't prove cheating and that they must rely on other forms of evidence – like getting teachers to confess to misbehavior – in their investigations. TEA decided not to use data drawn from student answer sheets – even with evidence of widespread copying in a classroom. “

DMN:

TEA response

G.O. Wesolowsky

Common Objections: We studied together, we are from a similar background, we are twins, etc.

Note that a huge number of pairs is being looked at.

1. We would expect that there would be a large number of highly similar pairs that couldn’t have cheated

2. Would expect that if prevention is implemented the high similarity rate would continue

Some direct studies have been done.

G.O. Wesolowsky

Both expectations have been proven false in thousands of data sets.

Prior to electronic cheating methods, no very high similarities lacking the opportunity to cheat (adjacent seating) were found.

Multiple versions of exams cause a drastic decrease in the cheating rate.

G.O. Wesolowsky

Pitfalls in interpretation

• High marks make detection difficult*• Non-responses can invalidate the model• Cheating must be substantial for detection• Too much cheating can invalidate the model• Hierarchical questions violate assumptions• Data sets may be too small

*In the extreme, if one student copies from another with a perfect test, there is no way statistical detection can distinguish this case from the case of two brilliant students with perfect papers. Scheck knows this.

G.O. Wesolowsky

Effective Prevention Measures

• Multiple versions

• Randomized or assigned seating• Seat spacing

• Invigilation

• Electronic counter-measures

• “Education on integrity” & ethics indoctrination

G.O. Wesolowsky

December 2002 exams McGill University. Proper prevention

measures are in place.

2

finals midterms

17

G.O. Wesolowsky

End Part 1

G.O. Wesolowsky

Multiple choice tests provide a substantial component of grades in undergraduate courses. Statistical detection of copying or collusion on such tests has proven to be quite successful, and has in turn demonstrated that simple and non-intrusive prevention measures are very effective. Why then is this subject conspicuously absent in most discussions on cheating prevention strategies?

G.O. Wesolowsky

The most common general attitudes of student leaders, instructors, and

administrators towards cheating are remarkably similar

See no Evil, Hear no Evil, Speak no Evil

G.O. Wesolowsky

Comments

• A “look the other way” strategy is actually the best one if the goal is to protect the integrity reputation of a university

• Why? Media assume that the cheating problem is proportional to the number of reported cases or the amount of discussion about cheating. Keeping quiet, therefore, creates the impression that there is no problem.

• A university that tries to do something about cheating often gets a reputation for being infested with cheating. (Otherwise, why would they talk about it?)

G.O. Wesolowsky

Instructor Attitudes

Instructor: Not on my tests you won’t!

G.O. Wesolowsky

#1

G.O. Wesolowsky

#2

G.O. WesolowskyInstructor: I’m not a Policeman or Prison Guard, I’m an Educator and

ResearcherTranslation: I am not going to charge any students with cheating or take any special prevention measures

This sounds idealistic and has the added benefit of saving a lot of work and avoiding unpleasant confrontations

G.O. Wesolowsky

Comments

• Implicit assumption: The university’s main function is to provide an education and grades are merely an unimportant side-issue.

• I disagree: The main product of a university is grades and degrees and diplomas. Without these, even if it continued to give a good education, the university would be out of business overnight.

On the other hand, degrees without education (diploma mills) are a growing phenomenon. http://chronicle.com/free/v50/i42/42a00901.htm

• The assumption of a degree is that a required level of academic achievement has been met. Grades and degrees without this level of achievement are defective.

• By giving degrees with based on defective grades it is giving the public a defective product.

G.O. Wesolowsky

Postscript: Education versus Grades

“Professor David Weale called it a "January clearance" -- and clear out they did. Dismayed by his crowded classroom, the history teacher at the University of Prince Edward Island offered his students a deal some couldn't resist: Drop this Christianity class and you'll get a B minus. “

By JANE ARMSTRONG Friday, January 27, 2006 Posted at 5:20 AM ESTFrom Friday's Globe and Mail

“The offer, also dubbed the "Weale deal" worked. The next week, about 20 of the 95 students were gone. So too is Prof. Weale after shocked administrators caught wind of the unorthodox academic transaction.”

G.O. Wesolowsky

Addendum

• The next week, he sweetened the offer, saying he'd give students who left a mark of 68. Students mulled it over during the break and negotiated the deal up to 70, which is a B minus. Departing students were required to send Prof. Weale an e-mail and pay the $450 for the course.

“Vice-president Gary Bradshaw said the school had no choice but to suspend Prof. Weale while a disciplinary probe begins. Offering students a credit without doing the work "strikes at the very heart of the academic principles," Mr. Bradshaw said.”

G.O. Wesolowsky

Student Leaders and Instructors

G.O. Wesolowsky

Student Leaders and Instructors: A Matter of Trust*

Instructor: “Cheating prevention and detection poisons the atmosphere and destroys the vital rapport and atmosphere of trust I have with my students”

Student Leader: “If you take prevention measures you show you don’t trust us and that cheating is expected. This will only increase cheating. Anyway, cheating is very rare.”

Translation: My teaching evaluations would go down.

Translation: My constituency would feel threatened.

*paraphrased

G.O. Wesolowsky

Administration

G.O. Wesolowsky

Administration: The Best Solution is ‘Ethics Reengineering’

“E-tegrity board members, …,have developed a range of new initiatives to integrate integrity into the college’s culture.”

“… suggest a more broadly focused approach that creates an educational community valuing academic integrity and focusing on the moral and ethical development of students”

Sounds good and plays well in the media. Often arises out of bad publicity on cheating incidents.

1) Will it work if it is seriously implemented? McCabe versus the economists http://www.arts.ubc.ca/Why_Students_Cheat.116.0.html

2) Will it ever be implemented beyond the talk level?Web pages, articles in school newsletters, “integrity officers”, inviting McCabe.Usual outcomes: a) effort fades away b) majority of students and instructors are not involved.

3) Contrast this with prevention measures on MC tests, which have been shown to virtually eliminate cheating.

G.O. Wesolowsky

Comments on Honor Codes

G.O. Wesolowsky

Honor Codes*

Traditional Honor Codes• Pledge and ceremonies• Proctoring of tests not allowed• Students in charge of tribunals• Students obliged to report infractions (rarely

enforced)

Modified Honor Codes• Proctoring allowed

*McCabe

G.O. Wesolowsky

Quote on Honor codesIn this same study that found over 75 percent of students admitting to cheating, McCabe saw that only 57 percent of students cheated at schools with honor codes. Chronic cheating also seems to reduce with honor codes. At schools without codes, 1 in 5 students admits to cheating more than twice. Only 1 in 16 students admits to the same offense at schools with honor codes. "I think it's a question of making your students understand that academic integrity is important to the school," McCabe said. "Just the fact that it's being discussed" can heighten student's awareness and reduce cheating, he concluded.

G.O. Wesolowsky

Do Honor Codes Work? Comments

• ?Only? 57%• Response bias: ‘Honor’ rhetoric leads to fewer admissions of cheating? Fear of Draconian penalties?• Control for other variables: Types of exams, subject

matter etc.• Low response rates → non-response bias

“it is clear the response rate is below desired levels, averaging between 10% and 15% and averaging between 10% and 15% and varying from as little as 5% to 10%varying from as little as 5% to 10% on some large campuses to over 50% on a limited number of small, residential campuses”

http://www.ojs.unisa.edu.au/journals/index.php/IJEI/article/viewFile/14/9

G.O. Wesolowsky

Advantages of Honor Codes

• Media friendly

• Reduce the number of reported cases of dishonesty

Disadvantages of Honor Codes?

G.O. Wesolowsky

G.O. Wesolowsky

G.O. Wesolowsky

Recommendations*

• Make statistical testing (even without identifying the students involved) a non-optional part of the scanning report.

• Institute mandatory or strongly “encouraged” prevention measures: multiple versions, assigned seating, electronic counter-measures, etc..

• As a last and least important step, use statistical testing to support charges of academic dishonesty. In other words, use it this way only after the cheating is cleaned up.

Observation: Statistical evidence is very unlikely to be sufficient by itself. Other evidence, such as a proctor’s observations, will be required to make charges stick.

* Plan is plagiarized from McGill

G.O. Wesolowsky

End

Date post:	10-Jan-2016
Category:	Documents
Upload:	aldis
View:	34 times
Download:	4 times

George O. Wesolowsky Professor Emeritus of Management Science De Groote School of Business

Documents