CMU Usable Privacy and Security Laboratoryhttp://cups.cs.cmu.edu/
Anti-Phishing PhilThe Design and Evaluation of a
Game That Teaches People Not to Fall for Phish
S. Sheng, B. Maginien, P. Kumaraguru, A. Acquisti, L. Cranor, J. Hong, E. Nunge
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 2
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 3
Phishing emailPhishing emailSubject: eBay: Urgent Notification From Billing Department
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 4
We regret to inform you that you eBay account could be suspended if you don’t update your account information.
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 5
https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&sid=verify&co_partnerid=2&sidteid=0
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 7
What is phishing?What is phishing?
Social engineering attack
Misrepresents electronic identity
Tricks individuals into revealing personal credentials
Defrauds users
Financial Services Technology Consortium. Understanding and countering the phishing threat: A financial serviceindustry perspective. 2005.
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 8
Countermeasures for phishingCountermeasures for phishing
Silently eliminating the threat• Regulatory & policy solutions
• Email filtering (SpamAssasin)
Warning users about the threat• Toolbars (SpoofGuard, TrustBar)
Training users not to fall for attacks
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 9
Design RationaleDesign Rationale
Security is a secondary task
Learning by doing
Fun and engaging
Better strategies
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 10
Online game• http://cups.cs.cmu.edu/antiphishing_phil/
Teaches people how to protect themselves from phishing attacks• Identify phishing URLs• Use web browser cues• Find legitimate sites with search engines
Anti-Phishing PhilAnti-Phishing Phil
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 17
More about the gameMore about the game
Four rounds• Two minutes in each round
• Increasing difficulty
Eight URL “worms” in each round• Four phishing and four legitimate URLs
• Users must correctly identify 6 out of 8 URLs to advance
In-between round tutorials
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 18
User StudyUser Study
Test participants’ ability to identify phishing web sites before and after training• 10 URLs before training, 10 after, randomized• Up to 15 minutes of training
Training conditions: • Web-based phishing education• Tutorial • Game
14 participants in each condition• Screened out security experts• Younger, college students
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 21
ResultsResults
No significant difference in false negatives among the three groups
Game group had fewest false positives
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 22
The effectsThe effects
Improvement could be due to • Learning to distinguish legitimate from phish
• Raising suspicion about all web sites
Learning is better than raising suspicion• Fewer false positives
• Will help people more in the long run
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 23
ConclusionsConclusions
Used signal detection theory to measure effects• Existing training materials increased suspicion
with little learning
• Game did not raise suspicion but resulted in players learning to distinguish legitimate from phish In some cases a little more suspicion would have
helped
Game condition performed best overall!
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 24
AcknowledgementsAcknowledgements
Members of Supporting Trust Decision research group
Members of CUPS lab
CMU Usable Privacy and Security Laboratory
http://cups.cs.cmu.edu/
Play Anti-Phishing Phil:http://cups.cs.cmu.edu/antiphishing_phil/
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 26
Falling for PhishingFalling for Phishing
0.43
0.34
0.12
0.19 0.17
0.38
0
0.1
0.2
0.3
0.4
0.5
Existing trainingmaterials
Tutorial Game
Fal
se N
egat
ive
Rat
e
Pre test
Post test
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 27
Misidentifying Legitimate SitesMisidentifying Legitimate Sites
0.300.27
0.30
0.41
0.210.14
0
0.1
0.2
0.3
0.4
0.5
Existing trainingmaterial
Tutorial Game
Fa
lse
Po
sit
ive
Ra
te
Pre test
Post test
• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 28
Lessons LearnedLessons Learned
Pilot test• Users be able to identify phishing
• But they misidentify real ones
Users tend to get the specifics, but not the underlying concepts • Conceptual – procedural knowledge
User didn’t ask father for help too much
