MoLe: Motion Leaks through Smartwatch Sensors
Authors: He Wang, Ted Tsung-Te Lai, Romit Roy Choudhury
Presentation: Joe Sirrianni
Central Question
• This work tries to answer the following question:
Can an accelerometer and gyroscope data from smart watches be mined to infer the words that a user is typing?
Assumptions and Setup
Assumptions:• Smart watch is on the left hand.• Only individual words are being typed.
Setup:• Samsung Gear Live smart watch
Approach: Use the motion of the hand to determine which letter are being typed.
Initial analysis: using camera to measure distance
MoLe System Overview
Subproblem: Keystroke Detector
• Used bagged decision trees to determine if a Z-axis acceleration was a key press or not.
Subproblem: Key-Press Location Estimation
• Created a process to determine key press location
• The process had 5 steps:1. Find Gravity and define
coordinates2. Estimate and Remove Gravity 3. Estimate hand location and
calculate project acceleration4. Calibrate by mean removal5. Kalman smoothing – Remove
remaining gravity error.
Subproblem: Point Cloud Fitting
• Essentially matching incoming data to training data. • Used Convex hulls to overlay the training data on the incoming data
Subproblem: Inferring the typed word
• This involved estimating:• The total number of keystrokes (The size of the word)• Consecutive Characters that won’t be detected by motion (“er”, “re”, “ea”,
”fa”)• Accounting for watch displacement (accounting for minute orientation
differences)• Accounting for transitioning between characters (v -> a, r -> a)• Keystroke intervals – how long it takes to type• Matching to dictionary words – had to estimate letters typed by right hand.
Experiment
• Training set – 2 people typed 500 common words• Testing set – 8 different people typed 300 common words
Results: How good is MoLe at guessing each word?
• User who typed more toward guidelines had better accuracy.
Results: Keyboard Variant
• Does using a different keyboard affect the results?• They trained the model on a laptop keyboard and tested with a user
on a desktop keyboard.
• They found that using a different keyboard did not affect the accuracy significantly.
Take-aways
• Wearable technology can leak information that can be exploited!• Smart watches can be used to guess the words a user is typing just
from the accelerometer and gyroscope data.
Criticisms
• I think this technology is more suited for breaking passwords than reconstructing blocks of text.
• Both the @ symbol and the TAB button are typed by the left hand.
Bonus: Can you guess the correct sentence?
Answer
The most profound technologies are those that disappear - Mark Weiser 1991