Using Mobile Phones to Write in Air

Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Kevin Cave, Frank DeRuyter,

Romit Roy Choudhury

Wishful Thinking

What if we could:

Hold a phone like a pen,and write quick notes in the

air.

More Specifically …

User holds phone like a pen Writes short messages in air

Phone’s built-in accelerometer Records acceleration due to hand motion Converts signals image text

Displays text on screen And sends a copy to user’s email address

Could be a Cool Toy

But will anyone care?

Could be a Cool Toy

But will anyone care? Caption photos on-the-fly

Sketch maps, directions … even quick diagrams

One handed use … while carrying baby/suitcase or while driving

Duke Univ.

Let’s Build It

Prototype on Nokia N95 Equipped with 3-axes accelerometer Programmed usign Python + MATLAB

Did not expect this to be hard But we had heavily under-estimated …

Design Challenges (1)

Background Vibration (Noise) Sensitive accelerometers Jitter from hand vibrations

Approach: • Smooth signal by filtering high freq.

components• Suppress residual values < threshold

Design Challenges (2)

Computing Displacement Ambiguity when acceleration is zero Can be constant velocity or stationary Worse with noise

Approach: • Detect change of direction • Reset velocity between strokes Reset

Design Challenges (3)

Lifting Pen from the Paper e.g., dotting the “i” and crossing the “t”

Approach: • Impulse on the Z axis during the lift• Off the Record – On the RecordP

Z-axisspikes

Design Challenges (4)

Character transition e.g., B vs 13

Approach: • Use delimiters such as “.” (1.3 for 13)• Soft leftward motion between alphabets

Design Challenges (5)

Hands rotate while writing Accelerometers only measure linear acceleration Rotation injects ambiguity (Wii uses gyroscope)

Approach: • Hold phone in non-rotating grip• Determine angular orientation during the

pause

Drawing a Rectangle

Raw Accelerometer Reading

Noise Smoothing Residual Noise Suppression

Velocity Plot after Avoiding Velocity Drifts

Final Rectangle

Raw Accelerometer Data for M and Y

More PhonePen Words …

Character Recognition

While geometric version can be displayed as image, text format necessary for searching

English text expressed in strokes:

Simple Grammar Tree

AB R

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N

s0

I

\

s2

I

I

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

However, imperfect human gestures induce

distortions:

N

s0

I

\

s2

I

I

DP

B R

\

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

However, imperfect human gestures induce

distortions:

N

s0

I

\

s2

I

I

DP

B R

\ I

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N

s0

I

\

s2

I

I

DP

B R

\ I

N = I + + I

However, imperfect human gestures induce

distortions:

Dense Grammar Tree

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

N = I + \ + /

N = / + \ + /

N = / + I + /…

However, imperfect human gestures induce

distortions:

Spelling Correction

English spelling correction useful Improvement possible over edit distance

MQM MOM, MUM, MAM (all edit distance = 1)

P3 mostly confuses O as Q

NIET NET, MET (edit distance = 1, 2)P3 mostly confuses M as NI

All things combined PhonePoint Pen

On the Shoulders of Related Work

Stylus based sketch recognition SketchREAD, Electronic Cocktail Napkin, Unistrokes

Gesture recognition uWave, SmartQuil,

Vision based gesture recognition TinyMotion, Microsoft Write in Air, VisionWand

Others … Wiimote, Air-Mouse, SmartPen, NiiMe and others

Results

10 test users (students) 6 trained (practiced each letter once, < 5min) 4 novices Each user wrote 26 test characters in air

Results

Metrics Human readability Character recognition Word recognition (with spelling correction) Writing speed

Readability and Recognition

Human Readability

Character Recognition

Per-User Accuracy … Speed

Short Words

Words chosen randomly from the dictionary Written by trained users

Video Demo and Experience

Search “PhonePen” in YouTube for full video

Thank You

For demo video and more information,

Visit synrg.ee.duke.edu

Swipe your credit card and sign in air …

The vision is:

Of Course, Not a Product Yet

Lowercase character recognition Cursive handwriting more complicated

Need smaller hand movements in air

Youtube comment: looking crazy waving hands Slashdot: Nearby people can get injured

Quicker writing 3.02 seconds/alphabet is way too slow

Of Course, Not a Product Yet

What about longer messages Sentences? Email replies?

Erasing alphabets Shake your hands like using a eraser

Ongoing work is pursuing

these directions …

Evaluation

10 test users (students) 6 trained (practiced each letter once, <5min) 4 novices Each user wrote 26 test characters

5 patients from Duke Hospital To analyze applicability into assistive

technology

So, need a solution that is

Easy to use

Always-with-me

Allows sketching

Online/Searchable

Proposed Approach: PhonePoint Pen

User holds phone like a pen Writes messages in air

Built-in accelerometer Captures hand movement Converts to image / text

Display text on screen Also sent to user’s email address

Thank You!

email: romit@ee.duke.edu

Web: http://synrg.ee.duke.edu/

Noting small pieces of information,

quickly and effortlesslycan be useful

So, what are the options ...

CallJohn @ 6658

Buy Milk

Right on3rd

traffic light

Velocity Plot after Avoiding Velocity Drifts

Final Rectangle

Noting small pieces of information,

quickly and effortlesslycan be useful

But aren’t there several solutions ...

Buy Milk

State of the Art

Sticky notes … Difficult to organize electronically Not always handy

Typing on phone keyboards

Tiny keyboard sizes Small inter-key spacing

Audio recording Cannot sketch diagrams Time consuming to browse through voice messages

So, need a solution that is

Easy to use

Always-with-me

Allows sketching

Online/Searchable

Incorporating Tolerance to Errors

Patient Results

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

N = I + \ + /

N = / + \ + /

N = / + I + /…

However, human gestures induce

distortions:

N

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

N = I + \ + /

N = / + \ + /

N = / + I + /…

However, human gestures induce

distortions:

N

s0

I

\

s2

I

I

DP

B R

\

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

N = I + \ + /

N = / + \ + /

N = / + I + /…

However, human gestures induce

distortions:

N

s0

I

\

s2

I

I

DP

B R

\

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

N = I + \ + /

N = / + \ + /

N = / + I + /…

However, human gestures induce

distortions:

N

s0

I

\

s2

I

I

DP

B R

\ I

Incorporating Tolerance to Errors

Standard “N” = I + \ + I

N = I + + I

N = I + \ + /

N = / + \ + /

N = / + I + /…

However, human gestures induce

distortions:

N

s0

I

\

s2

I

I

DP

B R

\ I

Readability and Recognition