CHI'15 - WonderLens: Optical Lenses and Mirrors for Tangible Interactions on Printed Paper

WonderLens: Optical Lenses and Mirrors for Tangible Interactions on Printed PaperRong-Hao Liang, Chao Shen, Yu-Chien Chan, Guan-Ting Chou, Liwei Chan, De-Nian Yang, Mike Y. Chen, and Bing-Yu ChenNational Taiwan University and Academia SinicaThis is Rong-Hao from National Taiwan University, Today we are glad to talk about WonderLens, a system of optical lenses and mirrors for tangible interactions on printed paper.

Display devices become more and more portable

and even cheaper with lower power consumption.

Even so, paper still plays an important role today.because paper is more comfortable to see and play with than electronic displays.Also, paper affords natural interactions, because we already know how to use paper.

Printed content is not interactiveHowever, paper has its limitation, the content printed on paper is not interactive.When we interact with the printed content, the content does not change its state,So, the interactions stop here.

The MagicBook [Billinghurst et. al. 2001]

Handheld Displays or HUDs

Adding a Visual DisplayTo add interactivity on paper, many researches augmented a visual displays for users,such as holding a display or wearing a head-mounted display.

The Digital Desk [Wellner 1993]

Tabletop Pro-Cam Modules

Adding a Visual Displaymounting a projector-camera module to augment the paper.

MouseLight [Song et. al. 2010]

Handheld Pro-Cam Modules

Adding a Visual Displayand making projector-camera modules graspable.

HideOut [Willis et. al. 2013]

Handheld Pro-Cam Modules

Adding a Visual DisplayThese methods are effective to provide dynamic visual feedback.

HideOut [Willis et. al. 2013]

MouseLight [Song et. al. 2010]

The Digital Desk [Wellner 1993]

The MagicBook [Billinghurst et. al. 2001]

External Device and Notable Latency Reduce the Immersion

However, these external devices maybe bulky, or may introducing additional latencies to reduce the immersion of interactions.

Listen Reader [Back et. al. 2001]

Hiding the RFID sensor behind paperIncreasing Immersion

To increase immersion, designers “hide” the sensing mechanism behind paper, and use audio feedback instead.such as placing an RFID sensor behind paper with voice feedback.

JabberStamp [Raffel et. al. 2007]

Hiding the EMR sensor behind paperIncreasing Immersion

or placing an EMR sensor behind paper with voice feedback.

provide only auditory feedback

Listen Reader [Back et. al. 2001]

JabberStamp [Raffel et. al. 2007]

However, the modality of interaction is limited to auditory.

Actually, adding a visual display on paper does not necessarily reduce immersion,For example, the magnifying glass helped us to see the printed content in details, provides very immersive user experiences.

Interaction Model of Lenses & Mirrors

Optical Illusions

Printed content on PaperUser

Spatial Operations

Tool: Spatial Operations ∼ Optical Illusions

When we move the magnifying glass, it feeds the enlarged content back to us immediately.So, we feel this tool useful.Optical lenses and mirrors are likely to have this property.

PDMSpoly-dimethylsiloxane

Therefore, we explore the use of lenses and mirrors by molding acrylic and PDMS

PDMSpoly-dimethylsiloxane

By the way, PDMS is the material of making disposable contact lens.

ca b d e

Low NHigh NLow SHigh Stilted

cylindrical lens

convex lens

concave lens

prism angled mirror

5 basic lenses and mirrors

After analyzing the affordances and optical illusions,we identify a set of 5 basic lenses and mirrors to be the most useful.

tilted cylindrical lens

Tilted Cylindrical lens.

tilted cylindrical lens

Rotating it on a Circular symbol results in visual illusion of rotation.

flexible convex lensmade by PDMS

Flexible Convex Lens


Flexible Convex Lens not only allows for magnifying content by lifting it


but also allows for changing the magnification by pressing it.

concave lens

Concave lens,

concave lens

Concave lens allows users to change the shrinkage by lifting it.

prism

Prism, the beam-splitter,

prism

allows users to break a line by rotating it.

prism

or connect the lines by moving it.

prism

π/3 angled mirror

Angled Mirror is similar to a kaleidoscope.

π/3 angled mirror

Moving it can proportionally scale a pattern.

π/2 angled mirror

generates several replicates,

π/2 angled mirror

π/2 angled mirror

or freely scales a pattern.

Printed content

on PaperUser

Lenses and Mirrors

Immediate Visual & Haptic

Feedback

Close the Interaction Loop

The lenses and mirrors provide immediate visual and haptic feedback, close the interaction loop, and therefore allow for more interaction designs on printed paper.

Printed content

on PaperUser Computer

Lenses and Mirrors

ImmediateVisual & Haptic

Feedback

Dynamic Information


But, to facilitate Human-Computer Interaction, such as guiding a multi-step process.the system should allow for communicating dynamic information.

Printed content

on PaperUser Computer

Lenses and Mirrors


Feedback

Input

Output


If so, the computer should sense users operations as input, and display the output accordingly.

North South

TILTED CYLINDER CONVEX CONCAVE PRISM ANGLED MIRROR

c

a

bLow N High NLow S High S

released pressed released pressed

ca b d e

Low NHigh NLow SHigh S

On the input side, to get the lenses and mirrors sensed.

North South


c

a



Adding magnets on the lenses and mirrorsInvisible magnetic fields get tracked above the paper

We choose to add magnets on the lenses and mirrors.Because the magnets are small,and the Invisible magnetic fields can be tracked on and above the paper.

North South


c

a




Each magnetic unit is designed in unique pattern.

North South


c

a




So the magnetic fields can represent their ID and states.

GaussSenseAnalog Hall-Sensor Grid

[Liang et al. 2012]

Sense the Lenses and Mirrors

To sense the magnetic lenses and mirrors,We use GaussSense, the thin-form analog Hall-Sensor Grid, as the sensing platform.

Sense the Lenses and Mirrors

The GaussSense senses multiple magnetic lenses and mirrors through the paper.

RFID Reader

magnets

GaussSenseAnalog Hall-Sensor Grid

[Liang et al. 2012]

For detecting paper, we mount an additional RFID reader and two small magnets on the sensing platform.

Paperwith RFID attached

magnets

Each piece of paper is attached with an RFID tag and two magnets.

When the piece of paper snaps to the platform, the platform recognizes the RFID tag, and loads the content for interactions.The magnets align the coordinates between paper and the GaussSense.

User Computer

Lenses and Mirrors


Feedback

MagneticGaussSense

RFID ReaderPrinted content

on RFID Paper

[Liang et al. 2012]

OutputEnvironmental: ambient light, audioDistant: remote display

Nearby: point light

Kenneth P. Fishkin. 2004. A taxonomy for and analysis of tangible interfaces. Personal Ubiquitous Comput. 8, 5 (September 2004), 347-358.


After sensed the operation,the system can provide additional output in different levels of embodiment.We show 3 examples to illustrate them.

Application #1: Storytelling

Audio Feedback

First, Output with only Audio Feedback. In storytelling,


Audio Feedback

when a user rolls the character’s eye, and the character vocally tells the user what he sees.


Audio Feedback

When the two characters look at each other, they chat.

Application #2: CPR-Learning

Nearby Point Light

stethoscope

Second, Output with a Nearby Point Light. In the CPR learning program, a user uses an LED-mounted flexible convex lens as a stethoscope.


Nearby Point Light

stethoscope

When placing the stethoscope on a patient’s heart, the user sees the point light changes and hears the heartbeats.


Nearby Point Light

Then, the user presses and releases the soft convex lens at a constant rate to save the patient.The blinking point light shows it is well done.

BLE+Battery

By the way, the LED-mounted convex lens can be made wirelessly with BLE modules and battery.

Application #3: Hide-and-Seek

Ambient Light + Remote Display

Third, Output with Ambient Light & Remote Display. In the hide-and-seek game,a user sets the time of the game by placing an angled mirror.



then, the ambient light and remote display changes.



Then, use the magnifying glass to find out the character.



When a character is found, the glowing lens prompts the user to check the remote display, to see who is found and what the character is doing.

User Computer

Lenses and Mirrors

Immediate Visual & HapticFeedback

MagneticGaussSense

RFID ReaderPrinted content

on RFID Paper

Environmental: ambient light, audioDistant: remote display

Nearby: point light

Kenneth P. Fishkin. 2004. A taxonomy for and analysis of tangible interfaces. Personal Ubiquitous Comput. 8, 5 (September 2004), 347-358.

Conclusion Close the Interaction Loop

Conclusion, we introduced WonderLens, a system of lenses and mirrors that augments tangible and embodied interactions with printed paper. The double interaction loop allows for communicating dynamic information.

Future Work

Printed Optics [Willis et al. 2012]

Magic Lens [Willis et al. 2012]

Paper Generator [Karagozler et al. 2013]

3D Optic Printing Energy Harvesting

Advanced Lens Fabrication and Visual Designs

Future work can consider using advanced methods of lens fabrication and visual designs.or incorporating energy harvesting mechanism with paper.

Attachable Stylus Sensing Using Magnetic Sensor GridGaussSense

GaussBits

GaussBricks

GaussBitsMagnetic

Tangible Bits[Liang et al. CHI 2013]

GaussBricksMagnetic

Building Blocks[Liang et al. CHI 2014]

GaussSenseAnalog

Hall-Sensor Grid[Liang et al. UIST 2012]

GaussStonesShielded

Magnetic Tangibles[Liang et al. UIST 2014]

GaussStones

FingerPad: Private and Subtle Interaction Using Fingertips

Liwei Chan⇤ Rong-Hao Liang⇤† Ming-Chang Tsai‡ Kai-Yin Cheng† Chao-Huai Su†

Mike Y. Chen‡ Wen-Huang Cheng⇤ Bing-Yu Chen‡

⇤Academia Sinica †‡National Taiwan University⇤{liwei,whcheng}@citi.sinica.edu.tw †{howieliang,keynes,domossu}@cmlab.csie.ntu.edu.tw

‡{r98944021,mikechen,robin}@ntu.edu.tw

ABSTRACTWe present FingerPad, a nail-mounted device that turns thetip of the index finger into a touchpad, allowing private andsubtle interaction while on the move. FingerPad enablestouch input using magnetic tracking, by adding a Hall sen-sor grid on the index fingernail, and a magnet on the thumb-nail. Since it permits input through the pinch gesture, Fin-gerPad is suitable for private use because the movements ofthe fingers in a pinch are subtle and are naturally hidden bythe hand. Functionally, FingerPad resembles a touchpad, andalso allows for eyes-free use. Additionally, since the nec-essary devices are attached to the nails, FingerPad preservesnatural haptic feedback without affecting the native functionof the fingertips. Through user study, we analyze the three de-sign factors, namely posture, commitment method and targetsize, to assess the design of the FingerPad. Though the resultsshow some trade-off among the factors, generally participantsachieve 93% accuracy for very small targets (1.2mm-width)in the seated condition, and 92% accuracy for 2.5mm-widthtargets in the walking condition.

Author KeywordsInstant-available, private input, subtle interaction, eyes-freeinteraction, nail device, finger-mounted device.

ACM Classification KeywordsH.5.m. Information Interfaces and Presentation (e.g. HCI):Miscellaneous

General TermsDesign, Human Factors

INTRODUCTIONRecent developments have seen new proposals for glass-mounted displays for use in mobile computing. Thoughsimilar to head-mounted displays, glass-mounted displays(e.g., Google Glass) are specially designed to be lightweight,attachable, non-obstructive to natural vision, and with in-creased social acceptance.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected]’13, October 6–9, 2013, St. Andrews, United Kingdom.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-2271-3/13/10...$15.00.

a d

c

b c

Figure 1. FingerPad enables touchpad function through pinch ges-ture. The user can (a) enter passwords to the private glass display (b)by drawing numbers with the thumb tip on the index fingertip. (c) Theproposed technology is realized through magnetic tracking by adding amagnet and Hall sensor grid on the fingernails.

Although these types of displays permit personal and pri-vate visual outputs, their input methods may not offer thesame privacy. For example, voice input is commonly usedfor glass-mounted displays because it is expressive and ef-fective. However, voice input can be problematic in loudenvironments, and privacy issues arise with its use in pub-lic spaces (e.g., password input)[11]. Gesture input suffersfrom similar privacy concerns because input actions are eas-ily observable.

To permit private input, recent research proposes subtle inter-actions [2, 10, 15], which are based on implicit movementsand generally considered socially acceptable. For example,muscle interface [10] allows input through unobservable mus-cle movement. Foot gesture [12] detects subtle foot motions.Ring devices [1, 9] and fabrics [6] have been developed tosupport tap, spin, and slide inputs. Although these methodsallow subtle inputs (and thus allow for privacy and social ac-ceptability), they generally suffer from limited input space.

This paper presents FingerPad, a nail-mounted device thatturns pinched fingertips into a touchpad, allowing private,and subtle interactions. As illustrated in Figure 1, the usertreats the tip of their index finger as the touchpad, and theirthumb as the touch stylus. FingerPad enables touchpad func-tion using magnetic tracking, by adding a magnet and Hallsensors on the fingernails. Functionally, FingerPad resemblesa touchpad that users can easily learn to use. Allowing for 2Dtouch input, FingerPad is suited for rich interactions, includ-ing pointing, menu selection, and stroke input.

Sensing UIST’13, October 8–11, 2013, St. Andrews, UK

255

FingerPadWearable

Private Input[Chan et al. UIST 2013]

GaussSense

WonderLens: Optical Lenses and Mirrors for Tangible Interactions on Printed PaperRong-Hao Liang, Chao Shen, Yu-Chien Chan, Guan-Ting Chou, Liwei Chan, De-Nian Yang, Mike Y. Chen, and Bing-Yu ChenNational Taiwan University and Academia Sinica

WonderLens

WonderLensTUI on

Printed Paper[Liang et al. CHI 2015]

FingerPad

This project, WonderLens, shows another applications of GaussSense,that is enabling tangible interactions on Printed Paper.

For makers and researchers, who want to try out the GaussSense technology, we are happy to announce that.

Attachable Stylus Sensing Using Magnetic Sensor GridGaussSense

GaussBits

GaussBricks

GaussBitsMagnetic

Tangible Bits[Liang et al. CHI 2013]

GaussBricksMagnetic

Building Blocks[Liang et al. CHI 2014]

GaussSenseAnalog

Hall-Sensor Grid[Liang et al. UIST 2012]

GaussStonesShielded

Magnetic Tangibles[Liang et al. UIST 2014]

GaussStones

FingerPad: Private and Subtle Interaction Using Fingertips

Liwei Chan⇤ Rong-Hao Liang⇤† Ming-Chang Tsai‡ Kai-Yin Cheng† Chao-Huai Su†

Mike Y. Chen‡ Wen-Huang Cheng⇤ Bing-Yu Chen‡

⇤Academia Sinica †‡National Taiwan University⇤{liwei,whcheng}@citi.sinica.edu.tw †{howieliang,keynes,domossu}@cmlab.csie.ntu.edu.tw

‡{r98944021,mikechen,robin}@ntu.edu.tw

ABSTRACTWe present FingerPad, a nail-mounted device that turns thetip of the index finger into a touchpad, allowing private andsubtle interaction while on the move. FingerPad enablestouch input using magnetic tracking, by adding a Hall sen-sor grid on the index fingernail, and a magnet on the thumb-nail. Since it permits input through the pinch gesture, Fin-gerPad is suitable for private use because the movements ofthe fingers in a pinch are subtle and are naturally hidden bythe hand. Functionally, FingerPad resembles a touchpad, andalso allows for eyes-free use. Additionally, since the nec-essary devices are attached to the nails, FingerPad preservesnatural haptic feedback without affecting the native functionof the fingertips. Through user study, we analyze the three de-sign factors, namely posture, commitment method and targetsize, to assess the design of the FingerPad. Though the resultsshow some trade-off among the factors, generally participantsachieve 93% accuracy for very small targets (1.2mm-width)in the seated condition, and 92% accuracy for 2.5mm-widthtargets in the walking condition.

Author KeywordsInstant-available, private input, subtle interaction, eyes-freeinteraction, nail device, finger-mounted device.

ACM Classification KeywordsH.5.m. Information Interfaces and Presentation (e.g. HCI):Miscellaneous

General TermsDesign, Human Factors

INTRODUCTIONRecent developments have seen new proposals for glass-mounted displays for use in mobile computing. Thoughsimilar to head-mounted displays, glass-mounted displays(e.g., Google Glass) are specially designed to be lightweight,attachable, non-obstructive to natural vision, and with in-creased social acceptance.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected]’13, October 6–9, 2013, St. Andrews, United Kingdom.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-2271-3/13/10...$15.00.

a d

c

b c

Figure 1. FingerPad enables touchpad function through pinch ges-ture. The user can (a) enter passwords to the private glass display (b)by drawing numbers with the thumb tip on the index fingertip. (c) Theproposed technology is realized through magnetic tracking by adding amagnet and Hall sensor grid on the fingernails.

Although these types of displays permit personal and pri-vate visual outputs, their input methods may not offer thesame privacy. For example, voice input is commonly usedfor glass-mounted displays because it is expressive and ef-fective. However, voice input can be problematic in loudenvironments, and privacy issues arise with its use in pub-lic spaces (e.g., password input)[11]. Gesture input suffersfrom similar privacy concerns because input actions are eas-ily observable.

To permit private input, recent research proposes subtle inter-actions [2, 10, 15], which are based on implicit movementsand generally considered socially acceptable. For example,muscle interface [10] allows input through unobservable mus-cle movement. Foot gesture [12] detects subtle foot motions.Ring devices [1, 9] and fabrics [6] have been developed tosupport tap, spin, and slide inputs. Although these methodsallow subtle inputs (and thus allow for privacy and social ac-ceptability), they generally suffer from limited input space.

This paper presents FingerPad, a nail-mounted device thatturns pinched fingertips into a touchpad, allowing private,and subtle interactions. As illustrated in Figure 1, the usertreats the tip of their index finger as the touchpad, and theirthumb as the touch stylus. FingerPad enables touchpad func-tion using magnetic tracking, by adding a magnet and Hallsensors on the fingernails. Functionally, FingerPad resemblesa touchpad that users can easily learn to use. Allowing for 2Dtouch input, FingerPad is suited for rich interactions, includ-ing pointing, menu selection, and stroke input.

Sensing UIST’13, October 8–11, 2013, St. Andrews, UK

255

WonderLens

WonderLensTUI on

Printed Paper[Liang et al. CHI 2015]

FingerPadWearable

Private Input[Chan et al. UIST 2013]

FingerPad

GaussSense

WonderLens: Optical Lenses and Mirrors for Tangible Interactions on Printed PaperRong-Hao Liang, Chao Shen, Yu-Chien Chan, Guan-Ting Chou, Liwei Chan, De-Nian Yang, Mike Y. Chen, and Bing-Yu ChenNational Taiwan University and Academia Sinica

http://gausstoys.com

$19 GaussSense is coming soon!subscribe this information on:

The 19-dollar GaussSense is coming soon.We have worked very hard on it, and really excited to see this finally happens.If you are interested in, please subscribe us on GaussToys.com. So we can keep you in the loop.Thanks for your attention, and I’m happy to take all of your questions.



Date post:	30-Jul-2015
Category:	Technology
Upload:	rong-hao-liang
View:	546 times
Download:	3 times

CHI'15 - WonderLens: Optical Lenses and Mirrors for Tangible Interactions on Printed Paper

Technology