1
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
ACIVS 2011August 22-25, Ghent, Belgium
Peter Meijer
Metamodal BVThe Netherlands
www.artificialvision.com
Camera-based sensory substitution
and augmented reality for the blind
0%
© 2011 Peter B.L. Meijer
2
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
cameraglasses
2%
2
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
3
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
4%
© 2011 Peter B.L. Meijer
4
Computer vision & vision technology
for the blind
Computer vision, thinking of…
- contrast enhancement, dehazing (cataracts) - color identification / filtering
- edge detection / enhancement (low acuity vision)
- OCR (Optical Character Recognition)- object detection / recognition (SIFT, SURF, …)
- collision threat analysis (e.g. optic flow)- saliency detection / enhancement - SLAM (Simultaneous Localization and Mapping)
6%
3
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
5
Computer vision & vision technology
for the blind
Goals
- improve blind orientation and mobility (O&M)through obstacle detection, navigation aids,object recognition (also crowdsourcing), …
- offer “sight” through sensory substitution(convey live camera views by sound or touch)
[also hybrid approaches and augmented reality]
9%
© 2011 Peter B.L. Meijer
6
Electronic glasses…
Image enhancement visually impaired
Bristol Mobility Aid - University of the West of England, Bristol, UKMobility performance of low-vision adults using an electronic mobility aid, Jones et al., Clinical and Experimental Optometry 2006 89:1 pp 10-17
11%
4
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
7
Robotic navigation aid for the blind?
SLAM (Simultaneous Localization and Mapping), USC on NTDTV July 4 2011
URL
James Weiland, USC Viterbi School of Engineering
13%
© 2011 Peter B.L. Meijer
8
Computer vision & vision technology
for the blind (news headlines in 2011)
- few technologies ever make it out of the lab- cost of commercialization, bulky, unreliable,physical limitations (Kinect fails in sunshine), …
EYE21- Universitat Politècnica de València, Spain
NAVI - Universität Konstanz, Germany
Kinect for the Blind - Moscow Institute of Physics
& Technology, Russia (MS Imagine Cup 2011)
MS Kinect
Stereo vision
More examples KinectFusion next?
15%
5
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
9
How to give sight to the blind?
ubiquitous
cameras
mobile
computing
brain
plasticity
augmented
reality
computer
vision
when conventional
medical approaches
do not suffice…
18%
© 2011 Peter B.L. Meijer
10
How to give sight to the blind?
The vOICe @ APA2011Michael Proulx
Preview
20%
6
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
11
gene therapy
retinal implants
cortical implants
LGN implants
sensory substitution
How to give sight to the blind?
when conventional
medical approaches
do not suffice
22%
© 2011 Peter B.L. Meijer
12
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
cameraglasses
25%
7
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
13
Sensory substitution
Tactile or auditory display
Wicab’s BrainPort® The vOICe
ear buds
camera
tongue display
camera
27%
© 2011 Peter B.L. Meijer
14
The vOICe image-to-sound mapping
Fre
quency
Fre
quency
TimeTime
Spectrographicsynthesis
29%
8
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
15
Could this really work?
Check: map sounds back to images
The vOICe image-to-sound mapping
Yes!(at least technically)
Theory
31%
© 2011 Peter B.L. Meijer
16
The vOICe image reconstruction
“Soundscape”
34%
9
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
17
The vOICe mental imagery?
Warning:
Representationmay dramaticallyaffect recognition!
36%
© 2011 Peter B.L. Meijer
18
The vOICe demo sounds (1 of 5)
Fre
quency
Fre
quency
TimeTime
38%
10
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
19
The vOICe demo sounds (2 of 5)
Fre
quency
Fre
quency
TimeTime
40%
© 2011 Peter B.L. Meijer
20
The vOICe demo sounds (3 of 5)
Fre
quency
Fre
quency
TimeTime
43%
11
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
21
The vOICe demo sounds (4 of 5)
Fre
quency
Fre
quency
TimeTime
45%
© 2011 Peter B.L. Meijer
22
The vOICe demo sounds (5 of 5)
Fre
quency
Fre
quency
TimeTime``Slow motion’’
Informational
masking? Apply
mental saccades
47%
12
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
23
Setups for The vOICe cost < $1000,
or < $500 when using a netbook PC
Michael Proulx
50%
© 2011 Peter B.L. Meijer
24
Grasping exercise (can be mastered in a few hours)
52%
13
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
25
Grasping exercise (video clip from PLoS ONE 2008)Source: Proulx MJ, Stoerig P, Ludowig E, Knoll I, 2008 Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution. PLoS ONE 3(3)
54%
© 2011 Peter B.L. Meijer
26
[early blind] Metamodal LLC, California: NSF-fundedSBIR Phase I project with The vOICe, Pasadena 2010
http://www.youtube.com/playlist?list=PLDE409461FD43A4D056%
14
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
27
[late-blind] Metamodal LLC, California: NSF-fundedSBIR Phase I project with The vOICe, Pasadena 2010
59% http://www.youtube.com/playlist?list=PLDE409461FD43A4D0
© 2011 Peter B.L. Meijer
28
[late-blind] Wicab BrainPort® vision device, Pittsburgh 2011
Benchmarking?
61%
15
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
29
The vOICe
What do blind users see?
> In the top right hand corner of the picture is something which I
> recognise as architecture. It has 3 light coloured vertical columns
> below a sloping roof or sloping structure at the front above the
> columns. It is surrounded in shadow, especially on the right.
>
> Given that this is an architectural feature and not something smaller,
> that must mean that the vegetation on the top left of the picture is one
> or several trees.
> In the bottom left hand corner is 3 short dark columns and above it a V
> shaped band. These columns are shorter than the trees. To the right of
> the small columns are several short and long horizontal lines on a
> bright short band. I don't think they are columns but not sure what
> they are.
Photograph: courtesy Michael Larsen
63%
© 2011 Peter B.L. Meijer
30
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
65%
16
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
31
PF, late blind female, US, after hearing about visual cortex implants:
Just sound?.... No, It is by far more, it is sight ! There IS true light preception generated by the vOICe. When I am not wearing the voice the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image. True it is not color but it is definitely like looking at a black and
white TV show. The light generated is very white and clear then it erodes down
the scale of color to the dark black. I don't really see adiffrence in this light as compaired to the "light phosphenes " they are talking about.
The vOICe
Artificial synesthesia?
See also: Ward, J. and Meijer, P., ``Visual experiences in the blind induced by an auditory
sensory substitution device'', Consciousness and Cognition, Vol. 19, 2010, pp. 492–500.
68%
© 2011 Peter B.L. Meijer
32
Philosophy and The vOICe
What does
that mean?
What does
that mean?
17th Century 17th Century ““MolyneuxMolyneux problemproblem”” revisited?revisited?
•• Q:Q: Would someoneWould someone born blind, born blind, in case sight was restored,in case sight was restored,
be able to be able to tell a cube from a sphere by sighttell a cube from a sphere by sight alonealone??
A:A: ““NoNo”” ((according to John Locke & William according to John Locke & William MolyneuxMolyneux))
•• Q:Q: Would someoneWould someone born blind, born blind, using using ““seeingseeing--withwith--soundsound””,,
be able to be able to tell a cube from a sphere by tell a cube from a sphere by ““sightsight”” alonealone??
A:A: ““YesYes””
70%
17
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
33
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
72%
© 2011 Peter B.L. Meijer
34
Neuroscience:Sound can activate “visual” cortex of blind
Kujala, Teija et al. (1995) “Visual cortex activation in blind humans during sound discrimination,” Neurosci. Lett. 183, 143–146
Early-blind person detecting pitch changes
75%
18
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
35
The vOICe in Neuroscience 2007:Cross-modal plasticity & visual sounds
Amedi et al., Nature Neuroscience, June 2007:
Shape conveyed by visual-to-auditory sensory substitutionactivates the lateral occipital complex
LOtv V1
dx.doi.org/10.1038/nn1912
Late-blind expert user of The vOICe(Talairach normalized inflated cortex reconstruction)
Percent signal change analysis of LOtv
LOtv acts as a metamodal operator for shape
77%
© 2011 Peter B.L. Meijer
36
Harvard Medical School using The vOICe, Boston 2007
The Nature of Things
CBC television 2008
79%
19
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
37
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
81%
© 2011 Peter B.L. Meijer
38
Augmented Reality
Camera view overlays
Smartphone-based
visual overlays
• Wikitude
• Layar
• …
For those without eyesight,
a sonic overlay: The vOICe
Layar Wikitude
84%
20
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
39
86%
© 2011 Peter B.L. Meijer
40
live soundscapes
talking locator (GPS)
talking compass
color identifier/filters
tactile feedback
foveal enlargement
The vOICe for Androidaugmented reality for the blind
voice commands multilingual (Russian, Spanish, Chinese, …)
barcode reader Google Goggles
88%
21
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
41
The vOICe for Android
Google Analytics & mobile vision
Android Market
niche, yet in August 2011: 90,000+ total installs
4-week use: 135 countries
90%
© 2011 Peter B.L. Meijer
42
Augmented Reality
Hands-free future?
Ralph Osterhout, EG 2011
93%
22
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
43
Netbook + $30 USB camera glasses + optional $20 wide-angle lens
Back to reality: what blind users of The vOICe are using today
95%
© 2011 Peter B.L. Meijer
44
Camera-based sensory substitution
and augmented reality for the blind
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
Overview
• introduction
• sensory substitution
synesthesia & philosophy
• brain plasticity
• augmented reality
• conclusions
97%
23
ACIVS 2011 - Camera-based sensory substitution and augmented reality for the blind
E-mail: [email protected] (Peter Meijer) URL: http://www.artificialvision.com
© 2011 Peter B.L. Meijer
45
Conclusions:
� Research ongoing
� Affordable
� Non-invasive
� Available worldwide
http://http://www.artificialvision.comwww.artificialvision.com
Camera-based sensory substitution
and augmented reality for the blind
with thanks to all blind users and research partners around the world!100%