Post on 15-Jan-2016
transcript
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Ramesh RaskarMitsubishi Electric Research Labs (MERL)
Cambridge, MA, USA
Special Effects in the Real WorldSpecial Effects in the Real World
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Parent Organization: Parent Organization: MELCOMELCO
• Mitsubishi Electric (MELCO), including subsidiaries– $32B in annual sales– 125,000 people– 10 business sectors:
• Space Development and Satellite Communications• Communications and Information Processing• Visual Equipment and Systems• Electronic Devices• Home Electronics• Energy• Industrial Equipment• Public-Use Systems• Transportation
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
MERLMERL
• Mitsubishi Electric Research Laboratories– Dick Waters, President & CEO
• Two labs, co-located in Cambridge, MA:– MERL Technology Lab
• Advanced systems development• ~35 researchers
– MERL Research Lab• Joe Marks, Director• Basic research in applied computing• 23 researchers, interns, consultants• CV, HCI, Multimedia, CG, AI
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Special EffectsSpecial Effects
– Photorealism and Animation •Stays on screens•Yet to affect daily life in real time
– Post-processed content– Unlike internet,databases, UI
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Changing AppearanceChanging Appearance
Virtual light source
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Changing Virtual IlluminationChanging Virtual Illumination
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Special Effects in the Real WorldSpecial Effects in the Real World
– Photorealism and Animation •Limited to display screens•Yet to affect daily life in real time
– Unlike other fields of info-tech
– Fusion: real world with graphics•Next big challenge in CG ?•New opportunities in advt, gaming, animation•Believable co-existence
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Changing AppearanceChanging Appearance
Projector Projector
Virtual light source
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Changing Virtual IlluminationChanging Virtual Illumination
Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Virtual MotionVirtual Motion
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Virtual MotionVirtual Motion
Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Dynamic AugmentationDynamic Augmentation
Projecting on Tracked Objects
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
ShaderLampsShaderLamps
www.ShaderLamps.com
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Projector-based AugmentationProjector-based Augmentation
www.ShaderLamps.com
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Poor Man’s PalacePoor Man’s Palace
Surround CG for Daily Environment
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Maya: World is an IllusionMaya: World is an Illusion
• Fake World– We all live in one
• Real-life Fakes– Not just photos and videos but surroundings
• Privacy
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Augmented Reality (AR)Augmented Reality (AR)
With See-through Head-mounted Display
© Andrei State
Superman X-ray vision
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
hand-helddisplay
spatial see-throughdisplay
head-mounted display
retinal display
Head-attached Body-attached spatial
projector
projector
projector
Spatial Augmented Reality
Classification of Augmented RealityClassification of Augmented Reality
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Being ThereBeing There
• Recreate Large Environments– ‘BeingThere’, walk-around– Museums, Exhibitions, Training
Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Desired Virtual Model
© Andrei State
Projected Guidance for Placement
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Head-mounted Display Projectors
© Andrei State
Augmented RealityAugmented Reality
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
OpportunitiesOpportunities
• Entertainment- Animating objects, Museums, Special Effects, Laser Shows
• Advertising- Car showrooms, Furniture store
• Design and Ergonomics- Prototyping products, visualize material changes
• Information Augmentation- Office, factory or assembly Training
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Spatially Augmented Reality Spatially Augmented Reality (SAR)(SAR)
HMD-VR Spatially Immersive-VR
AR using HMDSpatially Augmented
Reality
e.g. CAVE
Video or Optical see-through
SAR, Shaderlamps
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
StepsSteps
– Preprocessing• Scan 3D object• Roughly align projectors• Compute correction
– Run time• Render images of 3D model• Correct
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
StepsSteps
– Preprocessing• Scan 3D object and create virtual model
Faro arm
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
StepsSteps
– Preprocessing• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
StepsSteps
– Preprocessing• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose
– Run-time• Inverse Rendering of 3D Models
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Shape cue allows perceptual unwrappingShape cue allows perceptual unwrapping
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
‘Wallpaper the surface’‘Wallpaper the surface’
Shape Adaptive ProjectionShape Adaptive ProjectionMinimum Stretch DisplayMinimum Stretch Display
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Shape Adaptive ProjectionShape Adaptive ProjectionMinimum Stretch DisplayMinimum Stretch Display
Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,
SIGGRAPH 2003
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Spatially Augmented RealitySpatially Augmented Reality
Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,
SIGGRAPH 2003
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Mitsubishi Pocket ProjectorMitsubishi Pocket Projector
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
A new way to play tic-tac-toeA new way to play tic-tac-toe
• Handheld projector.
• Grid projects to a fixed position on the wall.
• Cursor is guided by pointing the projector.
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Pocket Projector GamesPocket Projector Games
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
WarehousingWarehousing
RoutingRouting
Library Library
Baggage
handling
Baggage
handling
CurrencyCurrency
Livestock trackingLivestock tracking
RFID
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Micro Controller
Memory Computer
READER
Micro Controller
Memory
Conventional Passive RFID
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Tagged Books in a LibraryTagged Books in a Library Id : List of books in RF range
No Precise Location Data Are books in sorted order ?
Which book is upside down ?
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Prototype TagPrototype Tag
RF tag + photosensor
RF tag + photosensor
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Find tag location using
handheld Projector
Find tag location using
handheld Projector
Photosensing
Wireless Tags
Photosensing
Wireless Tags
Many geometric ops Many geometric ops
R F I R F I
R F I DR F I D
Interactive stabilized projection
Interactive stabilized projection
(Radio Frequency Id & Geometry)(Radio Frequency Id & Geometry)
GG
Siggraph 2004Siggraph 2004
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
AR with Photosensing RFID AR with Photosensing RFID and Handheld Projectorand Handheld Projector
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Visual feedback of 2D positionVisual feedback of 2D position
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
RFID
(Radio Frequency Identification)
RFIG
(Radio Frequency Id and Geometry)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
CurvedPlanar Non-planar
SingleProjector
MultipleProjectors
Projector
jUser : T
?
Pocket-ProjObjects
Projectors and SurfacesProjectors and Surfaces
2000
2000 2002
2001
1999
2002
20031999
1998
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Augmentation of “five senses”Augmentation of “five senses”
• Visual Displays• Auditory Displays• Haptic Displays• Locomotion Interfaces• Olfactory Displays• Taste Displays ?• Sixth Sense ..
Extendingsensorymodalities
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Sense of BalanceSense of Balance
Galvanic Vestibular Stimulation
Masahiko Inami (UEC)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Scent DisplayScent Display
Tracking nose direction
Aiming atthe nose
Air cannon
Deliveringscented air byvortex rings Camera
Platform
Yasuyuki Yanagida, ATR
Spatio-Temporal Controlled Delivery of Scent
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Air CannonAir Cannon
Generating a doughnut-shaped vortex ring
Aperture
Push
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
DisplaysDisplays ContendersContenders
OrganicLED
LightEmittingPolymers
E-Ink
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
HolodustHolodustPerlin et alPerlin et al
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Fake WorldFake World
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Poor Man’s PalacePoor Man’s Palace
Surround CG for Daily Environment
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
RecapRecap
• Special Effects in Real World– Photorealism yet to impact daily life– Poor Man’s Palace
• Spatial Augmented Reality– Solution for fusion: Projectors, RFID, Sensors– Games for pocket projector– Display anywhere for advertising
• Future Problems– Natural phenomenon, complex reflectance, other displays– All senses: haptic, olfactory, preprioception
• Next Challenge in Real World Animation– Movies and games in our surroundings– 3D Advertising
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Spatial Augmented RealitySpatial Augmented Reality
• Graphics– Photorealism yet to impact daily life– Special effects in real world
• Spatial Augmentation– Solution for fusion– Needs hardware components– Projectors, RFID, Sensors
• Next Challenge in Animation
raskar.com