Optical Sensing: 1D to 3D using Time-of-Flight Technology
Shaping the Future of MEMS & SensorsSeptember 10, 2013
Optical Sensors Intro
Time-of-Flight Technology
ToF for 1D ranging
ToF for 3D gestures
Next Steps
Agenda
Marc Drader
Presentation
2
STMicroelectronics Imaging Division 3
Camera
Modules
Fixed focus cameraWafer Level re-flowable cameraEDoF cameraAuto-focus cameraInnovative optics, assembly & test technologies
Image
Sensors
Production from 1.4um to 5.6um pixel1.1um developmentFrom VGA to 24Mpix
Imaging
Processors
Stand alone ISPFull ST video pipe IPIntegration of third party IP on demand
Photonic
Sensors
User detection, Proximity, ALS, Optical navigation, Man Machine Interface, Automotive, Medical
Brief Overview: Optical Sensors 4
• May seem obvious but…
• Optical path considerations
• Transmission spectrum
• Transmission path efficiency
• Field of view
• Target object characteristics
• System considerations
• Optical crosstalk!
• Ambient or background illumination (noise)
Proximity Detection 5
• Conventional IR sensor �
• Attempt to detect whether object/user is near or far, based on reflected signal amplitude
• � impossible to know object DISTANCE
• Amplitude• Signal & Noise
1 Output
Noise
Far
Near
Threshold setting will detect user anywhere from 0.5 to 6cm
• Target Distance• Target Reflectance
2 Unknowns
90%
17%
3%
What is “Time-of-Flight” Sensing? 6
Active Illumination system:
1. Emit light (photons) towards a target
2. Light (partially) reflects from the target
3. Sensor determines “when” light (photons) arrive
Photon travel time multiplied by speed of light = distance
• 1cm = 66ps round-trip travel time at the speed of light
EmitterSensor
Target object
Photon(s)
Single photon travel time
Distance
Photon travel time NOT affected by target reflectance
Motivation for ToF
• Real-world application/need
• Best example: Smartphone proximity sensor detects user’s head during a phone
call; shuts off touchscreen & display
• But… it doesn’t work 100% of the time
• Search: “face hang-up” + any smartphone brand, to find frustrated users whose touchscreen did not shut off before their cheek pressed a button
• Time-of-Flight technology adds value by providing true, accurate distance measurements
• Independent of target object reflectance
• Immune to ambient illumination & optical path variations
(glass, plastic cover)
8
9Motivation for ToF
Dark hair
SkinS
ma
ll d
isp
lace
me
nts
Threshold
Threshold
Far(screen on)
Near(screen off)
Far(screen on)
Near(screen off)
• Conventional IR sensor bouncing between far and near
states vs robust ToF solution
• Photon travel time NOT affected by the object reflectance
Reflected power
(Conventional PS (*) )
ST ToF distance measurement
(*) Reflected power information also available on ST ToF Proximity Module
Time-of-Flight Physics
• Emit and receive photons :
• ….follow Poisson distribution
• …may be correlated or uncorrelated (ambient, dark current) to emitter
• Finally � photon arrival rate does depend on object reflectance & distance
10
= returned photons (correlated)
= ambient photons (uncorrelated)Emitted pulse
Delay we want to measure
Received pulse
(delay=distance)
���� Many repeated pulses required for correlation
Single Photon Avalanche Diode
• SPAD digital output used to:
• Count arrival of single photons and/or
• Time arrival of single photons
• Unique Properties
� each photon provides valuable time/distance info
• Fully Integrated in CMOS
11
More Time-of-Flight Challenges
• Optical constraints
• Coverglass contributes optical crosstalk (shortcut from emitter to sensor)
• Ambient light is main contributor of uncorrelated photons
• Co-existence of visible & NIR systems for ALS & ranging
• Conflicting wavelength and field-of-view requirements
12
Ambient light
Sensor field of view
IR emission
Phone Window
Airgap
Optical Crosstalk 13
• Time-of-Flight Sensor always “sees” two targets:
• Product-level cover (glass/plastic)
• fixed distance, and (relatively) fixed optical characteristics
• distorts reflected signal in both time & amplitude domain
• Target object
• varying distance and optical characteristics
EmitterSensor
Target object
Photon
�FlightSense technology compensates for optical crosstalk automatically
�Opens up use cases in very challenging optical environments
Crosstalk Compensation
• Compensation algorithm
• Firmware uses known crosstalk characteristics to correct the time-domain measure
• Simple register write (absolute value of photons from emitter to sensor coupled
through phone housing)
14
Raw range results (no compensation applied)
Raw range results (no compensation applied)
Crosstalk compensation applied (register setting)Crosstalk compensation applied (register setting)
Ambient Immunity
• System performance
• Keep ambient photons out
• Optical filtering (notch around 850nm)
• Reject remaining ambient photons
• Time-domain rejection
• System-level noise management
• SNR limit
15
�FlightSense technology will NOT report false distance in high ambient light conditions
• Ranging specifications
• 0 to 100mm, 3% to 90% reflectance
• 0 to 250mm for a 45% reflectance target (i.e.. Human hand)
• Eye-safe, low power IR (850nm) emitter
• Accuracy: σ = 3mm (resolution = 1mm steps)
• FlightSense architecture allows “zero” mm measurement
• * 0mm defined at the product/system-level
• There must be an available emitter�sensor optical path!
Ranging Conditions 16
Simple Optical Module 17
• Simple (reflowable) package
• Small size (2.8 x 4.8 x 1.0mm)
• Integrated emitter/sensor & optics/filters
• Opposing requirements
• Proximity: near infra-red wavelengths, narrow field of view
• Ambient Light Sensor: visible wavelengths, wide field of view
• Device delivered full calibrated
• Simple electrical integration
• Single power supply (2.8V)
• I2C & GPIO (1.8V or 2.8V)
• Programmable I²C address
• Flexible window & threshold interrupts
VL6180X Ranging Performance 18
Reflective charts (in %):
10x measurements per chart, 10mm step, in the dark,
0.2mm air-gap, no gasket, Oval artwork (75%>800nm)
• Ranging performance is independentof target reflectance/color
• Distance standard deviation < 3mm
Ranging performance is independent
of target reflectance/color
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 10 20 30 40 50 60 70 80 90 100
Co
nv
erg
en
ce T
ime
(u
s)
Target Object Distance (mm)
Convergence Time
3% 5% 17% 88%
VL6180X Convergence Time 19
Reflective charts (in %):
Distance (mm) vs Convergence Time (us)Target reflectance from 3% to 88%
0.00
0.50
1.00
1.50
2.00
0 20 40 60 80 100
Av
era
ge
Cu
rre
nt
(mA
)
Target Object Distance (mm)
Average Current Consumption10Hz repetition rate
88%
17%
5%
3%
Low Power Consumption
• Real-world current consumption
• Varies with object distance & reflectance
• Max consumption set by user
• Examples (2.8V supply)
• 10Hz ranging, object held @ 5cm
• 88% (white): 40µA
• 18% (grey): 200µA
• 5% (black): 550uA
• 3% (deep black): 760uA
• 1Hz ALS, 100ms integration
• ALS: 32uA average
• Low standby current
• HW standby <1uA
• SW standby <7uA
20
Conv. time
Conv. time
Peak current consumption
New FlightSense™ VL6180X Sensor 21
Fast, Accurate Distance Ranging• Independent of object reflectance (color)• Ambient rejection (sunlight, etc)• Phone window “crosstalk” compensation• Enable creative use cases (1D gesture application)
Disruptive Time-of-Flight Technology• 6 years of R&D• Key patents for innovative sensor/system architecture• Differentiating, unique technology• Manufactured in ST’s custom process
High-sensitivity ALS• “Invisible” for Industrial Design• Ultra-wide dynamic range• Calibrated output value in Lux
Simplified Integration & Manufacturing• Small reflowable module with embedded light emitter• No additional optics or gasket• No phone-to-phone calibration required• Robust to phone glass manufacturing dispersion • Robust to phone drop / minimize field return• Robust supply chain with dual sourcing strategy
Production in H1 2014
Product Readiness
• Mass Production in H1/2014
• CP code : VL6180XV0NR/1
• Not just for mobile phone applications
• Robust proximity detection
• Consumer robotics
• Gaming
• And much more!
• SPAD/ToF potential applications are endless
22
Check out our page on ST.com
ToF for Gestures: Motivation
• Multiple outputs eliminate ambiguity for gesture detection
23
Up/Down
Swipe
Distance
Distance
Amplitude
Amplitude
Multiple ToF sensor
Reduce ambiguity with more info:
• Order
• Position
24
1 device gestures capabilities : 2 devices gestures capabilities :
3D Gesture Detection
• High potential for differentiation
• Setting Expectations• Unlike a touchscreen
• No “touch” or “release” – Detecting user intention more difficult • No physical boundary
• “Live” interaction vs post-processed result
�Optical 3D gestures can complement existing systems• Off screen/over-screen sensing volume
• New uses cases • Wakeup or UI response as user approaches• Hands-free interaction (many ideas)• Gaming controller
25
1D Gesture Detection 26
• Time domain• Distance
• Amplitude• Signal & Noise
• Time of Flight IR2 Outputs
• Distance
• Reflectance• Surface
• % fill factor
• Multiple objects
2 Unknowns2 Unknowns• Object Properties
• Distance measurement alone
• Tap / double-tap
• Up/down level control
• IF we assume fixed object reflectance
• Lateral motion can be estimated from a single pixel!
Lateral Motion Estimation
• Assume
• Surface properties are stable (same object) within a given time period
• � only change must be due to % filled FOV (represents x-y motion)
• We can therefore calculate the % of the Field of View filled by the object
• Independent of object distance!
27
20% 80% 100% 80% 20%
Amplitude can be normalized (using distance info)
% Field-of-View Coverage
• Object reflectance modeled at all distances
• Model needs to include non-linearities
28
Sensor saturation
Emitter blocked
3D Gesture Detection 2 ToF Pixels 29
• Spatially or angularly separated ToF detectors
• Linear continuous slider
• Smooth triangulation of position/speed in X (horizontal) and Z (vertical)
Gesture Definitions 4 ToF Pixels
• Continuous control � hover & tilt
• Hover & tilt
• Great gameplay
• Can act as mouse/track pad
• Post-processed movement examples
30
1 2
1
2
Swipe Press Wave
More Gesture Definitions
• Movement properties that can be detected
• Motion lateral/angular speed
• Object width
• Closed vs spread fingers
• Hand-tilt detection
• Goal is for ROBUST detection of gestures/motion
31
Flat hand swipe
4 fingers swipe
Tilted hand swipe
Tilted hand swipe
What’s Next?
• 3D Gestures is a wide field � more to come!
32
33
Q&A