Continuous 360 degree real-time 3D laser...

09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 1

Continuous 360 degree real-time 3D laser scanner

Marko Reimer, Oliver Wulf, Bernardo Wagner

Research Centre L3S andInstitute for Systems Engineering

University of Hannover


Motivation

Localization for mobile robots

Path planning for mobile robots

Environment modeling for mobile robots

Goals: Unrestricted autonomous navigation and 3D map building in real-time

3D visualization for human operators


Why use 3D data for navigation?

Real indoor environments are often cluttered

Good landmarks like walls are often partially occluded and thus invisible for 2D sensors

Obstacles not within the 2D scanline may exist

Negative obstacles can not be detected in 2D scans


Ways to reach the 3.rd dimension

4 Alternative scanning methods


Main components of the used hardware

ScanDrive (rotating platform)Capable of continuous rotationNo accelerations

- Less mechanical load- Less power consumption

Data- and Power-Transfer via slip rings

Scalable Processing Box (SPB)Embedded PC running “LiRE” (Linux Realtime Environment) as real-time operating system


Result of real-time synchronisation

3D-laser scans consist of data fusion from multiple sensors

(2D laser scanner data, laser scanner orientation and robot position)

Systematic errors can be avoided by exact synchronization

Poor adjustment Good adjustment


Software / Sensor architecture

3D Scanner

2D Scanner Rotating plate Odometrie

Chassis3D Gyro


Real-time data~32.000 points in ~2.4 sec

Two main aspects encourage the use of a real-time operating system

All gathered data is marked with a synchronized timestamp

Later and offline processing is possible while keeping the data synchronized

Data can be gathered during movement

The data processing is always done within a specified time

The system acts like a sensor ->no higher-level system computations needed

Resulting data can be used for time critical operations


Visualization of the 3d data I

As depth image (gray values representing distance)


Visualization of the 3d data II

As 3D point cloud (already with colored objects)


3D Perception and 2D Maps

3D map representations

Registered 3D point clouds

Digital elevation maps (2½D)

Computational expensive

Combination of 3D perception and 2D map representation by use of Virtual 2D Scans

Full 3D information is not needed for robot navigation

Existing 2D navigation algorithms work more efficient


Complexity reduction

Map building requires landmarks (2D)

Path planning requires obstacle information. (2D)

=> Two “different maps” needed

Red points are detected landmarks

(virtual landmark scan)

Blue points are detected obstacles

(virtual obstacle scan)


Virtual 2D Scan Real Environment

Virtual 2D Scan3D-point cloud


Designed a 3D laserscanner

For use on a mobile platform

Delivering 3D data in real-time

Selectable point density

Developed an

Fast

Efficient

Robust

algorithm for navigation of a mobile system.

Conclusion

……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange

Robust 3D Measurement with PMD Sensors

“CCD goes 3D !”1st range imaging research day

8 September 2005, Zurich

CMOS

PMDTechnologies GmbHAm Eichenhang 50D-57076 Siegen

phone + 49 271 238538 -800fax + 49 271 238538 [email protected]


TheThe CompanyCompany

THE COMPANYTHE COMPANY


Company Profile: PMD Technologies GmbHCompany Profile: PMD Technologies GmbH

- Founded in 2002

- Employees: 20+

- Shareholders: - Audi AG (50%) - ifm electronics (50%)

- Worldwide patents

- Markets: Automotive, Industrial and consumer.

- 1st all-solid-state-3D series-product worldwide

- Camera and chip prototypes available for evaluation of 3D technology

- Automotive: camera systems in field test with several OEMs.

- Awards:- „German future award“ 2002 nominee- Hermes award 2005 for “efector PMD”

http://www.nv.et-inf.uni-siegen.de/inv/akt_zp.htm


BASIC PRINCIPLEBASIC PRINCIPLE


What makes 3DWhat makes 3D--PMD so smart?PMD so smart?

CCD or CMOS 2D-Imagers“Each pixel is a photodiode”

PMD“Each pixel is a complete distance

measurement system”


TTimeime--OOff--FFlight Basic Principlelight Basic Principle

SENDER

RECEIVER

00:00:00

Start

Stop

Distance is measured by determining the turn- around time a light pulse needs to travel from the sender to the target and back to the receiver. With knowledge of speed of light the distance can easily be calculated.

Distance is measured by determining the turnDistance is measured by determining the turn-- around time a light pulse needs to around time a light pulse needs to travel from the sender to the target and back to the receiver. Wtravel from the sender to the target and back to the receiver. With knowledge of ith knowledge of speed of light the distance can easily be calculated.speed of light the distance can easily be calculated.


3D TOF3D TOF-- CameraCamera

Chip measures both intensity and distance (arrival time of light) in each pixel 3D-camera

Chip measures both intensity and distance Chip measures both intensity and distance (arrival time of light) in each pixel (arrival time of light) in each pixel 3D3D--cameracamera


PMDPMD-- Principle: Basic Cross SectionPrinciple: Basic Cross Section


PMDPMD--operationoperation

d=0mPopt_sent

Popt_received

mod_A mod_B

Q_a Q_b

mod_A mod_B

Q_a Q_b

Mod_A, Mod_B A B

Q_A, Q_BA

B

d=2mPopt_sent

Popt_received

mod_A mod_B

Q_a Q_b

mod_A mod_B

Q_a Q_b

Mod_A, Mod_B A B

Q_A, Q_BA

B

d=3.75mPopt_sent

Popt_received

mod_A mod_B

Q_a Q_b

mod_A mod_B

Q_a Q_b

Mod_A, Mod_B A B

Q_A, Q_BA

B

d=5mPopt_sent

Popt_received

mod_A mod_B

Q_a Q_b

mod_A mod_B

Q_a Q_b

Mod_A, Mod_B A B

Q_A, Q_B

AB

d=7.5mPopt_sent

Popt_received

mod_A mod_B

Q_a Q_b

mod_A mod_B

Q_a Q_b

Mod_A, Mod_B A B

Q_A, Q_B

A

B


Analysis of AKF (autocorrelation function)Analysis of AKF (autocorrelation function)

λ = 15 m fmod = 20MHz

τ ~ ϕ

Α1 Α2 Α3 Α4Α4

b

a

ϕ

AKF

Signal Phase ϕ: distanceSignal Phase Signal Phase ϕ: ϕ: distancedistance

Amplitude a: signal strength of active illumination: quality of 3D measurement.

Amplitude a: Amplitude a: signal strength of active illumination: signal strength of active illumination: quality quality of 3D measurement.of 3D measurement.

Offset b: gray scale imageOffset b: Offset b: gray scale imagegray scale image


Key Innovation of PMDKey Innovation of PMD

1. Non-scanning 3D Arrays.Highly integrated Arrays of individual Range Finding Pixels possible. Low-cost, Robust

2. Classical TOF Range Finders drastically simplified.Complete Range Finder realized in quasi-optical Domain.No HF- Signal Processing.


CHALLENGESCHALLENGES


CHALLENGESCHALLENGES

Need to measure picoseconds. (What is a picosecond?)How much light do I need for a certain resolution?What impact has background light (sun)?

How bright is the sun?What are the problems?


What is a What is a picosecondpicosecond??

speed of light is 3E8 m/s = 2mm/6.67psTOF: 6.67ps resolution required for 1mm precisionspeed of light is 3E8 m/s = 2mm/6.67psTOF: 6.67ps resolution required for 1mm precision

1s 1E-3s 1E-6s 1E-9s 1E-12s

1s 1/1000s 300ps 6.67ps(3GHz) 1mm

1000 1000 1000 1000


Predictable measurement precisionPredictable measurement precisionk: contrastS: # signal electronsk*S: # active (demodulated) electronsN: # noise electronsλmod: “length of modulation wave” (=c/fmod)

π⋅λ

⋅⋅

⋅=8N/Sk

1N

1dR mod

phase

What do we need for a high distance accuracy (low dR) ?

k: high high contrast PMD-pixels (+bandwidth), high contrast light source

S: high strong light source, large lens aperture, high pixel sensitivity (good fill factor, large pixels)good reflecting targets, close targets.

N: low low shot noise, low background light, low dark current low camera noise (system noise)

λmod: low (fmod high) high bandwidth.


Predictable measurement precisionPredictable measurement precision

π⋅λ

⋅⋅

⋅=8N/Sk

1N

1dR mod

phase

0

0.5

1

1.5

2

2.5

3

3.5

4

50 100 150 200 250 300 350 400distance (cm)

stan

dard

dev

iatio

n (c

m)

PMD[vision] 19k MeasurementPMD[vision] 19k Theory

0

0.5

1

1.5

2

2.5

3

3.5

4

50 100 150 200 250 300 350 400distance (cm)

stan

dard

dev

iatio

n (c

m)

PMD[vision] 19k MeasurementPMD[vision] 19k Theory


Influence of sunlightInfluence of sunlight

LED illumination

For most outdoor applications sunlight is much brighter than theactive illumination. This has two bad impacts on the TOF-system:

(1) Shot noise is increased ( distance noise)

(2) The dynamics of the sensor can not handle the sunlight (saturation).

For most outdoor applications sunlight is much brighter than theactive illumination. This has two bad impacts on the TOF-system:

(1) Shot noise is increased ( distance noise)

(2) The dynamics of the sensor can not handle the sunlight (saturation).


Influence of sunlight Influence of sunlight –– simple workaroundssimple workarounds

LED illumination

Filter

Filter

LED illumination

LED illumination

Sunlight much brighter than LED illumination. Very poor SNR (active light / background light).

Sunlight much brighter than LED illumination. Very poor SNR (active light / background light).

Optical Spectral Filter:

Transmission only of LED- spectrum. Improvement of SNR by reduction (filtering) of background light.

Optical Spectral Filter:

Transmission only of LED- spectrum. Improvement of SNR by reduction (filtering) of background light.

Burst operation of LEDs:

Improvement of SNR by increase of active peak power.

Burst operation of LEDs:

Improvement of SNR by increase of active peak power.


Calculation: Sunlight vs. Active lightCalculation: Sunlight vs. Active light

LED SUNLED 0.3 nm/KTemp. Automotive min -40 °CTemp. Automotive max 120 °CTemp. Automotive range 160 K

48 nm

LED 50 nmFilter 98 nm

Viewing angle 20 deg (diagonal)Range 40 mImage Format 1:1edge of image (object space) 10.0 marea of image (object space) 99 m^2#LEDS 100 LEDsoptical power per LED 0.04 W/LEDtotal optical power of LEDs (CW) 4 Wpower density in object space (CW) 0.040 W/m^2 726 W/m^2 power density of sunpower density in object space (CW, filtered) 0.039 W/m^2 83 W/m^2 power density of sun (filtered)Burst-Operation 5 Factor

20 Wpower desnisty in object space (Burst) 0.20 W/m^2power desnisty in object space (Burst, filtered) 0.19 W/m^2


Situation 1: No Filter, No BurstSituation 1: No Filter, No Burst

0.01

0.1

1

10

100

1000

10000

400 500 600 700 800 900 1000 1100

wavelength in nm

optic

al p

ower

den

sity

in m

W/m

2/nm

sunLEDs (4W, 20deg FOV, 40m)

0.04W/m2

750W/m2

ratio sun/active = 18‘000 = 85dBratio sun/active = 18‘000 = 85dB


Situation2: Spectral FilteringSituation2: Spectral Filtering

0.01

0.1

1

10

100

1000

10000

400 500 600 700 800 900 1000 1100wavelength in nm

optic

al p

ower

den

sity

in m

W/m

2/nm

0

10

20

30

40

50

60

70

80

90

100

%

sunLEDs (4W, 20deg FOV, 40m)sun, filterednormalized spectral response of PMDspectral filter

0.04W/m2

80W/m2 (improvement of factor 9)

ratio sun/active = 2‘000 = 66dBratio sun/active = 2‘000 = 66dB

750W/m2


Situation 3: Spectral Filtering and Burst Operation of Situation 3: Spectral Filtering and Burst Operation of LEDsLEDs

0.01

0.1

1

10

100

1000

10000

400 500 600 700 800 900 1000 1100

wavelength in nm

optic

al p

ower

den

sity

in m

W/m

2/nm

0

10

20

30

40

50

60

70

80

90

100

%

sunLEDs (4W, 20deg FOV, 40m)sun, filteredLEDs (bursted, filtered)spectral filter

0.2W/m2 (factor 5)

83W/m2 (improvement of factor 9)

ratio sun/active = 400 = 52dBratio sun/active = 400 = 52dB

750W/m2

0.04W/m2


Influence of sunlight Influence of sunlight -- SummarySummary

LED illumination

Filter

Filter

LED illumination

LED illumination

ratio of sunlight / active

18‘000 = 85dB

2‘000 = 66dB

Filter:-19dB

-33dB

Burst Operation-14dB

400 = 52dB


Influence of sunlight Influence of sunlight -- SummarySummary

• Spectral Filtering and burst operation of light source are effective methods to improve „optical“ SNR.

• Reasonable sunlight rejection of these methods: -33dB.

• The remaining ratio of sunlight / active signal is still poor:

• Even the filtered sun is still a factor of 400 brighter than thepulsed LEDs.

Additional sunlight rejection methods have to be found.

• Spectral Filtering and burst operation of light source are effective methods to improve „optical“ SNR.

• Reasonable sunlight rejection of these methods: -33dB.

• The remaining ratio of sunlight / active signal is still poor:

• Even the filtered sun is still a factor of 400 brighter than thepulsed LEDs.

Additional sunlight rejection methods have to be found.


Sum up: ChallengesSum up: Challenges

BandwidthSensitivityDynamics and sunlight robustness

… are the key challenges for optical TOF measurement.


TECHNICAL SOLUTIONSTECHNICAL SOLUTIONS


Contrast Contrast –– measurementmeasurement

Bandwidth: beyond 100MHzBandwidth: beyond 100MHz


Size of PMD Pixels is scalable.Size of PMD Pixels is scalable.

4040µµmm 118118µµmm 180180µµmm 280280µµmm

Due to the scalable size of PMD pixels, good fill factors (>50%) can be achieved, even when integrating additional pixel- functionality like SBI, A2I, Multi- sampling, …

Due to the scalable size of PMD pixels, good fill factors Due to the scalable size of PMD pixels, good fill factors (>50%) can be achieved, even when integrating additional (>50%) can be achieved, even when integrating additional pixelpixel-- functionality like SBI, A2I, Multifunctionality like SBI, A2I, Multi-- sampling, sampling, ……


SBI: Suppression of background illuminationSBI: Suppression of background illumination

SBI

Working Principle of SBI:

The SBI circuitry removes the charge packets of background illumination and dark current from the readout nodes. The complete dynamic range of the readout chain can be used for the active signal.

Working Principle of SBI:

The SBI circuitry removes the charge packets of background illumination and dark current from the readout nodes. The complete dynamic range of the readout chain can be used for the active signal.

no SBI SBI


active SBI (3) active SBI (3) –– resultsresults

no SBI SBIZ10_12P1 vs

Z9_2P2 vs, sbi

01

10100

100015pA

150pA1500pA

0

10

20

30

40

50

60

70

Sta

ndar

dabw

eich

ung

[cm

]

iphot o :iH GL 1:xiphot o [pA]

Standardabweichung Tint=2ms

15pA150pA1500pA

01

10100

100015pA

150pA1500pA

0

10

20

30

40

50

60

70

Sta

ndar

dabw

eich

ung

[cm

]

iphot o :iHGL 1:xiphot o [pA]


15pA150pA1500pA


Dynamics Measurement (no SBI)Dynamics Measurement (no SBI)

10-3

10-2

10-1

100

101

102

10-3 10-2 10-1 100 101 102 103

Phasedeviation [°] vs Signal and Backlight

Backlightillumination [W/m²]

Sig

nalil

lum

inat

ion

[W/m

²]

0

2

4

6

8

10

12

14

16

18

20

saturation boundaries


Dynamics Measurement (SBI)Dynamics Measurement (SBI)

10-3

10-2

10-1

100

101

102

10-3 10-2 10-1 100 101 102 103

Phasedeviation [°] vs Signal and Backlight

Backlightillumination [W/m²]

Sig

nalil

lum

inat

ion

[W/m

²]

0

2

4

6

8

10

12

14

16

18

20saturation boundaries

SBI

SBI

Factor of 300 = 50 dB !!!

Factor of 300 = 50 dB !!!


SBI SBI -- SummarySummary

LED illumination

Filter

Filter

LED illumination

LED illumination

18‘000 = 85dB

2‘000 = 66dB

400 = 52dB

Filter:-19dB

Burst Operation-14dB

-33dB

SBI: 50dB

LED illumination

Only SBI circuitry allows robust 3d measurement under full sunlight.

Only SBI circuitry allows robust 3d measurement under full sunlight.

Filter + SBI


Technical challenges and solutionsTechnical challenges and solutions

Technical challenge

high bandwidth

PMD solution

finger structures,short modulation channels

1cm 67ps >100 MHz

high dynamics (sunlight) active SBI, (spectral filter, LED burst)

0.5-5m, r=0.05..1, 10levels 86dBplus sunlight: up to >> Factor 100

01

10100

100015pA

150pA1500pA

0

10

20

30

40

50

60

70

Stan

dard

abw

eich

ung

[cm

]

iphot o :iHGL 1:xiph ot o [pA]


15pA150pA1500pA

01

10100

100015pA

150pA1500pA

0

10

20

30

40

50

60

70

Stan

dard

abw

eich

ung

[cm

]

iphot o :iHGL 1:xip hot o [pA]


15pA150pA1500pA

50 dB

high sensitivity scalable pixel size

typical photo currents down to few 100 fA50% FF


ADVANTAGES and USPs

ADVANTAGES and USPs


Comparison to existing solutionsComparison to existing solutions

Stereo Vision• Shadowing effects• Problems with low contrast scenes• Elaborate computing required• Significant delay times for 3D

calculation• 2 complete cameras PLUS

powerful processor

Laser Scanners• Moving parts• Lots of components• Elaborate mechanics• Expensive• vibration sensitive• slow

PMD is cheaper.PMD has better performance.PMD is smaller in size.


AdvantagesAdvantages

Suppression of background illumination (SBI)Various sensor resolutions (pixels scalable in size/sensitivity)

16x16, 32x24, 64x16, 64x48,160x120Highest sensitivityHighest bandwidthHighest contrast


PRODUCTSPRODUCTS


efectorefector PMDPMD

1D Distance sensor based on PMD principle1D Distance sensor based on PMD principleSensor introduced on HMI 2005Sensor introduced on HMI 2005current pricing (2005): 248,current pricing (2005): 248,-- €€


Cameras: PMD[vision]®Cameras: Cameras: PMD[visionPMD[vision]]®®

Camera platform: FireWire; Ethernet; SC520 133 MHz CPU, FPGA Spartan II, eLinOS operating systemAvailable Sensors: 64x48 SBI, 64x16 SBI, 160x120 (SBI: Suppression of background illumination)

PMD [vision]®


SENSORS: SENSORS: PhotonICsPhotonICs®® PMD PMD

PhotonICs® PMD 1k-S23D Video Sensor Array with Active SBI64x16 Pixels

PhotonICs® PMD 19k3D Video Sensor Array160x120 Pixels, QQVGA

PhotonICs® PMD 3k-S3D Video Sensor Array with Active SBI64x48 Pixels


ApplicationsApplications

APPLICATIONSAPPLICATIONS


Markets and ApplicationsMarkets and Applications

Industrialsafetysurveillanceobject measurementobject recognition…

Industrialsafetysurveillanceobject measurementobject recognition…

Automotivepedestrian safetyACC stop & go (ACC)collision avoidanceparking aidemergency breakman-machine interface (gesture recognition)smart airbag...

Automotivepedestrian safetyACC stop & go (ACC)collision avoidanceparking aidemergency breakman-machine interface (gesture recognition)smart airbag...

RoboticsnavigationAGV ‘scollision avoidance…

RoboticsnavigationAGV ‘scollision avoidance…

Multimediavirtual realityuser interfacesgames video conferencingfilm & televisionproduct presentation

» ...

Multimediavirtual realityuser interfacesgames video conferencingfilm & televisionproduct presentation

» ...


Background replacementBackground replacement

original data background replaced


Parking AssistantParking Assistant


Rear View CameraRear View Camera


Pedestrian SafetyPedestrian Safety


Gesture ControlGesture Control


Keyless DialingKeyless Dialing


Advantages of 3D for Gesture RecognitionAdvantages of 3D for Gesture Recognition

Classical 2D-camera, no depth information

Additional information of a 3D-camera


3D Siam3D Siam-- ProjektProjekt

3D-Siam 3D-Sensorik für vorausschauende Sicherheitssysteme im Automobil.

BMBF- Projekt (08/2001 bis 07/2005)

Fachhochschule Trier


3D3D--Siam Test CarSiam Test Car

IR-Sender

PMD-Kamera


THANKS FOR YOUR ATTENTIONTHANKS FOR YOUR ATTENTION

Systematic Investigation of Properties of PMD-Sensors

Systematic Investigation of Properties of PMD-Sensors

1st Range Imaging Research Day, Zürich

Jens Pannekamp, Alexander Steitz

Motivation

• Tremendous success of machine vision(2005: 1 billion € for machine vision in Germany)

• Many real world interactions need 3D!

• Abundant applications in automation & robotics, mobile systems and safety & security

• Compact range sensors become available(PMD Technologies, CSEM, Canesta, 3DV Systems, Matsushita)

Guide for evaluation and selection of 3D-sensors

Applications for 3D-Sensors

• Driver assistence (PMD Technologies)

• Guidance (Claas)

• Collision avoidance (Götting)

• Consumer tracking (Vitracom)

• Bin picking (3 x IPA)

Investigated Range Sensor

• PMD[vision] 1k-S of PMD Technologies

• Sensor array with 64 x 16 pixels

• Suppression of background illumination

• Visualization software CamVis Pro

Data sheet

• Max. range: 7.5 m

• Z-resolution: > 6 mm

• Field of view: 34° (h) x 12° (v)

• Framerate: up to 50 fpsSource: PMD Technologies, Siegen

Example Images

• Upper image:Intensity (grayscale) image of hand in front of chess board

• Lower image:Distance image of same scene – segmentation not disturbed by surface properties

• Problems in practice:Distance values not independent of surface (material, color, texture)

Experimental Setup

• Camera on vertical stage

• 11 materials on plates of size 35 x 35 cm2 :paperaluminium (smooth and sand blasted)steel (smooth and sand blasted)plastics (smooth and rough)ceramic tile (smooth and rough) woodcloth

• Tiltable mount for exchangeable plates

Calibration and Image Evaluation Procedure

• Usage of calibration tool „CamVis Pro“

• Calibration distance: 50 cm

• Calibration material: white paper

• Manual optimization of shutter and integration time for each series of 50 images

• Optimization criterion: Reduction of visible noise in image

• Typical integration time: 10 000 µsec = 0.01 sec

• Manual selection of ROI for relevant pixels, ie. pixels measuring plate

Series of Experiments

a. Influence of distance and materialDistances = {50, 100, 200, 400} cmMaterials = {all materials}50 images for each combination with optimized shutter and integration time

b. Linearity of distance measurementDistances = {50, 100, 200, 400, 550} cmMaterials = {paper, sand blasted aluminium, smooth steel}50 images for each combination with optimized shutter and integration time

c. Estimation of tilt angleTilt anlge = {0°, 5°, 10°, 15°, 30°, 45°, 60°}Distances = {200 cm}Materials = {paper, sand blasted aluminium, smooth steel}50 images for each combination with optimized shutter and integration time

Camera Properties

1. Systematic error for single pixel”Mean that would result from an infinite number of measurements of the same measurand carried out under repeatability conditions minus a true value of the measurand.”

2. Standard uncertainty for single pixel”Uncertainty of the result of a measurement expressed as a standard deviation.”

3. LinearityThe maximum deviation of any points from a straight line drawn as a "best fit"through the point cloud.

4. Estimation of tilt angleTilt angle is extracted from ROI by means of a best fit algorithm and compared to the true value.

Systematic Error: Derivation

For each distance/material combination of measurement series A, distance values of all 50 images and all relevant sensor pixels are averaged. The difference between this averaged value and the true distances is considered as systematic error.

Systematic Error

Systematic Error: Data

Systematic Error [mm] Distance [mm]Material 500,00 1000,00 2000,00 4000,00sand blasted steel 44,87 43,99 169,98 94,76rough ceramic tile 31,80 54,74 187,29 143,75smooth ceramic tile 19,58 42,47 163,66 103,83rough plastics 40,71 41,26 172,95 98,93smooth plastics 28,78 46,95 186,05 92,50smooth steel 9,83 10,48 161,81 117,77smooth aluminium 57,30 62,02 163,30 71,91sand blasted aluminium 7,07 1,36 172,09 194,10wood 36,07 57,91 203,53 142,26paper 3,58 5,25 190,35 196,24cloth 30,59 50,76 188,56 145,99

Systematic Error: Interpretation

• Short distances (50 cm – 100 cm):0.5 cm .. 6 cmDiffuse scattering materials (paper, sand blasted aluminium) perform best

• Longer distances (200 cm – 400 cm):7 cm .. 20 cmSmooth surfaces (smooth aluminium, smooth plastics) perform bestDiffuse scattering surfaces probably do not return enough light (?)

• Material and surface roughness determine quality of distance measurement

• High errors – especially for longer distances – can be attributed partially to calibrationat 50 cm

• Potential for improving (one-point) calibration

Standard Uncertainty: Derivation

For each distance/material combination of measurement series A, the standard deviation is calculated over all 50 images and all relevant sensor pixels.

(Different approach in paper:First, for each distance/material combination of measurement series A an average range image is calculated from the 50 images. Then, the standard deviation is calculated over all relevant sensor pixels.)

Standard Uncertainty

Standard Uncertainty: Data

Standard Uncertainty [mm]Material 500,00 1000,00 2000,00 4000,00sand blasted steel 22,81 19,88 12,82 21,65rough ceramic tile 19,21 18,14 34,24 95,62smooth ceramic tile 27,34 22,49 16,69 31,43rough plastics 21,11 24,94 46,86 128,36smooth plastics 38,26 42,81 74,38 170,09smooth steel 32,46 25,50 22,32 35,00smooth aluminium 115,33 68,32 56,56 30,64sand blasted aluminium 19,75 18,42 23,23 57,32wood 18,33 19,26 21,25 50,36paper 18,72 22,47 26,66 85,33cloth 18,51 20,28 23,45 65,84

Distance [mm]

Standard Uncertainty: Interpretation

• Standard uncertainty for single pixel: 2 cm .. 12 cm

• Plastics performs badly for longer distances (13 cm .. 17 cm)

• For diffuse scatterers (wood, paper, cloth) uncertainty rises from low levels

• Materials with good and near constant performance: sand blasted steel, smooth ceramic tile, smooth steel

• Smooth aluminium with inverse behaviour

• For deeper understanding: measurement of roughness and optical properties

• Standard uncertainty can be reduced by ost-processing (smoothing, median filtering)

Linearity: Derivation

The linearity is derived from plots of measurement series B depicting measured distance values versus true distances. The maximum deviation between any measured distance value and the regression line obtained by a least squares fit is taken as linearity. The distance values are obtained by averaging over all 50 images and all relevant sensor pixels.

Linearity

Linearity: Data

Linearity as Deviationfrom Regression Line [mm]Material 500,00 1000,00 2000,00 4000,00 5500,00sand blasted aluminium 17,30 42,28 77,42 3,86 21,70smooth steel 30,03 24,62 91,47 39,47 2,65paper 10,32 46,89 84,95 23,14 4,60

Distance [mm]

Linearity: Interpretation

• Data shows only little deviations from linear behaviour

• Suggestion for calibration: Removal of systematic errors by means of linear regression

• Remaining non-linear errors 0.3 cm .. 9 cm for measuring range 50 cm .. 550 cm

• Remaining relative errors 1% .. 5%But: This calibration depends on material!

Estimation of Tilt Angle: Derivation

The tilt angle of the plate is estimated from the range images of measurement series C. First, all 50 images of one tilt angle/material combination are averaged. Then, a plane is fitted to the data by means of a least squares fit. The derived orientation of this plane is compared to the true tilt angle.

Estimation of Tilt Angle

Estimation of Tilt Angle: Data

Tilt Angle [°] Reference Angle [°]Material 0,00 5,00 10,00 15,00 30,00 45,00 60,00sand bl.alum. -4,27 1,00 8,22 16,53 40,73 58,19 60,53paper -3,40 1,61 5,88 13,72 32,01 50,32 67,90smooth steel -5,12 4,07 8,33 15,61 42,14 57,39

Estimation of Tilt Angle: Interpretation

• Tilted surfaces can be measured up to at least 60°

• Paper shows a near linear behaviour

• Number of available pixels for plane fit decrease with tilt angle

ConclusionSystematic error and standard uncertaintySystematic error and standard uncertaintySystematic error and standard uncertaintySystematic error and standard uncertainty•••• For short distances (50 cm, 100 cm) diffuse scattering materials such as paper or sand

blasted aluminium perform best (systematic error 0,4 cm resp. 0,7 cm).•••• For longer distances (200 cm, 400 cm) smooth surfaces such as smooth aluminium or

smooth plastics yield best results (systematic error 7 cm resp. 9 cm).•••• Standard uncertainty ranges from 1 cm to 17 cm and depends strongly on material and

distance

LinearityLinearityLinearityLinearity•••• If systematic errors are removed by means of a linear regression, the remaining non-

linear errors are in the range of 0,3 cm to 9 cm for a measuring range of 50 cm to 550 cm. This corresponds to relative errors ranging from less than 1% up to 5%.

Tilted surfacesTilted surfacesTilted surfacesTilted surfaces•••• Tilted surfaces can be measured up to at least 60°.

Open Questions• Investigation how systematic error and uncertainty depend on surface parameters

(roughness, optical properties, color etc.)

• Comparison of cameras of different manufacturers

Acknowledgement

The technical support of PMD Technologies is strongly appreciated.

Research Project „SmartVision“

• Project in German InnoNet-Program

• Start: January 2006

Objectives

• Development of algorithms for object tracking and object recognition(IWR, IPA, 3Soft)

• Applications in the fields of automation & robotics, mobile systems and safety & security

Unified analysis of the performanceand physical limitations of opticalrange-imaging techniques

Peter SeitzVice President, CSEM SA, ZurichProfessor of Optoelectronics, University of Neuchâtel

Range Imaging Days, ETH Zurich, September 8-9, 2005

Peter Seitz :: 05.10.2005 :: Page 2

Content

§ Introduction and motivation

§ Taxonomy of optical range imaging techniques

§ Limitations of the generation and detection of light

§ Linear shift-invariant (LSI) systems

§ A unified framework for optical RIM system analysis

§ Quantum noise limited phase measurement

§ Comparison of RIM system performance limitations

§ Conclusions

§ Summary




Taxonomy of optical range imaging techniques

Interferometry Thomas Young (1801)

lens

light source

rotatingcogwheel

observer

beam splitter planemirror

lens

lens

lens

lens

light source

rotatingcogwheel

observer

beam splitter planemirror

lens

lens

lens

Time-of-flight ranging Armand Fizeau (1849)

Triangulation Egyptian surveyors (1400 B.C.)


Optical setups of the basic ranging methods

x

α

z

Triangulation

z

Interferometry

z

Time-of-flight


0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0 5 10 15 20 25

Photon number

Pro

babi

lity

Poisson

Bose-Einstein

n=10

Limitations of light generation and detection (I)

Generation and detection of light are statistical, noisy processes: - Coherent light (independent random events): Poisson statistics - Thermal light (correlated events): Bose-Einstein statistics

NPoisson =σ

2NN

EinsteinBose

+

=−σ


1.E+00

1.E+02

1.E+04

1.E+06

1.E+08

1.E+10

1.E+00 1.E+02 1.E+04 1.E+06 1.E+08 1.E+10

Number of photons

Sta

nd

ard

dev

iati

on

Poisson Bose-Einstein

Limitations of light generation and detection (II)


Linear shift invariant (LSI) systems

LSI

In the complex function space, harmonic functions are eigenfunctions of all LSI systems. The function system of the harmonic functions is complete in the linear space of square integrable functions

Peter Seitz :: 05.10.2005 :: Page 10

Unified framework for optical RIM system analysis

x

z

x

P

z

P

t

Pλ/2 1/fΛ

α

z

z

πϕ

α 2tanTRIz

Λ=

πϕλ22

INTz =π

ϕ22TOF

fc

z =

Peter Seitz :: 05.10.2005 :: Page 11

Quantum noise limited phase measurement (I)

s

F(s)

B

A

ϕ

S=1/f

∆s

Fi

−−

=

+++=

−+−=

20

13

3210

220

213

arctan

4

2)()(

FFFF

FFFFb

FFFFa

ϕ

Ss

sb

Bs

aA

∆=

∆=

∆=

πδ

δδ

sin

Peter Seitz :: 05.10.2005 :: Page 12

s

F(s)

B

A

ϕ

S=1/f

∆s

Fi

Quantum noise limited phase measurement (II)

ab

sF k

k k 2

23

0

=

∂∂

= ∑=

ϕσϕ

variances si = Fi

Peter Seitz :: 05.10.2005 :: Page 13

Comparison of RIM system performance limitations

πσ

ασ ϕ

2tan,Λ

=TRIz π

σλσ ϕ

22, =INTz π

σσ ϕ

22, fc

TOFz =

l

b

αΛ

z

xx

bbz

l==

αtan

Triangulation

z

Interferometry Time-of-flight

Peter Seitz :: 05.10.2005 :: Page 14

Extending the non-ambiguity range

21

2112 Λ−Λ

ΛΛ=Λ

The non-ambiguity range is not restricted to half of the wavelength: By using two different wavelengths Λ1 and Λ2 a synthetic wavelength Λ12 is produced, leading to a much larger non-ambiguity range

Peter Seitz :: 05.10.2005 :: Page 15

Conclusions and Summary

§ Photon quantum noise is one of the key factors limiting the performance

of optical range imaging methods (diffraction, speckle, etc.)

§ Measurement = Comparison with known common standard. Conclusion:

Three fundamental methods: Triangulation, interferometry, time-of-flight

§ Unified analysis of all optical range imaging methods in the framework of

LSI systems: Identical functional relationship – phase measurement.

§ Phase measurement precision limited by photon quantum noise

§ Largely unexplored notion: Combination of various methods

§ The different methods have more in common than one might think.

Conclusion: Share approaches, insights and ideas – work together !

T h a n k y o u f o r y o u r a t t e n t i o n.

CCD/CMOS Lock-In Pixel for Range ImagingChallenges, Limitations and State-of-the-Art

Bernhard BuettgenCSEM, Centre Suisse d’Electronique et de Microtechnique SA

September 8, 2005

Overview

Phase measurement for range imaging

Shot-noise limitation

Pixel concepts

Distance accuracy

Dynamic range

Background-light suppression

Enhanced pixel structures

Summary

Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 1

Homodyne phase measurementIntensity Modulation:

Pulse: direct ‘’time-of-flight’’

Continuous Wave: phase delay between emitted and detected signal

Simultaneous acquisition of depth information and brightness

pixelmatrixL

objects

Popt

illumination

sensor readout andimage display

start

stop

ϕπ

⋅⋅

=modf

cL4

c: speed of light

fmod: modulation frequency


Pixel-Inherent Sampling

Emitted signal

Detected signal

Measurements: four sampling points A0 ... A3

)2sin()( 00 tfPPtP modopt π⋅+=

)2sin()( ϕπ +⋅+= tfABtN modmeasmeasel

t

Nel

A0

A1

A2

A3

Popt

emitted signal

detected signal

ϕ Bmeas

Ameas

P0

⎟⎟⎠

⎞⎜⎜⎝

⎛−−

=31

20arctanAAAA

ϕ

Brightness information

Accuracy information

Distance information


Figure of Merit: Demodulation Contrast

sig

meas

AA

c =demod

Measured signal amplitude [#e]

Signal offset [#e]


Physical Limitation by Photon Shot Noise

sigL Ac

Bfc

⋅⋅

⋅⋅=

demodmod 24πσ

t

Nel

A0

A1

A2

A3

Popt

emitted signal

detected signal

ϕ Bmeas

Ameas

P0

c: light velocity [m/s]

fmod: modulation frequency [Hz]

cdemod: demodulation contrast [-]

B: intensity [#e]

A: amplitude [#e]


Pixel Concepts

1-Tap 2-Tap 4-Tap N-Tap

Exploitation of optical energy

50% lost 100% used 100% used 100% used

Mismatch problems No Yes Yes Yes

Fill-factor +++ ++ - ---

Number of acquisitions 4 2 1 1


2-Tap Pixel

Two samples per

acquisition

CCD-arrangement for

sampling the light

CMOS read-out amplifiers

Charge transport mainly

based on diffusion

processes

Buried channel enhances

the performance


Challenge I: Distance Accuracy

Precise 3D measurements in real-time without …increasing the total emitted optical power (keeping eye safety)increasing the optics aperture (keeping the systems small)using electronics with better noise performance(keeping the cost level low)

Goal: Std.=1mm or even below!

Enabling new applicationsBiometricsIndustrial packaging


Accuracy – Limitationssig

L AcB

fc

⋅⋅

⋅⋅=

demodmod 24πσ

104 105 1060

0.5

1

1.5

2

2.5

3

mean number of photo-generated electrons per sample [-]

accu

racy

[cm

]

fmodsweep from 10MHzto 80MHz

Assumptions:

- samples shifted by 90 degrees

- demodulation contrast of 64%

- no delays in the charge transport


Accuracy – Modulation Frequency

Charge transport dominated by …

… drift processes … diffusion processes

Ultdrift µ

2

=Dltdiff

2

=

110 3V U:Exp. ≈⇒=drift

diff

tt

Higher cut-off frequencies using drift fields!


Challenge II: Dynamic Range

3D-images are composed of objects at

short distances with low and high reflection coefficients

long distances with low and high reflection coefficients

Pixel capability of measuring all distance between the longest and shortest

distances

No saturation!

Minimum standard deviation of the distance measurement!

sigL Ac

Bfc

⋅⋅

⋅⋅=

demodmod 24πσ


Dynamic Range – Limitations

Dynamic range sets the minimal and maximal storable amount of

charges into relation!

Minimum amount of charges

defines the smallest uncertainty of the distance measurement

Maximum amount of charges

defined by the brightest points in the scenery

[ ]⎥⎥⎦

⎤

⎢⎢⎣

⎡⋅⎟⎟

⎠

⎞⎜⎜⎝

⎛⋅=

min

max

2

min

max10log20δδ

RRdBDR

sigL Ac

Bfc

⋅⋅

⋅⋅=

demodmod 24πσ


Dynamic Range – Example

Distance range: Rmin = 20cm, Rmax = 5m

Reflection coefficients: δ = 5%, δ = 100%

DR = 82dB (Factor of 12500)

Minimum accuracy of 1cm requires 60.000 electrons

Also 750.000.000 electrons must be storable!

sigL Ac

Bfc

⋅⋅

⋅⋅=

demodmod 24πσ


Challenge III: Background Light

400 500 600 700 800 900 1000 11000

0.5

1

1.5

wavelength [nm]

optic

al p

ower

den

sity

[kW

/m2 / µ

m]

bandwidth

~800kW/m2/µm


Background-Light Suppression – Limitations

0 5 10 15 200

2

4

6

8

background to signal ratio [-]

accu

racy

[cm

]

signal electrons:1) 100002) 316003) 1000004) 3160005) 1000000

signal power 1

2

34

5


Enhanced Pixel for High Dynamic Range

Pixel-wise integration (PWI)

Functionality verified

1 1.5 2 2.5 3 3.50

2

4

6

8

real distance [m]

mea

sure

d di

stan

ce [m

] normal modepwi mode


Enhanced Pixel for High Background Light

Difference of charge packets is processed in-pixel

0 20 40 60 80 100 120 1400

2

4

6

8

10

12

14

ratio background to signal [-]

accu

racy

[cm

]

theorymeasurement


Summary and Outlook

Smart pixels for fast optical distance measurement based on TOFChallenges for future designs

High accuracy high modulation frequenciesHigh dynamic range adaptive integration timeBackground-light suppression subtraction of charge packets

State-of-the-artEnhanced pixel structures for extended dynamic range and background light suppressionAspects of high modulation frequencies to be investigated!

New pixel architectures are required that enable the charge transport by electric drift fields.


T h a n k y o u f o r y o u r a t t e n t i o n.

Date post:	03-Jul-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Continuous 360 degree real-time 3D laser...

Documents