Post on 23-Sep-2020
transcript
Studying Dynamic Scenes with Time of Flight Cameras
Norbert Pfeifer, Wilfried Karel Sajid Ghuffar, Camillo Ressl norbert.pfeifer@geo.tuwien.ac.at
Research Group Photogrammetry
Department for Geodesy and Geoinformation Vienna University of Technology
geo.tuwien.ac.at
Images: Diploma thesis Stefan Niedermayer
Outline
Time of Flight Cameras Calibration and Scattering
still and dynamic imagery Optical Flow and Range Flow Range video analysis
dynamic imagery
General question: does high (temporal) sampling compensate for large random noise ?
2
ToF cameras: combining advantages
Area wise, simultaneous, robust determination of 3D object coordinates by range measurement of an emitted signal
• Suited for low contrast surfaces and in darkness • No homologous points necessary
No moving parts Low power consumption Small, compact, mobil Cheap Bundle of vectors
• Simplifies indirect orientation on moving platforms
• Simultaneous capturing of a dynamic object space
3
... of photography and laser scanning
Distance measurement
d = ½ c t Common for all methods:
• nIR • Amplitude modulation
Pulsed modulation • Avalange photo diodes (APD) for detection • APD in Geiger mode: single photon counting
(SPAD) • Multiple Double Short time Integration (MDSI)
Continuous wave modulation (Photo mix detectors, PMD)
4
Inte
nsity
Time
PH
φ
A
λmod
Amplitude and Phase (distance) measured
PMD: typical technical data
Precision: ≅ cm Accuracy: ≅ dm
5
Swissranger
Image matrix 144x176px² Modulation 5-30MHz
Field of view 39.6°x47.5° Uniqueness range 30-5m
Max. framerate 25 fps Dimensions 50x67x42.5mm³
Illumination 1W Mass 162g
Carrier WL 850nm Focal distance 8mm
PMD: random range errors
Empirically determined σ2obsDist vs. average amplitude
σ2obsDist ∼ 1/A2
6
PMD: systematic range errors
Local errors • Distance itself (not linear) • Position at sensor • Amplitude • Reflectivity / material (?) • Integration time • Internal / external temperature • Incidence angle (?) • Temporal drift
Not local errors • Internal: scattering • External: multipath
Relations unclear
7
Scattering cmp. lens flare
Multipath
Distance calibration approach
Self calibration: model selection and parameter estimation as integrated process on one data set
Amplitudes: lower relative noise in comparison to range and low local deformation
Lateral resolution very low (sensor matrix) expoit image space entirely circular, non-coded targets
Planar test field: no multipath, little scattering Bundle block adjustment IOR & EOR EOR mask test field area and use only areas outside targets distance residuals
Check residuals against posssible factors of influence: model selection, parameter estimation by LS adjustment
8
δρ
Calibration with range videos
Low lateral resolution vs. high temporal resolution : 25kPixel vs. 25fps
• Image sequences • Handheld camera guidance
permanente, random movements, dense sampling of parameter space
• Automatic target tracking • ~ 6000 frames • Only distances < 2.5m :
– A ∼ 1/d2 – Motion blur (espcially rotations)
9
Calirbation with still images
Averaging of thousands of frames with EOR=const to „still“ image suited for entire distance range
Automatic target detection and orientation Laborious search for correspondence between
object space and image space (no target code) 850 „stills“: with difference in integration
time, exterior orientation changes in amplitude, object distance, position in image space, ...
2 test fields with different reflectivity separate between object distance and amplitude
10
Model selection and parameter estimation
11
0 1 2 3 4 5 6 7 8-0.1
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
Observed dist. [m]
Der
. min
us o
bs. d
ist.
[m]
Der. minus obs. dist. corr. 4 all but obs. dist. , corr. modeloffset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs
0 1 2 3 4 5 6 7 80
2
4
6
8
10
12x 10
4
coun
t
0 2000 4000 6000 8000 10000 12000 14000 16000 18000-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Amplitude []
Der
. min
us o
bs. d
ist.
[m]
Der. minus obs. dist. corr. 4 all but amplitude , corr. modeloffset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs
0 2000 4000 6000 8000 10000 12000 14000 16000 180000
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5x 10
5
coun
t
10 30 60 90 120 150 180 210 255-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
int. time []
der.
min
us o
bs. d
ist.
[m]
Der. minus obs. dist. corr. 4 all but int. time , corr. modeloffset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs
Scattering
Internal relections Observation: mixture of focussed and scattered light Emphasized with high image contrast – active illumination Modell: addition in the complex plane
• Assumption: sinusoidal signal May produce „strange“ distances (phase wraps) Investigation of scattering
• As few assumptions as possible • Without prior calibration • Influencing factors
– Integration time? – Amplitude? – Distance? – Position at sensor?
12
Method
Background subtraction • Frames with / without foreground object • Vary parameters to be investigated • Background
– Black cardboard • Foreground
– Circular white target – Mounted on vertical staff
• Mask area of staff and half shadow (illumination not an ideal point source), investigation of remaining image area
2 types of differences: • Separated difference of amplitude & range → errors in the observation • Difference of the complex signal → error in the signal
High noise level • Static scene and fixed EOR • Pixel wise averaging over up to 60 hours of continuous data acquisition
13
a_0016: mean amp. diff [%]; iTime: 30
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
Variation of scattering with the position on the sensor
Complex subtraction Not shift invariant Radially symmetric to principal point
14
Range error
Findings of the experiment
Scattering linear in the amplitude double target size ~ double amplitude difference
Scattering additive in complex signal, independent of object space distance ... Internal effect
Varies with position on sensor Point spread function (using an unresolved target)
• 63% focussed light; 8 neighboring pixels : 1,4‰; 72% within disk (r=19px); side lobes (maxima) : 0,07‰
15
PSF: Application
16
𝑔𝑔 = 𝑓𝑓 ∗ ℎ + 𝜂𝜂 Observed image is a convolution of the “real” image with the PSF + noise
Inversion: Richardson-Lucy:
• 𝑓𝑓 𝑛𝑛+1 = 𝑓𝑓 𝑛𝑛 𝑔𝑔�̂�𝑓 𝑛𝑛 ∗ℎ
∗ ℎ𝑇𝑇
Reduce RMSE distance • spatially variant PSF:
73% • spatially invariant PSF:
69%
Amplitude and Range difference images with/without object in foreground before and after deconvolution
Analyzing dynamic scene content
Considerations Video sequences: image of object points move in data stream
vs. wide baseline: points „jump“ Range and brightness are functions parameterized over image space
r(x,y), i(x,y) Assumption: functions are continuous and differentiable
Brightness of points remains constant Wrong assumption because of active illumination (1/r2, cos(incidenceA))
Therefore Track range and brightness changes of points in image space for Computing camera trajectory (exterior orientation) Follow moving objects throughout the scene
17
Optical flow and range flow
Comparable least squares matching / ICP but approx. values = 0, no iterations, no re-assignment of correspondence
Apply to entire image area Range and brightness should be considered together (complementary?)
Use framework of optical flow and range flow
18
Intensity Image Depth Image Intensity Derivatives Depth derivatives
Optical and range flow
Equation for one pixel Two/three unknowns (optical/range flow) Image/surface gradients required, otherwise coefficients are zero Application in window – local method,
parameter estimation by least squares adjustment Applied to all pixels, e.g. independently
19
x
I(x) t1
t2
𝜕𝜕𝐼𝐼𝜕𝜕𝑥𝑥
𝜕𝜕𝑥𝑥𝜕𝜕𝑡𝑡
= −𝜕𝜕𝐼𝐼𝜕𝜕𝑡𝑡
𝐼𝐼𝑥𝑥𝑢𝑢 + 𝐼𝐼𝑦𝑦𝑣𝑣 + 𝐼𝐼𝑡𝑡 = 0
u
X(x)
Z(x) t1 t2
(U,W)
𝑍𝑍𝑥𝑥𝑈𝑈 + 𝑍𝑍𝑦𝑦𝑉𝑉 −𝑊𝑊 + 𝑍𝑍𝑡𝑡 = 0
Fusion of Range and Intensity for object tracking
20
𝐼𝐼𝑥𝑥u + 𝐼𝐼𝑦𝑦v + 𝐼𝐼𝑡𝑡 = 0 𝑍𝑍𝑥𝑥U+ 𝑍𝑍𝑦𝑦V − W + 𝑍𝑍𝑡𝑡 = 0
Optical Flow Constraint Equation Range Flow Constraint Equation
Intensity Image Depth Image Intensity Derivatives Depth derivatives
𝐼𝐼𝑥𝑥𝑓𝑓𝑍𝑍
𝐼𝐼𝑦𝑦𝑓𝑓𝑍𝑍
−𝐼𝐼𝑥𝑥𝑥𝑥𝑍𝑍− 𝐼𝐼𝑦𝑦
𝑦𝑦𝑍𝑍
𝑍𝑍𝑥𝑥𝑓𝑓𝑍𝑍
𝑍𝑍𝑦𝑦𝑓𝑓𝑍𝑍
−𝑍𝑍𝑥𝑥𝑥𝑥𝑍𝑍− 𝑍𝑍𝑦𝑦
𝑦𝑦𝑍𝑍− 1
𝑈𝑈𝑉𝑉𝑊𝑊
= −𝐼𝐼𝑡𝑡−𝑍𝑍𝑡𝑡
Long Term Trajectories
21
Independently moving objects • Long range trajectory generation • Spatio-temporal segmentation into independently moving object
22
Note noise level in depth and intensity image range errors at dark trousers of second person: calibration not applied missing texture around feet of first person: neither in range nor in brightness hardly background texture
movement within floor/wall plane pair depends on small sign at the wall: spatial regularization to determine full flow at all pixel
Video
23
Note • Illumination fall off • Systematic range errors • Full 3D estimation of flow, esp. lower train • Successful suppresion of erroneous background motion • Upper train shows consistent flow despite
grey level differences and range errors
Optical and Range Flow for Camera Relative Orientation
Motion fields generated by camera motion Estimating 6 parameters of relative orientation
using dense intensity and range data
24
Video
25
Add robust parameter estimation (reweighting in LS adjustment) to eliminate moving objects
Weights in LS adjustment
26
1 →0
Conclusions and findings
Calibration is possible and shows stability Systematic range errors in cm-dm order Stills provided better results in calibration than videos
• averaging of stills reduced random error and thus eased model selection • motion blur as additional source of signal (error) in dynamic scenarios
Internal scattering can become larger (dm-order) Internal scattering compensation possible but too time consuming
hardware solution to reduce internal scattering of light (better absorption) Combined optical and range flow as natural combination of both channels
can consider stochastic signal properties, as it is embedded in LSA High noise level propagates to flow vectors due to local analysis
e.g. fluctuation of flow vectors, boundaries of segments Random noise can only be effectively removed with very rigid models ToF cameras rather trigger method than application development ( Quantitative evaluation, details on methods, etc.: in the publications )
27
41
42
Ende
43