Post on 04-Jan-2016
description
transcript
Image NoiseImage Noise
John Morris
Department of Computer Science, Tamaki CampusThe University of Auckland
2
Stereo Image Noise Sources Stereo Image Noise Sources • Signal noise
• Electromagnetic interference eg cross-talk • Quantum behaviour of electronic devices
eg resistor shot-noise• Quantization: digitization of real-valued signals
• Geometric sources• Discrete pixel sensors with finite area• Occlusions• Perspective distortion
• Opto-Electronic sources• Sensitivity variations between cameras• Different ‘dark noise’ levels• Real lenses• Depth-of-focus
Single camera sources Stereo (2-camera) sources
Note that we use the term ‘noise’ for all problem sources!
3
Electronic NoiseElectronic Noise
Antennae (Receivers) Wires act as antennae for EM waves ‘Wire’ includes discrete wires
but also Tracks on circuit boards Interconnects on chips
Transmitters Any wire with a changing current emits EM waves
4
Electronic NoiseElectronic Noise
Digital circuits very rapid transitions (switching events) High frequency signals
Crosstalk One wire is influenced by neighbouring wires
Ideal digital signal
‘Instaneous’ rise or fall≡ infinite frequency perfect radiator
Real digital signal
‘Fast’ rise or fall high frequency very good radiator
Signal driven into purple wire
Signal picked up on green wire
EM coupling
5
Electronic noiseElectronic noise
Quantum effects Resistor ‘shot’ noise
Resistive element is composed of discrete atoms Always in motion for all T > 0oK (absolute zero) Noise as effective resistance changes Moving atoms ‘collide’ with electrons moving to form the
current Random fluctuations in current
or
Noise as effective resistance changes Similar effects in all current carrying or producing devices
• Transistors• Capacitors• Inductors, etc
e-
e-
6
Electronic noiseElectronic noise
Digitization noise Analogue signal
Taking all possible values• At least at a macroscopic level!
Digital signal Represented by a range of integers
• 0 .. 255 (8 bit signal)• 0 .. 4095 (12 bit signal)• -2048 .. 2047 (12 bit signed signal)
A to D converter Decides to which integer value to map a real value
Discretization Values which differed (in real domain) become the same (in integer domain)
7
Geometric noiseGeometric noise
‘Pixelisation’ of images Sensor is divided into discrete regions – pixels ‘Edges’ in images don’t conveniently fall onto pixel
boundaries
Red object
Blue object Real image
has blurrededges
8
Geometric noiseGeometric noise
Occlusions Points visible from one camera onlyPoints which it is impossible to match
Perspective distortion Field of view in one camera differs from
other Left and right images contain different
numbers of pixels Impossible to match all pixels correctly
Stereo Problem
9
Opto-electronic noiseOpto-electronic noise
Cameras have different gain settings Amplifiers are not ‘matched’ perfectly
Sensors have different ‘dark current’ characteristics All sensors produce some electrons (current) with no light Quantum ‘tunneling’ out of the sensor device
Stereo Problem
Light Intensity
Cu
rren
t Different slopesGains differ
Different offsetsDark currents differ
10
Effect of NoiseEffect of Noise
but …
What happens if we use ‘noise-free’ images?
L Image - ‘corridor’ setSynthetic (ray traced)
Precise ‘ground truth’is available
11
Noise-free Image MatchingNoise-free Image Matching
MismatchIL(x)-IR(x-dx)
Intensity
Disparity(from ground truth)
Examine one scan line – line 152
12
Real Image MatchingReal Image Matching
MismatchIL(x)-IR(x-dx)
Intensity
Disparity(from ground truth)
Tsukuba – line 173
13
Distribution of signal differences
Pixel-wise correspondences – ‘Tsukuba’ pair (line 173)Pixel-wise correspondences – ‘Tsukuba’ pair (line 173)
Grey-codedsignal differences
14
Previous work: pros & consPrevious work: pros & consConventional approach: energy minimisation combining image
dissimilarity, surface curvature and occlusions• Exact minimisation with dynamic programming:
global 1D optimum matching under ordering constraints; can account for local photometric (offset or contrast) deviations and occlusions; fast processing;
no inter-scan-line constraints; random deviations on textureless regions; error propagation along scan-lines
• Approximate minimisation with Min-Cut techniques: 2D surface curvature constraints (MRF);
a provably close approximate solution of an NP-hard problem; can account for local occlusions;
cannot account for local or global photometric deviations; high computational complexity
• Heuristic approximate minimisation with Belief Propagation 2D surface curvature constraints (MRF);
can account for local occlusions; cannot account for local or global photometric deviations;
high computational complexity
15
Conventional approaches: Conventional approaches: basic problems basic problems
No account for intrinsic ill-posed nature of stereo problemsSearch for a single surface giving the best correspondence
between stereo imagesbut the single surface assumption is too restrictive in practice
Heuristic or empirical weights of energy terms dramatically affect matching accuracy
Large images and large disparity ranges lead to high computational cost of min-cut or belief propagation algorithms
16
right - scanline signal
left
scan
-line
sig
nal
signal-basedcorrespondingareas
Actual disjoint surface profiles and Actual disjoint surface profiles and piecewise-constant corresponding signalspiecewise-constant corresponding signals
•Single surface reconstruction:•Extreme disjoint variant:
17
Distribution of signal differences
Pixel-wise correspondences for a “Tsukuba” stereo pair (scan-line Pixel-wise correspondences for a “Tsukuba” stereo pair (scan-line y y = 173)= 173)
Grey-codedsignaldifferences
18
Concurrent Stereo Matching: Main ideasConcurrent Stereo Matching: Main ideas
Human ‘stroke-wise’ analysis of a 3D scene • Eyes browse from low to high frequency regions, from
sharp points to smooth areas rather than scan line-by-line (Torralba, 2003)
Appropriate (likely) correspondence rather than best matching
Separation of noise estimation and signal matching from selection of surfaces and occlusion handling
Stereo matching should avoid the ‘best match’ or signal difference minimisation almost universally
used nowin favour of a
likely match based on a local signal noise model
19
Modular structure of CSMModular structure of CSM
Step 1:Estimate the image noise model
(allow it to be spatially variant)
Segment based on noise
Select candidate 3D volumes
Step 2:Fit constrained surfaces to the
candidate volumes
Could use K-Mean, SUSAN, etc
Could be surface optimisation
20
Noise MapNoise Map
Scaled (Amplitudex6) Noise Map
White regions have higher noise - almost always
appears in occluded regions
Technique A: use a fast, efficient stereo matching technique (SDPS) to produce a disparity map – use mismatches as noise estimates
21
Noise-Driven SegmentationNoise-Driven Segmentation
Colour Mean Shift Segmentation
Colour-position clustering in a 5D feature space: 3D-colour model L*u*v and 2D-lattice coordinates
The noise map is considered to be the extra, sixth dimension
• Convert an image into data tokens• Choose initial search window
locations• Compute the ‘mean shift’ window
location for each initial position• Merge windows that end up on the
same ‘peak’ or mode• Cluster data over the merged
windows
After noise-driven segmentation: occluded regions are segmented into small isolated blocks
22
CSM: candidate 3D volumes and surface fittingCSM: candidate 3D volumes and surface fitting
Black regions contain likely matching points in the ‘slice’ for each disparity, d Surface fitting shrinks or expands each segmented region from slice to slice (based on counts of candidate points)
d= 5
d=8
d= 6
d=10d=9
d=12d=11 d=14d=13
d= 4
d= 7
d= 3
23
CSM: candidate 3D volumes and surface fittingCSM: candidate 3D volumes and surface fitting
Ideal disparity slices
CSM surface fitting
d=5 d=8d=6 d=10d=9
d=12d=11 d=14d=13 Disparity map
d=5 d=8d=6 d=10d=9
d=12d=11 d=14d=13 CSM Disparity map
24
Symmetric CSM: Symmetric CSM: candidate 3D volumes and surface fittingcandidate 3D volumes and surface fitting
d=5
d=8
d=6
d=10d=9
d=12d=11 d=14d=13
d=7
d=3 d=4
25
Symmetric CSM: Symmetric CSM: candidate 3D volumes and surface fittingcandidate 3D volumes and surface fitting
Ideal disparity slices
SCSM surface fitting
d=5 d=8d=6 d=10d=9
d=12d=11 d=14d=13 Disparity map
d=5 d=8d=6 d=10d=9
d=12d=11 d=14d=13 SCSM Disparity map
26
Algorithm ComparisonAlgorithm Comparison
Symmetric DP stereo Graph cut Symmetric BP
SCSM CSM Ground Truth
27
Middlebury Benchmark (MB)Middlebury Benchmark (MB)
Algorithms
Tsukuba Sawtooth
all untex disc all untex disc
MB-SBPO 0.97 1 0.28 3 5.45 3 0.19 1 0.00 1 2.09 3
CSM 1.15 3 0.80 12 1.86 2 0.98 13 0.62 25 1.69 2
SCSM 0.97 1 0.74 11 1.80 1 0.96 12 0.60 24 1.57 1
* MB-SBPO – symmetric belief propagation algorithm (best-performing Middlebury benchmark)
Algorithms
Venus Map
all untex disc all disc
MB-SBPO 0.16 3 0.02 3 2.77 1 0.16 1 2.20 1
CSM 1.18 12 1.04 10 1.48 3 3.08 34 7.34 18
SCSM 1.15 11 0.91 9 1.38 2 3.05 33 7.03 17
rank among 40
algorithms
28
ConclusionsConclusions
Stereo matching is an ill-posed problem, reconstruction of actual 3D optical surfaces is impracticalMore reasonable goal: mimic human binocular stereo vision
Conventional constrained best matching does not explicitly account for a multiplicity of equivalent matches, for noise in both images of a stereo pair and for local contrast or offset image distortions
Concurrent stereo matching gives promising results because it separates the problem into • search for all the candidate volumes with equivalent good matches
(allowing for the estimated noise) and • search for surfaces fitting to the volumes
Even the simplest implementation of the new approach competes with the best-performing conventional algorithms
Sloping surfaces challenge our CSM algorithm – watch this space!
29
IVCNZ’2006IVCNZ’2006
For a conference with a different style, consider IVCNZ’2006(Image and Vision Computing, New Zealand)
Great Barrier Island, New Zealand• Gateway to the Hauraki Gulf
and Auckland• 40 mins by light plane from
Auckland,3 hours by ferry
• Full range of accommodation options:
• Hotel style, cabins, … , even tents!
Book early and
you can sail there
30
Great Barrier IslandGreat Barrier Island
31
Stereo: Correspondence ProblemStereo: Correspondence Problem
Stereo Pair Images from identical cameras separated by some distance
to produce two distinct views of a scene
xL xR
Disparity = xL - xR 1z
Corresponding RegionsLeft Image Right Image