Image NoiseImage Noise

John Morris

Department of Computer Science, Tamaki CampusThe University of Auckland

2

Stereo Image Noise Sources Stereo Image Noise Sources • Signal noise

• Electromagnetic interference eg cross-talk • Quantum behaviour of electronic devices

eg resistor shot-noise• Quantization: digitization of real-valued signals

• Geometric sources• Discrete pixel sensors with finite area• Occlusions• Perspective distortion

• Opto-Electronic sources• Sensitivity variations between cameras• Different ‘dark noise’ levels• Real lenses• Depth-of-focus

Single camera sources Stereo (2-camera) sources

Note that we use the term ‘noise’ for all problem sources!

3

Electronic NoiseElectronic Noise

Antennae (Receivers) Wires act as antennae for EM waves ‘Wire’ includes discrete wires

but also Tracks on circuit boards Interconnects on chips

Transmitters Any wire with a changing current emits EM waves

4

Electronic NoiseElectronic Noise

Digital circuits very rapid transitions (switching events) High frequency signals

Crosstalk One wire is influenced by neighbouring wires

Ideal digital signal

‘Instaneous’ rise or fall≡ infinite frequency perfect radiator

Real digital signal

‘Fast’ rise or fall high frequency very good radiator

Signal driven into purple wire

Signal picked up on green wire

EM coupling

5

Electronic noiseElectronic noise

Quantum effects Resistor ‘shot’ noise

Resistive element is composed of discrete atoms Always in motion for all T > 0oK (absolute zero) Noise as effective resistance changes Moving atoms ‘collide’ with electrons moving to form the

current Random fluctuations in current

or

Noise as effective resistance changes Similar effects in all current carrying or producing devices

• Transistors• Capacitors• Inductors, etc

e-

e-

6

Electronic noiseElectronic noise

Digitization noise Analogue signal

Taking all possible values• At least at a macroscopic level!

Digital signal Represented by a range of integers

• 0 .. 255 (8 bit signal)• 0 .. 4095 (12 bit signal)• -2048 .. 2047 (12 bit signed signal)

A to D converter Decides to which integer value to map a real value

Discretization Values which differed (in real domain) become the same (in integer domain)

7

Geometric noiseGeometric noise

‘Pixelisation’ of images Sensor is divided into discrete regions – pixels ‘Edges’ in images don’t conveniently fall onto pixel

boundaries

Red object

Blue object Real image

has blurrededges

8

Geometric noiseGeometric noise

Occlusions Points visible from one camera onlyPoints which it is impossible to match

Perspective distortion Field of view in one camera differs from

other Left and right images contain different

numbers of pixels Impossible to match all pixels correctly

Stereo Problem

9

Opto-electronic noiseOpto-electronic noise

Cameras have different gain settings Amplifiers are not ‘matched’ perfectly

Sensors have different ‘dark current’ characteristics All sensors produce some electrons (current) with no light Quantum ‘tunneling’ out of the sensor device

Stereo Problem

Light Intensity

Cu

rren

t Different slopesGains differ

Different offsetsDark currents differ

10

Effect of NoiseEffect of Noise

but …

What happens if we use ‘noise-free’ images?

L Image - ‘corridor’ setSynthetic (ray traced)

Precise ‘ground truth’is available

11

Noise-free Image MatchingNoise-free Image Matching

MismatchIL(x)-IR(x-dx)

Intensity

Disparity(from ground truth)

Examine one scan line – line 152

12

Real Image MatchingReal Image Matching

MismatchIL(x)-IR(x-dx)

Intensity

Disparity(from ground truth)

Tsukuba – line 173

13

Distribution of signal differences

Pixel-wise correspondences – ‘Tsukuba’ pair (line 173)Pixel-wise correspondences – ‘Tsukuba’ pair (line 173)

Grey-codedsignal differences

14

Previous work: pros & consPrevious work: pros & consConventional approach: energy minimisation combining image

dissimilarity, surface curvature and occlusions• Exact minimisation with dynamic programming:

global 1D optimum matching under ordering constraints; can account for local photometric (offset or contrast) deviations and occlusions; fast processing;

no inter-scan-line constraints; random deviations on textureless regions; error propagation along scan-lines

• Approximate minimisation with Min-Cut techniques: 2D surface curvature constraints (MRF);

a provably close approximate solution of an NP-hard problem; can account for local occlusions;

cannot account for local or global photometric deviations; high computational complexity

• Heuristic approximate minimisation with Belief Propagation 2D surface curvature constraints (MRF);

can account for local occlusions; cannot account for local or global photometric deviations;

high computational complexity

15

Conventional approaches: Conventional approaches: basic problems basic problems

No account for intrinsic ill-posed nature of stereo problemsSearch for a single surface giving the best correspondence

between stereo imagesbut the single surface assumption is too restrictive in practice

Heuristic or empirical weights of energy terms dramatically affect matching accuracy

Large images and large disparity ranges lead to high computational cost of min-cut or belief propagation algorithms

16

right - scanline signal

left

scan

-line

sig

nal

signal-basedcorrespondingareas

Actual disjoint surface profiles and Actual disjoint surface profiles and piecewise-constant corresponding signalspiecewise-constant corresponding signals

•Single surface reconstruction:•Extreme disjoint variant:

17

Distribution of signal differences

Pixel-wise correspondences for a “Tsukuba” stereo pair (scan-line Pixel-wise correspondences for a “Tsukuba” stereo pair (scan-line y y = 173)= 173)

Grey-codedsignaldifferences

18

Concurrent Stereo Matching: Main ideasConcurrent Stereo Matching: Main ideas

Human ‘stroke-wise’ analysis of a 3D scene • Eyes browse from low to high frequency regions, from

sharp points to smooth areas rather than scan line-by-line (Torralba, 2003)

Appropriate (likely) correspondence rather than best matching

Separation of noise estimation and signal matching from selection of surfaces and occlusion handling

Stereo matching should avoid the ‘best match’ or signal difference minimisation almost universally

used nowin favour of a

likely match based on a local signal noise model

19

Modular structure of CSMModular structure of CSM

Step 1:Estimate the image noise model

(allow it to be spatially variant)

Segment based on noise

Select candidate 3D volumes

Step 2:Fit constrained surfaces to the

candidate volumes

Could use K-Mean, SUSAN, etc

Could be surface optimisation

20

Noise MapNoise Map

Scaled (Amplitudex6) Noise Map

White regions have higher noise - almost always

appears in occluded regions

Technique A: use a fast, efficient stereo matching technique (SDPS) to produce a disparity map – use mismatches as noise estimates

21

Noise-Driven SegmentationNoise-Driven Segmentation

Colour Mean Shift Segmentation

Colour-position clustering in a 5D feature space: 3D-colour model L*u*v and 2D-lattice coordinates

The noise map is considered to be the extra, sixth dimension

• Convert an image into data tokens• Choose initial search window

locations• Compute the ‘mean shift’ window

location for each initial position• Merge windows that end up on the

same ‘peak’ or mode• Cluster data over the merged

windows

After noise-driven segmentation: occluded regions are segmented into small isolated blocks

22

CSM: candidate 3D volumes and surface fittingCSM: candidate 3D volumes and surface fitting

Black regions contain likely matching points in the ‘slice’ for each disparity, d Surface fitting shrinks or expands each segmented region from slice to slice (based on counts of candidate points)

d= 5

d=8

d= 6

d=10d=9

d=12d=11 d=14d=13

d= 4

d= 7

d= 3

23

CSM: candidate 3D volumes and surface fittingCSM: candidate 3D volumes and surface fitting

Ideal disparity slices

CSM surface fitting

d=5 d=8d=6 d=10d=9

d=12d=11 d=14d=13 Disparity map

d=5 d=8d=6 d=10d=9

d=12d=11 d=14d=13 CSM Disparity map

24

Symmetric CSM: Symmetric CSM: candidate 3D volumes and surface fittingcandidate 3D volumes and surface fitting

d=5

d=8

d=6

d=10d=9

d=12d=11 d=14d=13

d=7

d=3 d=4

25

Symmetric CSM: Symmetric CSM: candidate 3D volumes and surface fittingcandidate 3D volumes and surface fitting

Ideal disparity slices

SCSM surface fitting

d=5 d=8d=6 d=10d=9

d=12d=11 d=14d=13 Disparity map

d=5 d=8d=6 d=10d=9

d=12d=11 d=14d=13 SCSM Disparity map

26

Algorithm ComparisonAlgorithm Comparison

Symmetric DP stereo Graph cut Symmetric BP

SCSM CSM Ground Truth

27

Middlebury Benchmark (MB)Middlebury Benchmark (MB)

Algorithms

Tsukuba Sawtooth

all untex disc all untex disc

MB-SBPO 0.97 1 0.28 3 5.45 3 0.19 1 0.00 1 2.09 3

CSM 1.15 3 0.80 12 1.86 2 0.98 13 0.62 25 1.69 2

SCSM 0.97 1 0.74 11 1.80 1 0.96 12 0.60 24 1.57 1

* MB-SBPO – symmetric belief propagation algorithm (best-performing Middlebury benchmark)

Algorithms

Venus Map

all untex disc all disc

MB-SBPO 0.16 3 0.02 3 2.77 1 0.16 1 2.20 1

CSM 1.18 12 1.04 10 1.48 3 3.08 34 7.34 18

SCSM 1.15 11 0.91 9 1.38 2 3.05 33 7.03 17

rank among 40

algorithms

28

ConclusionsConclusions

Stereo matching is an ill-posed problem, reconstruction of actual 3D optical surfaces is impracticalMore reasonable goal: mimic human binocular stereo vision

Conventional constrained best matching does not explicitly account for a multiplicity of equivalent matches, for noise in both images of a stereo pair and for local contrast or offset image distortions

Concurrent stereo matching gives promising results because it separates the problem into • search for all the candidate volumes with equivalent good matches

(allowing for the estimated noise) and • search for surfaces fitting to the volumes

Even the simplest implementation of the new approach competes with the best-performing conventional algorithms

Sloping surfaces challenge our CSM algorithm – watch this space!

29

IVCNZ’2006IVCNZ’2006

For a conference with a different style, consider IVCNZ’2006(Image and Vision Computing, New Zealand)

Great Barrier Island, New Zealand• Gateway to the Hauraki Gulf

and Auckland• 40 mins by light plane from

Auckland,3 hours by ferry

• Full range of accommodation options:

• Hotel style, cabins, … , even tents!

Book early and

you can sail there

30

Great Barrier IslandGreat Barrier Island

31

Stereo: Correspondence ProblemStereo: Correspondence Problem

Stereo Pair Images from identical cameras separated by some distance

to produce two distinct views of a scene

xL xR

Disparity = xL - xR 1z

Corresponding RegionsLeft Image Right Image