Date post: | 19-Dec-2015 |
Category: |
Documents |
View: | 225 times |
Download: | 5 times |
A Gentle Introduction
to Bilateral Filtering and its Applications
A Gentle Introduction
to Bilateral Filtering and its Applications
07/10: Novel Variantsof the Bilateral Filter
Jack Tumblin – EECS, Northwestern University
Review: Bilateral FilterReview: Bilateral Filter
A 2-D filter window: weights vary with intensityA 2-D filter window: weights vary with intensity
cc
ss
DomainDomain
RangeRange
f(x)f(x)xx
2 Gaussian Weights:2 Gaussian Weights:product = product = ellisoidal footprintellisoidal footprint
Normalize weights toNormalize weights toalways sum to 1.0always sum to 1.0
Piecewise smooth result Piecewise smooth result – averages local small details, ignores outliersaverages local small details, ignores outliers– preserves steps, large-scale ramps, and curves,...preserves steps, large-scale ramps, and curves,...
• Some equivalence to anisotropic diffusion, robust statisticsSome equivalence to anisotropic diffusion, robust statistics [Black98,Elad02,Durand02][Black98,Elad02,Durand02]
• Simple & Fast Simple & Fast (esp. w/ (esp. w/ [Durand02][Durand02], , [Paris06][Paris06], , [Porikli08][Porikli08] other speedup methods) other speedup methods)
Review: Review: Bilateral Strengths Bilateral Strengths
cc
ss
Review: Bilateral FilterReview: Bilateral Filter
cc
ss
cc
ss
Why it works: Why it works: graceful segmentationgraceful segmentation• Smoothing for Smoothing for ‘similar’‘similar’ parts parts ONLYONLY• Range Gaussian Range Gaussian ss acts as a ‘filtered region’ finder acts as a ‘filtered region’ finder DomainDomain
RangeRange
f(x)f(x)xx
Bilateral FilterBilateral Filter Variants Variants
• Before the ‘Bilateral’ name :Before the ‘Bilateral’ name : – Yaroslavsky (1985): T.D.R.I.M.Yaroslavsky (1985): T.D.R.I.M.– Smith & Brady (1997): SUSANSmith & Brady (1997): SUSAN
And now, a growing set of extended variants:And now, a growing set of extended variants:
• ‘‘Trilateral’ Filter (Choudhury et al., EGSR 2003)Trilateral’ Filter (Choudhury et al., EGSR 2003)
• Cross-Bilateral (Petschnigg04, Eisemann04) Cross-Bilateral (Petschnigg04, Eisemann04)
• NL-Means (Buades05)NL-Means (Buades05)
• Bilateral Retinex(Elad05), Joint-Bilateral Bilateral Retinex(Elad05), Joint-Bilateral Upsampling (Kopf07), many more exist…Upsampling (Kopf07), many more exist…
And many more coming: application driven…And many more coming: application driven…
Who was first? Who was first? ManyMany Pioneers Pioneers
• Elegant, Simple, Broadly useful Idea Elegant, Simple, Broadly useful Idea
‘‘Invented’Invented’ several times several times
• Different Approaches, Increasing ClarityDifferent Approaches, Increasing Clarity– Yaroslavsky(1985): Yaroslavsky(1985):
‘Transform Domain Image Restoration ‘Transform Domain Image Restoration Methods’Methods’
– Smith & Brady (1995): Smith & Brady (1995): ‘SUSAN’‘SUSAN’ “ “SSmallest mallest UUnivalue nivalue
SSegment-egment-AAssimilating ssimilating NNucleus”ucleus”
– Tomasi & Manduchi(1998): Tomasi & Manduchi(1998): ‘Bilateral Filter’‘Bilateral Filter’
New Idea!New Idea!1985 Yaroslavsky: 1985 Yaroslavsky:
A 2-D filter window: A 2-D filter window: weights vary with intensity ONLY weights vary with intensity ONLY
cc
ss
DomainDomain
RangeRange
f(x)f(x)xx
Square neighborhood,Square neighborhood,Gaussian WeightedGaussian Weighted‘‘similarity’ similarity’
Normalize weights toNormalize weights toalways sum to 1.0always sum to 1.0
New Idea!New Idea!1995 Smith: ‘SUSAN’ Filter1995 Smith: ‘SUSAN’ Filter
A 2-D filter window: weights vary with intensityA 2-D filter window: weights vary with intensity
cc
ss
DomainDomain
RangeRange
f(x)f(x)xx
2 Gaussian Weights:2 Gaussian Weights:product = product = ellisoidal footprintellisoidal footprint
Normalize weights toNormalize weights toalways sum to 1.0always sum to 1.0
Output at is Output at is
average of a average of a tiny regiontiny region
Bilateral Filter:Bilateral Filter: 3 Difficulties 3 Difficulties
• Poor Smoothing in Poor Smoothing in High Gradient RegionsHigh Gradient Regions
• Smoothes and bluntsSmoothes and bluntscliffs, valleys & ridgescliffs, valleys & ridges
• Can combine disjoint Can combine disjoint signal regions signal regions
cc
ss
Bilateral Filter:Bilateral Filter: 3 Difficulties 3 Difficulties
• Poor Smoothing in Poor Smoothing in High Gradient RegionsHigh Gradient Regions
• Smoothes and bluntsSmoothes and bluntscliffs, valleys & ridgescliffs, valleys & ridges
• Can combine disjoint Can combine disjoint signal regions signal regions
c
ss
Bilateral Filter:Bilateral Filter: 3 Difficulties 3 Difficulties
• Poor Smoothing in Poor Smoothing in High Gradient RegionsHigh Gradient Regions
• Smoothes and bluntsSmoothes and bluntscliffs, valleys & ridgescliffs, valleys & ridges
• Disjoint regions Disjoint regions can blend together can blend together
c
ss
‘‘Blunted Corners’ Blunted Corners’ Weak Halos Weak Halos
Bilateral :Bilateral :
What What we getwe get
‘‘Blunted Corners’ Blunted Corners’ Weak Halos Weak Halos
‘‘Trilateral’Trilateral’resultresult
What What we wantwe want
Try to fix this:Try to fix this:Trilateral Filter (Choudhury 2003)Trilateral Filter (Choudhury 2003)
Goal:Goal: Piecewise linear smoothing, not piecewise constantPiecewise linear smoothing, not piecewise constant
Method:Method:‘Steer’ Bilateral Filter with smoothed gradients ‘Steer’ Bilateral Filter with smoothed gradients
PositionPosition
IntensityIntensity
EXAMPLE:EXAMPLE: remove noise from a piecewise linear scanline remove noise from a piecewise linear scanline
Outline: BilateralOutline: BilateralTrilateral FilterTrilateral Filter
Three Key Ideas:Three Key Ideas:• TiltTilt the filter window the filter window
according to bilaterally-according to bilaterally-smoothed gradientssmoothed gradients
• LimitLimit the filter window the filter window to connected regions to connected regions of similar smoothed gradient.of similar smoothed gradient.
• AdjustAdjust Parameters Parameters from measurements from measurements of the windowed signalof the windowed signal
ccss
Outline: BilateralOutline: BilateralTrilateral FilterTrilateral Filter
Key Ideas:Key Ideas:• TiltTilt the filter window the filter window
according to bilaterally-according to bilaterally-smoothed gradientssmoothed gradients
• LimitLimit the filter window the filter window to connected regions to connected regions of similar smoothed gradient.of similar smoothed gradient.
• AdjustAdjust Parameters Parameters from measurements from measurements of the windowed signalof the windowed signal
ccss
Outline: BilateralOutline: BilateralTrilateral FilterTrilateral Filter
Key Ideas:Key Ideas:• TiltTilt the filter window the filter window
according to bilaterally-according to bilaterally-smoothed gradientssmoothed gradients
• LimitLimit the filter window the filter window to connected regions to connected regions of similar smoothed gradient.of similar smoothed gradient.
• AdjustAdjust Parameters Parameters from measurements from measurements of the windowed signalof the windowed signal
ccss
..
..
BilateralBilateral
..
..
TrilateralTrilateral
. .
• ,,
Trilateral Filter (Choudhury 2003)Trilateral Filter (Choudhury 2003)
• StrengthsStrengths– Sharpens Sharpens cornerscorners– Smoothes similar Smoothes similar gradientsgradients– Automatic Automatic parameter parameter settingsetting– 3-D 3-D mesh de-noisingmesh de-noising, too!, too!
• WeaknessesWeaknesses– S-L-O-W;S-L-O-W; very costly connected-region finder very costly connected-region finder– Shares Bilateral’s Shares Bilateral’s ‘Lonely Outlier Pixel’ artifacts‘Lonely Outlier Pixel’ artifacts– Noise ToleranceNoise Tolerance limits; disrupts ‘tilt’ estimates limits; disrupts ‘tilt’ estimates
NEW IDEA : ‘NEW IDEA : ‘JointJoint’ or ‘’ or ‘CrossCross’’ Bilateral Bilateral’ ’ Petschnigg(2004) and Eisemann(2004)Petschnigg(2004) and Eisemann(2004)
Bilateral Bilateral two kindstwo kinds of weights of weights
NEW :NEW : get them from get them from two kindstwo kinds of images. of images.
• Smooth image Smooth image AA pixels locally, but pixels locally, but• Limit to “similar regions” found in image Limit to “similar regions” found in image BB
Why do this?Why do this? To get To get ‘best of both images’‘best of both images’
OrdinaryOrdinary Bilateral Bilateral FilterFilter
Bilateral Bilateral two kindstwo kinds of weights, one of weights, one image A image A ::
S
AAAGGW
ABFq
qqpp
p qp ||||||1
][rs
cc
ss
Image A:
DomainDomain
RangeRange
f(x)f(x)xx
‘‘JointJoint’ or ‘’ or ‘CrossCross’’ Bilateral Bilateral FilterFilter
NEW:NEW: two kindstwo kinds of weights, of weights, twotwo images images
S
ABBGGW
ABFq
qqpp
p qp ||||||1
][rs
cc
ss
A: Noisy, dim(ambient image)
cc
ss
B: Clean,strong (Flash image)
Image A:Image A: Warm, shadows, but too Noisy Warm, shadows, but too Noisy(too dim for a good quick photo)(too dim for a good quick photo)
No-flash
Image B:Image B: Cold, Shadow-free, Clean Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)(flash: simple light, ALMOST no shadows)
MERGE BEST OF BOTH:MERGE BEST OF BOTH: apply apply‘Cross Bilateral’ or ‘Joint Bilateral’‘Cross Bilateral’ or ‘Joint Bilateral’
(it really is (it really is muchmuch better!) better!)
Recovers Weak Signals Hidden by NoiseRecovers Weak Signals Hidden by Noise
Noisy but Strong…Noisy but Strong…
Noisy and Weak…
+ Noise =
+ Noise =
Ordinary Bilateral Filter? Ordinary Bilateral Filter?
Noisy but Strong…Noisy but Strong…
Noisy and Weak…
BF
BF
Step feature GONE!!Step feature GONE!!
Ordinary Bilateral Ordinary Bilateral
Noisy but Strong…Noisy but Strong…
Noisy and Weak…
Signal too weak to rejectSignal too weak to reject
Range filter preserves signalRange filter preserves signal
‘‘Cross’ or ‘Joint’ Bilateral Idea:Cross’ or ‘Joint’ Bilateral Idea:
Noisy but Strong…Noisy but Strong…
Noisy and Weak…
Range filter preserves signalRange filter preserves signal
Use stronger signal’s range Use stronger signal’s range to set other’s filter weights…to set other’s filter weights…
‘‘JointJoint’ or ‘’ or ‘CrossCross’’ Bilateral Bilateral Filter FilterPetschnigg(2004) and Eisemann(2004)Petschnigg(2004) and Eisemann(2004)
• Useful Residues.Useful Residues. To transfer details To transfer details,,– CBF(A,B)CBF(A,B) to remove A’s noisy details to remove A’s noisy details– CBF(B,A)CBF(B,A) to extract B’s clean details, and to extract B’s clean details, and – Add to Add to CBF(A,B)CBF(A,B) clean, detailed image! clean, detailed image!
•CBF(A,B)CBF(A,B): smoothes image A only;: smoothes image A only;(e.g. the ‘no flash’ image)(e.g. the ‘no flash’ image)
•Limits smoothing to stay within regions Limits smoothing to stay within regions where Image B is ~uniform where Image B is ~uniform (e.g. flash) (e.g. flash)
New Idea:New Idea:NL-Means Filter (Buades 2005)NL-Means Filter (Buades 2005)
• Same goals: ‘Smooth within Similar Regions’Same goals: ‘Smooth within Similar Regions’
• KEY INSIGHTKEY INSIGHT: Generalize, extend ‘Similarity’: Generalize, extend ‘Similarity’– Bilateral:Bilateral:
Averages neighbors with Averages neighbors with similarsimilar intensitiesintensities;;
– NL-Means:NL-Means: Averages neighbors with Averages neighbors with
similar similar neighborhoodsneighborhoods!!
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)
• For each andFor each and
every pixel every pixel p: p:
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)
• For each andFor each and
every pixel every pixel p: p: – Define a small, simple fixed size neighborhood;Define a small, simple fixed size neighborhood;
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)
• For each andFor each and
every pixel every pixel p: p: – Define a small, simple fixed size neighborhood;Define a small, simple fixed size neighborhood;
– Define vector Define vector VVpp: a list of neighboring pixel values.: a list of neighboring pixel values.
Vp = 0.740.320.410.55………
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)‘‘Similar’Similar’ pixels pixels p, qp, q
SMALLSMALL vector distance; vector distance;
|| Vp – Vq ||2
p
q
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)‘‘Dissimilar’Dissimilar’ pixels pixels p, qp, q
LARGELARGE vector distance; vector distance;
|| Vp – Vq ||2
pq
q
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)‘‘Dissimilar’Dissimilar’ pixels pixels p, qp, q
LARGELARGE vector distance; vector distance;
Filter with this!Filter with this!
tiny tiny distdistance ance big big wweighteightbig big distdistance ance tiny tiny wweighteight
|| Vp – Vq ||2
pq
q
wt
dist
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005) p, q p, q neighbors defineneighbors define
a vector distance;a vector distance;
Filter with this:Filter with this:No spatial term!No spatial term!
|| Vp – Vq ||2 pq
S
IVVGGW
INLMFq
qqpp
p qp 2||||||||1
][rs
NL-Means Method:NL-Means Method:Buades (2005)Buades (2005)pixels pixels p, q p, q neighborsneighbors
Set a vector distance;Set a vector distance;
Vector Distance to Vector Distance to pp sets sets weight for each pixel weight for each pixel qq
|| Vp – Vq ||2 pq
S
IVVGW
INLMFq
qqpp
p2||||
1][
r
NL-Means Filter (Buades 2005)NL-Means Filter (Buades 2005)
• NoisyNoisysourcesourceimage:image:
NL-Means Filter (Buades 2005)NL-Means Filter (Buades 2005)
• GaussianGaussianFilterFilter
Low noise,Low noise,
Low detailLow detail
NL-Means Filter (Buades 2005)NL-Means Filter (Buades 2005)
• AnisotropicAnisotropicDiffusion:Diffusion:
(Note (Note ‘stairsteps’:‘stairsteps’:~ piecewise~ piecewiseconstant)constant)
NL-Means Filter (Buades 2005)NL-Means Filter (Buades 2005)
• BilateralBilateralFilter:Filter:
(better, but(better, butsimilarsimilar‘stairsteps’:‘stairsteps’:
NL-Means Filter (Buades 2005)NL-Means Filter (Buades 2005)
• NL-Means:NL-Means:
Sharp,Sharp,
Low noise,Low noise,
Few artifacts.Few artifacts.
Non-Local Similarity (You, 2008)Non-Local Similarity (You, 2008)
• Buades NL Means: vector similarity helps, Buades NL Means: vector similarity helps, but is only shift-invariant…but is only shift-invariant…
• You: You: expand to rotation & scale invariance;expand to rotation & scale invariance;exploit SIFT for similarity finding…exploit SIFT for similarity finding…
Weighted Least-Squares OptimizationWeighted Least-Squares Optimization
• Improved low-halo detail scales….Improved low-halo detail scales….http://www.cs.huji.ac.il/~danix/epd/
SIGGRAPH2008 “Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation” SIGGRAPH2008 “Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation” Z.Farbman, R. Fattal,lD. Lischinski R. SzeliskiZ.Farbman, R. Fattal,lD. Lischinski R. Szeliski
Many More Possibilities: Many More Possibilities: EXPERIMENT!EXPERIMENT!
• Bilateral goals are Bilateral goals are subjectivesubjective;;
--‘Local smoothing within similar regions’--‘Local smoothing within similar regions’--‘Edge-preserving smoothing’--‘Edge-preserving smoothing’--‘Separate large structure & fine detail’--‘Separate large structure & fine detail’--‘Eliminate outliers’--‘Eliminate outliers’--‘Filter within edges, not across them’--‘Filter within edges, not across them’
• It’s simplicity It’s simplicity invites new & inventive answers.invites new & inventive answers.
VariantsVariants
•15 Minutes15 Minutes
•<20 slides<20 slides