Date post: | 21-Dec-2015 |
Category: |
Documents |
Upload: | sharon-floyd |
View: | 217 times |
Download: | 0 times |
EE565 Advanced Image Processing Copyright Xin Li 2008
1
Wavelet-based Texture Synthesis Pyramid-based (using steerable pyramids)
• Facilitate the statistical modeling
Histogram matching• Enforce the first-order statistical constraint
Texture matching• Alternate histogram matching in spatial and wavelet
domain
Boundary handling: use periodic extension Color consistency: use color transformation
Basic idea: two visually similar textures will also have similar statistics
EE565 Advanced Image Processing Copyright Xin Li 2008
2
Histogram MatchingGeneralization of histogram equalization (the target is the histogramof a given image instead of uniform distribution)
EE565 Advanced Image Processing Copyright Xin Li 2008
3
Histogram Equalization
x
t
thLy0
)(Uniform
Quantization
L
t
th0
1)(Note:
L
1
x
t
ths0
)(
x
L
y
0
cumulative probability function
EE565 Advanced Image Processing Copyright Xin Li 2008
4
MATLAB Implementationfunction y=hist_eq(x)
[M,N]=size(x);for i=1:256 h(i)=sum(sum(x= =i-1));End
y=x;s=sum(h);for i=1:256 I=find(x= =i-1); y(I)=sum(h(1:i))/s*255;end
Calculate the histogramof the input image
Perform histogramequalization
EE565 Advanced Image Processing Copyright Xin Li 2008
6
Histogram Specification
ST
S-1*T
histogram1 histogram2
?
EE565 Advanced Image Processing Copyright Xin Li 2008
7
Texture MatchingObjective: the histogram of both subbands and synthesized image matches the given template
Basic hypothesis: if two texture images visually look similar, then theyhave similar histograms in both spatial and wavelet domain
EE565 Advanced Image Processing Copyright Xin Li 2008
9
I.I.D. Assumption Challenged
If wavelet coefficients of each subband are indeed i.i.d., then random permutation of pixels should produce another image of the same class (natural images)
The fundamental question here: does WT completely decorrelate image signals?
EE565 Advanced Image Processing Copyright Xin Li 2008
10
Image Example
High-band coefficientspermutation
You can run the MATLAB demo to check this experiment
EE565 Advanced Image Processing Copyright Xin Li 2008
11
Another Experiment
-4 -3 -2 -1 0 1 2 3 4-4
-3
-2
-1
0
1
2
3
4
5
Joint pdf of two uncorrelated random variables X and Y
X
Y
EE565 Advanced Image Processing Copyright Xin Li 2008
12
Joint PDF of Wavelet Coefficients
Neighborhood I(Q): {Left,Up,cousin and aunt}
X=
Y=
Joint pdf of two correlated random variables X and Y
EE565 Advanced Image Processing Copyright Xin Li 2008
13
Texture Synthesis via Parametric Models in the Wavelet Space
Instead of matching histogram (nonparametric models), we can buildparametric models for wavelet coefficients and enforce the synthesizedimage to inherit the parameters of given image
Model parameters: 710 parameters were used in Portilla and Simoncelli’s experiment (4 orientations, 4 scales, 77 neighborhood)
Basic idea: two visually similar textures will also have similar statistics
EE565 Advanced Image Processing Copyright Xin Li 2008
14
Statistical Constraints
Four types of constraints• Marginal Statistics
• Raw coefficient correlation
• Coefficient magnitude statistics
• Cross-scale phase statistics
Alternating Projections onto the four constraint sets• Projection-onto-convex-set (POCS)
EE565 Advanced Image Processing Copyright Xin Li 2008
15
Convex Set
A set Ω is said to be convex if for any two point yx ,We have 10,)1( ayaax
Convex set examples
Non-convex set examples
EE565 Advanced Image Processing Copyright Xin Li 2008
16
Projection Operator
f
g
Projection onto convex set C
C ||}||min||||{ fxfxCxPfgCx
In simple words, the projection of f onto a convex set C is theelement in C that is closest to f in terms of Euclidean distance
EE565 Advanced Image Processing Copyright Xin Li 2008
17
Alternating Projection
X0
X1
X2
X∞
Projection-Onto-Convex-Set (POCS) Theorem: If C1,…,Ck areconvex sets, then alternating projection P1,…,Pk will convergeto the intersection of C1,…,Ck if it is not empty
Alternating projection does not always converge in the caseof non-convex set. Can you think of any counter-example?
C1
C2
EE565 Advanced Image Processing Copyright Xin Li 2008
18
Convex Constraint Sets
● Non-negative set
}0|{ ff
● Bounded-value set
}2550|{ ff
● Bounded-variance set
}||||{ 2 Tgff
A given signal
}|{ BfAf or
EE565 Advanced Image Processing Copyright Xin Li 2008
19
Non-convex Constraint Set
Histogram matching used in Heeger&Bergen’1995
Bounded Skewness and Kurtosis
skewness kurtosis
The derivation of projection operators onto constraint sets are tediousare referred to the paper and MATLAB codes by Portilla&Simoncelli.
EE565 Advanced Image Processing Copyright Xin Li 2008
21
Image Examples (Con’d)
original
synthesized
EE565 Advanced Image Processing Copyright Xin Li 2008
23
Summary on Wavelet-based Texture Synthesis
Textures represent an important class of structures in natural images – unlike edges characterizing object boundaries, textures often associate with the homogeneous property of object surfaces
Wavelet-domain parametric models provide a parsimonious representation of high-order statistical dependency within textural images
EE591b Advanced Image Processing Copyright Xin Li 2006
24
Image Denoising
Theory of linear estimation Spatial domain denoising techniques
• Conventional Wiener filtering
• Spatially adaptive Wiener filtering Wavelet domain denoising
• Wavelet thresholding: hard vs. soft
• Wavelet-domain adaptive Wiener filtering Latest advances (we will cover it after patch-
based modeling)• Patch-based image denoising
• Learning-based image denoising
EE591b Advanced Image Processing Copyright Xin Li 2006
25
Denoising Problem
Noisy measurements
signal N(0,σw2)
MMSE estimator
aYX ˆ
Wiener’s idea To simplify estimation by computing the bestestimator that is a linear scaling of Y
Difficulty: we need to know conditional pdf
WXY
]|[ˆ YXEX
EE591b Advanced Image Processing Copyright Xin Li 2006
26
Orthogonality Principle
A linear estimator X of a random variable X aYX ˆ
0})ˆ{( YXXE
^
Minimizes E{(X-X)2} if and only if ^
Geometric Interpretation
X
Y
X-X̂
X̂
EE591b Advanced Image Processing Copyright Xin Li 2006
27
Linear MMSE Estimation
YXwx
x22
2
ˆ
),0(~ 2xNX For Gaussian signal
The optimal LMMSE estimation is given by
22
22
wx
xwMMSE
And it achieves
Note: it can be shown such linear estimator is indeedE[X|Y] for Gaussian signal
EE591b Advanced Image Processing Copyright Xin Li 2006
28
What if Signal Variance is Unknown?
222ˆ wx y
Maximum-likelihood estimation of 2x is given by
Since variance is nonnegative, we modify it
],0max[ˆ 222wx y
When multiple observations yi’s are available, we have
]1
,0max[ˆ 2
1
22w
N
iix y
N
EE591b Advanced Image Processing Copyright Xin Li 2006
29
From Scalar to Vector Case
Suppose X is a Gaussian process whose covariance matrix is adiagonalized matrix RX=diag{ηm}(m=0,…,N-1), the linear MMSEestimator is given by
1
022
2
][][ˆN
m wm
m mYnX
(A)
1
022
222}||ˆ{||
N
m wm
wmXXE
and the minimal MSE is given by
EE591b Advanced Image Processing Copyright Xin Li 2006
30
Decorrelating
Q: What if X={X[0],…,X[N-1]} is correlated (i.e., Rx is not diagonalized)?
A: We need to transform X into a set of uncorrelated basis and then apply the above result.
The celebrated Karhunen-Loeve Transform does this job!
XT R*
Diagonal matrix
XX T*'
Karhunen-Loeve Transform
EE591b Advanced Image Processing Copyright Xin Li 2006
31
Transform-Domain Denoising
ForwardTransform
InverseTransform
Denoisingoperation
e.g.,KLTDCTWT
e.g.,Linear Wiener filteringNonlinear Thresholding
Noisysignal
denoisedsignal
The performance of such transform-domain denoising is determinedby how well the assumed probability model in the transform domainmatches the true statistics of source signal (optimality can only beestablished for the Gaussian source so far).
EE591b Advanced Image Processing Copyright Xin Li 2006
32
Image Denoising
Theory of linear estimation Spatial domain denoising techniques
• Conventional Wiener filtering
• Spatially adaptive Wiener filtering Wavelet domain denoising
• Wavelet thresholding: hard vs. soft
• Wavelet-domain adaptive Wiener filtering Latest advances
• Patch-based image denoising
• Learning-based image denoising
EE591b Advanced Image Processing Copyright Xin Li 2006
33
Conventional Wiener Filtering
Basic assumption: image source is modeled by a stationary Gaussian process
Signal variance is estimated from the noisy observation data
Can be interpreted as a linear frequency weighting
EE591b Advanced Image Processing Copyright Xin Li 2006
34
Linear Frequency Weighting
),(),(
),(),(
),,(),(),(ˆ
2121
2121
212121
wwSwwS
wwSwwH
wwYwwHwwX
WX
X
22
2
,ˆwx
xaaYX
FT
Power spectrum |X|2
EE591b Advanced Image Processing Copyright Xin Li 2006
35
Image Example
Noisy, =50 (MSE=2500) denoised (MSE=1130)
EE591b Advanced Image Processing Copyright Xin Li 2006
36
Image Example (Con’d)
Noisy, =10 (MSE=100) denoised (MSE=437)
EE591b Advanced Image Processing Copyright Xin Li 2006
37
Conclusions from the Experiments
Why did it Fail? • Nonstationary
• NonGaussian
• Poor modeling
How to improve?• Achieve spatial adaptation
• Use linear transform
• Putting them together
EE591b Advanced Image Processing Copyright Xin Li 2006
38
Spatially Adaptive Wiener Filtering
Basic assumption: image source is modeled by a nonstationary Gaussian process
Signal variance is locally estimated from the windowed noisy observation data
]1
,0max[ˆ 2
1
22w
N
iix y
N
T
T
N=T2
Recall
EE591b Advanced Image Processing Copyright Xin Li 2006
39
Image Example
Noisy, =10 (MSE=100) denoised (T=3,MSE=56)
EE591b Advanced Image Processing Copyright Xin Li 2006
40
Image Example (Con’d)
Noisy, =50 (MSE=2500) denoised (MSE=354)
EE591b Advanced Image Processing Copyright Xin Li 2006
41
Image Denoising
Theory of linear estimation Spatial domain denoising techniques
• Conventional Wiener filtering
• Spatially adaptive Wiener filtering Wavelet domain denoising
• Wavelet thresholding: hard vs. soft
• Wavelet-domain adaptive Wiener filtering Latest advances
• Patch-based image denoising
• Learning-based image denoising
EE591b Advanced Image Processing Copyright Xin Li 2006
42
A Toy Example
18
2 2 N
MSE
01 N-1
σ2=4/N
Wiener Filter Estimator:
22sin4
)( mX N
m
NmP
12
1 1
0
21
022
22
N
mm
N
m m
mMSE
otherwise
nYifnYnX0
2
1|][|][][
~
Nonlinear thresholding Estimator:
]mod)[(][ NPnfnX f(n)
An integer uniformlydistributed on [0,N-1]
Conclusion: for sparse signals, linear filtering is not as effective asNonlinear thresholding
EE591b Advanced Image Processing Copyright Xin Li 2006
43
Why Wavelet Denoising?
We need to distinguish spatially-localized events (edges) from noise components
More about noise components
Wavelet is such a basis because exceptional event generates identifiable exceptional coefficients due to its good localization property in both spatial and frequency domain
As long as it does not generate exceptions
Additive White Gaussian Noise is just one of them
EE591b Advanced Image Processing Copyright Xin Li 2006
44
Wavelet Thresholding
DWT IWTThresholdingY X
~
otherwise
TnYifnYnX
0
|][|][][
~Hard thresholding
Soft thresholding
TnY
TnYTnY
TnYTnY
nX
|][|0
][][
][][
][~
Noisysignal
denoisedsignal
EE591b Advanced Image Processing Copyright Xin Li 2006
45
Choice of Threshold
NT elog2
Donoho and Johnstone’1994
Gives denoising performance close to the “ideal weighting”
][][
][][
~22
2
nYn
nnX
Reference: S. Mallat, “A Wavelet Tour of Signal Processing”, Section 10.2 (pp. 435-453)
EE591b Advanced Image Processing Copyright Xin Li 2006
46
Soft vs. Hard thresholding
|][||][~
|1][
][],[][
~22
2
nYnXn
nanaYnX
● It can be also viewed as a computationally efficient approximationof ideal weighting
|][||][~
|][,][][~
nYnXTnYTnYnX soft
ideal
● Soft-thresholding has the same upper bound as hard-thresholding asymptotically and larger error than hard-thresholding at the same threshold value, but perceptually it works better.
● Shrinking the amplitude by T guarantees with a high probability that.
|][||][~
| nXnX
EE591b Advanced Image Processing Copyright Xin Li 2006
47
Denoising Example
noisy image(σ2=100)
Wiener-filtering (ISNR=2.48dB)
Wavelet-thresholding (ISNR=2.98dB)
2
2
10||
~||
||||log10
XX
YXISNR
X: original, Y: noisy, X: denoised~
Improved SNR
EE591b Advanced Image Processing Copyright Xin Li 2006
48
Duality with Image Coding*
DWT IWTThresholding
DWT IWTQ Q-1Channel
Image denoising system
Image coding system
EE591b Advanced Image Processing Copyright Xin Li 2006
49
Basis Selection Problem
G0
G1
)(ˆ nxx(n)
H0
H1
y0(n)
y1(n)
x(n)
H0
H1
22 G0
22 G1
)(ˆ nx
s(n)
d(n)
complete expansion (with decimation)
overcomplete expansion (without decimation)
TceTce
-1
Toe Toe-1
EE591b Advanced Image Processing Copyright Xin Li 2006
50
What do We Buy from Redundancy?
0 1 N-1… …
x(n)
H1
T
-T
EE591b Advanced Image Processing Copyright Xin Li 2006
51
Translation Invariance (TI) Denoising
Toe Toe-1Thresholding
Tce Tce-1Thresholding
Tce Tce-1Thresholding
z
+
x(n) )(ˆ nx
x(n) )(ˆ1 nx
)(ˆ2 nx
2
)(ˆ)(ˆ)(ˆ 21 nxnx
nx
Implementation based on overcomplete expansion
Implementation based on complete expansion
z-1
EE591b Advanced Image Processing Copyright Xin Li 2006
52
2D Extension
Noisy image
Tce Tce-1ThresholdingWD =
shift(mK,nK) WD shift(-mK,-nK)
shift(m1,n1) WD shift(-m1,-n1)
Avg
denoised image
(mk,nk): a pair of integers, k=1-K (K: redundancy ratio)
EE591b Advanced Image Processing Copyright Xin Li 2006
53
Example
Wavelet-thresholding (ISNR=2.98dB)
Translation-Invariant thresholding (ISNR=3.51dB)
EE591b Advanced Image Processing Copyright Xin Li 2006
54
Go Beyond Thresholding
Challenges with wavelet thresholding• Determination of a global optimal threshold
• Spatially adjusting threshold based on local statistics
How to go beyond thresholding?• We need an accurate modeling of wavelet
coefficients – nonlinear thresholding is a computationally efficient yet suboptimal solution
EE591b Advanced Image Processing Copyright Xin Li 2006
55
Spatially Adaptive Wiener Filtering in Wavelet Domain
Wavelet high-band coefficients are modeled by a Gaussian random variable with zero mean and spatially varying variance
Apply Wiener filtering to wavelet coefficients, i.e.,
][][
][][
~22
2
nYn
nnX
estimated in the same wayas spatial-domain (Slide 15)
EE591b Advanced Image Processing Copyright Xin Li 2006
56
Practical Implementation
]1
,0max[ˆ 2
1
22w
N
iix y
N
T
T
N=T2
Recall
Conceptually very similar to its counterpart in the spatial domain
YXwx
x22
2
ˆ
ˆˆ
In demo3.zip, you can find a C-coded example (de_noise.c)
(ML estimation of signal variance)
EE591b Advanced Image Processing Copyright Xin Li 2006
57
Example
Translation-Invariant thresholding (ISNR=3.51dB)
Spatially-adaptive wiener filtering (ISNR=4.53dB)
EE591b Advanced Image Processing Copyright Xin Li 2006
58
Further Improvements
Gaussian scalar mixture (GSM) based denoising (Portilla et al.’ 2003)• Instead of estimating the variance, it explicitly
addresses the issue of uncertainty with variance estimation
Hidden Markov Model (HMM) based denoising (Romberg et al.’ 2001)• Build a HMM for wavelet high-band
coefficients (refer to the posted paper)
EE591b Advanced Image Processing Copyright Xin Li 2006
59
Gaussian Scalar Mixture (GSM)
Model definition: u~N(0,1)
Noisy observation model
Gaussian pdf
scale (variance) parameter
EE591b Advanced Image Processing Copyright Xin Li 2006
60
Basic IdeaIn spatially adaptive Wiener filtering, we estimate the variancefrom the data of a local window. The uncertainty with such varianceestimation is ignored. In GSM model, such uncertainty is addressedthrough the scalar z (it determines the variance of GSM). Instead of using a single z (estimated variance), we build a probabilitymodel over z, i.e., E{x|y}=Ez{E{x|y,z}}
EE591b Advanced Image Processing Copyright Xin Li 2006
61
Posterior Distribution
where
Due to
is so-called Jeffery’s prior
Question: What is E{xc|y,z}?
Bayesian formula
(proof left as exercise)
EE591b Advanced Image Processing Copyright Xin Li 2006
62
GSM Denoising Algorithm
http://decsai.ugr.es/~javier/denoise/index.htmlMATLAB codes available at:
EE591b Advanced Image Processing Copyright Xin Li 2006
63
Image Examples
Noisy, =50 (MSE=2500) denoised (MSE=201)
EE591b Advanced Image Processing Copyright Xin Li 2006
64
Image Examples (Con’d)
Noisy, =10 (MSE=100) denoised (MSE=31.7)
EE591b Advanced Image Processing Copyright Xin Li 2003
65
Early Attempts
Each band is modeled by a Guassian random variable with zero mean and unknown variance (e.g., WSQ)
Only modest gain over JPEG (DCT-based) is achieved
Question: is this an accurate model?
and how can we test it?
EE591b Advanced Image Processing Copyright Xin Li 2003
66
Proof by Contradiction (I)
Suppose each coefficient X in a high band does observeGaussian distribution, i.e., X~N(0,σ2), then flip the sign ofX (i.e., replace X with –X) should not matter and generatesanother element in Ω (i.e., a different but meaningful image)
Assumption: our modeling target Ω is the collection of natural images
Let’s test it!
EE591b Advanced Image Processing Copyright Xin Li 2003
67
Proof by Contradiction (II)
DWT
sign flip
IWT
EE591b Advanced Image Processing Copyright Xin Li 2003
68
What is wrong with that?
Think of two coefficients: one in smooth region and the other around edge, do they observe the same probabilistic distribution?
Think of all coefficients around the same edge, do they observe the same probabilistic distribution?
Ignorance of topology and geometry
EE591b Advanced Image Processing Copyright Xin Li 2003
69
The Importance of Modeling Singularity Location Uncertainty Singularities carry critical visual
information: edges, lines, corners … The location of singularities is important
• Recall locality of wavelets in spatial-frequency domain
Singularities in spatial domain → significant coefficients in wavelet domain
EE591b Advanced Image Processing Copyright Xin Li 2003
70
Where-and-What Coding
Communication context
Where The location of significant coefficients
What The sign and magnitude of significant
coefficients
Alice Bob
communicationchannelpicture
EE591b Advanced Image Processing Copyright Xin Li 2003
71
1993-2003
Embedded Zerotree Wavelet (EZW)’1993 Set Partition In Hierarchical Tree
(SPIHT)’1995 Space-Frequency Quantization (SFQ)’ 1996 Estimation Quantization (EQ)’1997 Embedded Block Coding with Optimal
Truncation (EBCOT)’2000 Least-Square Estimation Quantization
(LSEQ)’2003
EE591b Advanced Image Processing Copyright Xin Li 2003
73
Zerotree Data Structure
Parent-and-Children Ancestor-and-Descendent
EE591b Advanced Image Processing Copyright Xin Li 2003
74
Zerotree Terminology
Zerotree root (ZRT): it and its all descendants are insignificant
Isolated zero (IZ): it is insignificant but its descendant is not
Positive significant (POS): it is significant and have a positive sign
Negative significant (NEG): it is significant and have a negative sign
EE591b Advanced Image Processing Copyright Xin Li 2003
77
Toy Example
Note: T=32
LH1 contains POS
LH1 contains POS
EE591b Advanced Image Processing Copyright Xin Li 2003
78
Where-and-What Interpretation
Zerotree data structure effectively resolves the location uncertainty (where) of insignificant coefficients
The dominant and subordinate passes defined in EZW can be viewed as “where” and “what” coding respectively
Dyadic choice of T values (i.e., T=128,64, 32,16,…) renders embedded coding
EE591b Advanced Image Processing Copyright Xin Li 2003
79
A Simpler Two-Stage Coding
Position coding stage (where)• Generate a binary map indicating the location
of significant coefficients (|X|>T)
• Use context-based adaptive binary arithmetic coding (e.g., JBIG) to code the binary map
Intensity coding stage (what)• Code the sign and magnitude of significant
coefficients
EE591b Advanced Image Processing Copyright Xin Li 2003
80
A Different Interpretation
Two-class modeling of high-band coefficients• Significant class: |X|>T
• Insignificant class: |X|<T
Why does classification help?• Nonstationarity of image source
• A probabilistic modeling perspective
EE591b Advanced Image Processing Copyright Xin Li 2003
81
Classification-based Modeling
),0(~ 200 NX
Insignificant class
),0(~ 211 NX
Significant class
Mixture
20
21
2201 )1(),,0(~)1( aaNXaaXX
EE591b Advanced Image Processing Copyright Xin Li 2003
82
Classification Gain
RRD 22 2)(
Without classification
With classification
RaaRD 221
)1(20 2)('
Classification gain
0)1(
log10)('
)(log10
21
)1(20
20
21
1010
dBaa
dBRD
RDG
aa
EE591b Advanced Image Processing Copyright Xin Li 2003
84
Summary of Wavelet Coding
SPIHT: a simpler yet more efficient implementation of EZW coder
SFQ: Rate-Distortion optimized zerotree coder
EQ: Rate-Distortion optimized backward classification strategy
EBCOT (adopted by JPEG2000): a versatile embedded coder