IEEE Computer Graphics and Applications Vol. 28, Issue. 2, April 2008
Katrien Jacobs, Celine Loscos, and Greg Ward
Presented by Yuan Xi
School of Electrical Engineering and Computer Science Kyungpook National Univ.
Automatic High Dynamic Range Image Generation for Dynamic Scenes
Abstract Requirement of HDRI generation
– Multi-exposure LDR images • Static scene throughout LDR images
Proposed method – Ghost-free
• LDRI alignment − Focus on camera
• Deleting moving objects − Without camera curve − Independent from contrast between background and moving object
2/26
Introduction High dynamic range
– Weakness of LDR images • Loss of information in under or over exposed area
– High dynamic range image • Combining details in multi-exposure LDR images
Problems of High dynamic range image processing – Acquirement of exposure time
• Changing exposure time manually – Ghost
• Camera movement • Object movement
3/26
Instance for ghost free method
Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people walk through the viewing window. (b) An HDRI created from the sequence shown in (a) using conventional methods, showing ghosting effects (black squares). (c) Uncertainty image (UI) shows regions of high uncertainty (bright) due to the dynamic behaviour of those pixels. (d) HDRI of the same scene after applying movement removal using UI.
4/26
Background Debevec’s method
– Depend on good alignment between LDRIs
Rescal’s method – Providing manual image alignment method
Reinhard’s method – Generating binary-map with weighted variance between LDRIs
Kang’s method – Using camera response function to normalize LDRIs – Performing local image alignment using gradient-based optical flow
5/26
MTB method – Median threshold bitmap
• Threshold − Median pixels value
– Ghost detection • Getting summation of all bitmap image
− Classifying pixel neither 0 or N(number of LDR images) as movement
Sand’s method – Basis on feature-based method – Without camera response curve
Khan’s method(segmentation method) – Using kernel density estimation method to estimation motion
area
6/26
HDRI Generation: An Overview Flowchart of proposed method
Fig. 2. HDRI generation methodology: the rounded white and grey boxes are processes that operate on input data and produce output data. The rounded grey boxes are modules developed for this paper.
7/26
Camera Alignment Generate alignment method
– Calculating camera transformation with scene features • Euclidean transformation
− Rotation and translation • Scene features
− Different intensity − Different color − Edge
– Weakness • Impossible find edges perfectly in every image
8/26
• Failing instance of edge finding
Fig. 3. (a,b) Two LDRIs captured with different exposures. (c,d) Edge images of two LDRIs. (e,f) Bitmap images of two LDRIs after applying MTB transformation. 9/26
Proposed alignment method – Using MTB method instead of canny edge detection
• Threshold set as median intensity of each image – Using optimization method
XOR E N D
Minima?
NO!
YES! Euclidean transformation
10/26
Movement Detection Generate alignment method
– Detecting movement clusters • Clusters of pixels affected by movement in any of LDRIs
– Using two evaluation method • Movement detection based on variance • Contrast-independent movement detection
11/26
Movement detection based on variance – Focus on variance over irradiance images
• Pixels affected by movement showing large irradiance variation • Movement derived from variance image
– Achieving variance image • Definition of weighted variance:
− Weighted sum of squares at each pixel over the square of weighted average, the quantity minus 1
• Aim of weight-processing
− Decreasing effect of redundancy information stored in distortion pixels
2
0 0
2 2
0 0
( , ) ( , ) / ( , )( , ) 1
( ( , ) ( , )) / ( ( , ))
N N
i i ii i
N N
i i ii i
W k l E k l W k lVI k l
W k l E k l W k l
= =
= =
= −∑ ∑
∑ ∑(1)
where is weight used during HDRI generation.
( , )iW k l
12/26
• Weight-function − Hat function
13/26
• Binary image making with threshold − Setting threshold to
– Flowchart of proposed method 0.18
Fig. 4. An adaptation of figure 2 illustrates where movement detector based on variance fits inside general HDRI generation framework. Variance detector requires knowledge of camera curve, and therefore movement detector takes place after camera curve calibration.
14/26
– Weakness of movement detection based on variance • Other influences existing, besides remaining camera
misalignment and movement object − Camera curve
» Fail to convert intensity values to irradiance values − Weighting factors − Inaccuracies in exposure speed and aperture width
15/26
Contrast-independent movement detection – Definition of entropy
• In information theory − Uncertainty that remains about a system, after having taken into
account observable properties • Entropy of variable is given by: ( )H X X
( ) ( ) log( ( ))x
H X P X x P X x= − = =∑ (2)
where is random variable with probability function . ranging over a certain interval, for instance [0,255].
X ( ) ( )p x P X x= =x
16/26
– Entropy providing information • Entropy of image has positive value between [0, ]
− The lower entropy, the more less different intensity values • Actual order or organization of pixel intensities in image does not
influence entropy − Only focus on entirety, ignore specific distribution
• Applying scaling factor on intensity values of image does not change its entropy, if intensity values do not saturate
• Entropy of image gives measure of uncertainty of pixels in image − All intensity values are equal: entropy is zero − All intensity values are different: entropy is one
log( )M
17/26
– Calculating UI(uncertainty image) with 2D window • Generating entropy image from LDR image
− One pixel value in entropy image corresponding to entropy value of around region(2D window) at same position 1
0( , ) ( ) log( ( ))
M
ix
H k l P X x P X x−
=
= − = =∑ (3)
where is derived from normalized histogram constructed from the intensity values of pixels within 2D window. and, over all pixels in:
( )P X x=
p
{ ( : , : }ip L k k l lω ω ω ω∈ − + − + (4)
18/26
– Uncertainty image • Local weighted entropy difference
1,
,10 0
,0 0
( , ) ( , )j iN
i ji jj iN
i ji j
i j
UI k l h k lυ
υ
<−
<−= =
= =
=∑∑∑∑
(5)
, ( , ) | ( , ) ( , ) |i j i jh k l H k l H k l= − (6)
, min( ( , ), ( , ))i j i jW k l W k lυ = (7)
19/26
HDRI Generation Final flowchart for proposed method
Fig. 5. An adaptation of figure 2 illustrates where contrast-independent movement detector, explained in section V-B, fits inside general HDRI generation framework. Movement detector does not require knowledge about camera curve, therefore movement detector can take place before camera calibration.
20/26
Results Comparison of resulting aligned or not
(a) (d)
Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f).
21/26
(b) (e)
Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f).
22/26
Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f).
(c) (f)
23/26
HDRI generation and movement(general) removal
(d)
(a) (b)
(c)
Fig. 7. HDRI generation and movement removal for the exposure sequence shown in figure 1 (a). (a) HDRI after object movement removal using variance detector discussed in section V-A. (b) HDRI after object removal using uncertainty detector discussed in section V-B. (c) Variance image V I used to generate (a). (d) Uncertainty image UI used to generate (b).
24/26
HDRI generation and movement(fluid) removal
(d)
(a) (b)
(c)
Fig. 8.(a) HDRI without movement removal: leaves on left hand side show considerable ghosting. (b) HDRI after movement removal using uncertainty image UI shown in (d). (c) Variance image VI. (d) Uncertainty image UI used to generate (b).
25/26
Proposed method – Camera alignment
• Using MTB method − More better performance than simple edge based methods
– Novel criterion : Entropy • Contrast-independent movement detection • Proposed definition of UI(uncertainty image)
Experimental results – More stable than existing method – Pleasing resulting for fluid ghosts
Future work – Research for more better alignment method
Conclusion And Future Work
26/26
