ACM Transactions on Graphics
vol. 31, No. 4, 2012
Xuan Yang, Linling Zhang, Tien-Tsin Wong, Pheng-Ann Heng
Presented by Ran Shu
School of Electrical Engineering and Computer Science
Kyungpook National Univ.
Binocular tone mapping
Abstract
Binocular single vision phenomenon
– Fusing two displaced images
• With difference in detail and contrast and luminance
– Proposed binocular tone mapping framework
• Generating binocular LDR image pair
− Preserving more human-perceivable visual content
− Left for tone-mapped LDR image
» Without loss of generality
− Right for proposed optimally synthesize image
• LDR image pair presenting more human-perceivable visual richness
− Without triggering visual discomfort
» Using novel binocular viewing comfort predictor
» Based on finding in vision science
2/35
– Binocularly tone-mapped image pair
Fig. 1. LDR image pairs. Left shows global contrast, right shows
more local details
3/35
Introduction
Two binocular vision phenomenon
– Stereopsis for existing binocular display
• 3D movies
– Binocular single vision
• Fusing images from two eyes as single percept
− Even two images with difference
» Complex non-linear neurophysiological process
4/35
Proposed method
– Utilizing existing LDR binocular display
• Simultaneously presenting contrast and details in HDR images
− By proposed binocular tone mapping framework
– Given an HDR image and tone-mapped LDR image
• Generated by existing tone mapping techniques
• Optimally synthesizing counterpart LDR image
− Using proposed framework BVCP
− Presenting more human-perceivable visual content
5/35
Proposed binocular viewing comfort predictor
– Guiding binocular tone mapping
• Based on findings in vision science and experimental results
− Avoiding visual discomfort
− Making more difference in two views
6/35
Binocular single vision
– Combing two different images from two eyes into single vision
• Occurring only in small volume of retinal area
− Around fixating area
− Non-linear combination of luminance and contrast and color
» Regarded as combination of binocular fusion and suppression
– Binocular fusion
• Process of superimposing
• Combining similar content from two views into one percept
− Under two similar or same image
– Binocular suppression
• Under submissive view
– Binocular rivalry
Related work
7/35
– Binocular fusion and suppression and rivalry
Fig. 2. Fusion, suppression and rivalry.
8/35
– Assessment for binocular viewing comfort
• Existing metrics considering visible difference
− Between two images looking with both eyes
» Without considering binocular vision
• Designing brand new metric
− Binocular viewing comfort predictor
» Based on theories and experimental results
9/35
Tone mapping
– Using previous tone mapping method
• Global operators
− Adaptive logarithmic mapping
• Local operators
− Bilateral filtering approach
− Gradient domain HDR compression
− Perceptual based contrast processing
10/35
Overview
Overview of binocular tone mapping
– Input HDR image and left tone-mapping LDR image
– Generating optimal right LDR image
• Using same tone-mapping operator as left
• Objective function composed of two metrics
− Visible difference predictor
− Binocular viewing comfort predictor
Fig. 3. System overview.
11/35
BVCP
Fusional area
– Based on visual agreement of neighborhoods
• Panum’s fusional area
• Occupying constant solid angle subtended at eye
• Considering whole area for fusion stability
− In terms of contour and contrast and luminance and color difference
Fig. 4. Fusional area.
12/35
– Radius of fusional area
– Approximating fusional area by rectangle
• Go over all pixel pairs from LDR image pair
− Considering corresponding fusion areas
» For measuring fusion stability
tanfr d (1)
where is the maximal retinal disparity, which is around 60 to 70 arcmin
13/35
Contour fusion
– Contour difference more important than contrast or color difference
– Corresponding contour no need to exactly same
• Motor fusion
− Superimposing corresponding points or similar contour
» By movement of our eyes
• Sensory fusion
− Combing two views into one unified percept
Fig. 6. Motor fusion and sensory fusion. Fig. 5. Contour dominance.
14/35
– Contour has different definitions in different domains
• Applying scale representation to fusional area
− Constructing pyramid
» Fourier transform fusional area to frequency domain
» Applying pyramid of LPF in frequency domain
» Mesa filter as LPF
» Inversely Fourier transforming each low-passed frequency images
» Obtaining pyramid of low-passed fusional areas
Fig. 7. Mesa pyramid.
15/35
– Measuring color difference between adjacent pixels
• Defined as 2-norm distance of colors in LAB color space
• Color difference between a pair of pixels
1 2 1 2, ,k k k
CL p p E L p L p (2)
where is color difference of fusional area,
is color difference of two pixel.
kL
CE
Fig. 9. Contour matching.
16/35
• Looking up decision table
− With JND and obvious color difference
− Considering count of fusible pixels in fusional area
− Contour fusion state at higher levels override lower
− Final fusion state of two fusional areas
Table 1. Decision table for contour fusion where J=JND, O=OCD
1 2
1 2
1 2,
1 2,
0, , 0
1, , 0
1,
k
p p
k k
cf p p
if S p p
B elseif S p p
otherwise
(3)
1 0k k k
cf cf cfB B if B (4)
S
cf cfB B (5)
17/35
Contour and regional contrasts
– Two types of contrast influencing binocular single vision
• Contour contrast
− Coexisting with contour if detected by human eye
− Matched contour pair generally helping fusion
» Except obviously inversed contrast
• Regional contrast
− Contrast between two regions by adopting restrictive constraint
1 2 1 2 1 2
1 2 1 2
, 1 , ,
, , 0
k k k k k
k k k k
S p p if C L p L p OCD and C R p R p OCD
and C L p L p C R p R p
(6)
where computes the lightness difference between
the pixel pair
1 2,C c c
1 2,c c
2
11, ,
4
1,
pjrc
if E L p R p DCDrB
otherwise
(7)
where are two corresponding pixels located at
position p in L and R, respectively.
,L p R p
18/35
Failure of rivalry
– Contrast below certain threshold
• Stable single percept forming regardless of contour fusion
− Happen in low-contrast regions
Fig. 8. Failure of Rivalry.
19/35
– Measuring contrast in log space
• As log percentage contrast between pixel pair
• Correlation between luminance and frequency and contrast
1 2
1 2 10
1 2
, logY Y
P c cY Y
(8)
where are the normalized luma of ,
maps the normalized luma Y in [0,1] to the physical
measurement in the unit of
1 2,Y Y 1 2,c c
Y
2/cd m
(a) (b)
Fig. 10. Luminance and freq. vs. contrast. 20/35
• Obtaining two continuous curves
• Function of both luminance and spatial frequency
2
10 10min 2,1.999 0.362log 0.026logT l l l (9)
2 3
10min 2, log 3.557 1.334 1.881 0.108T f f f f (10)
where is average luminance in ,
is spatial frequency measured in cycles per degree,
represents log percentage contrast threshold.
l
f
2/cd m
T
2 3 2
10 10 10, min 2, log 3.557 1.334 1.881 0.108 0.514 0.362log 0.026logT l f f f f l l (11)
where is user parameter and is set to a value in [-0.15,0.15]
Fig. 11. The constructed log percentage contrast threshold.
21/35
– Contrast threshold for left fusional area
• Obtained by feeding parameters
• Take second revision to fusion state variable
,k
L L kT T l f (12)
1 2 1 2 1 2 1 2, 0 , 0 , ,k k k k k k k kS p p if S p p and P L p L p T and P R p R p T (13)
22/35
The overall fusion predictor
– Predict ultimate viewing comfort
• For arbitrary pair of fusional areas
1, 1
,
rc
cf
if BB
B otherwise
(14)
Fig. 12. Visualization of BVCP assessment.
23/35
Optimization
Adopt visible difference predictor
– Detecting visible difference
• With VDP and proposed BVCP
− Maximizing E without Violating BVCP
,
1,
i j
E H V i j (15)
where is a user-defined probability threshold and generally set as 75%,
is the Heaviside step function,
is the total number of pixels in the images.
H
24/35
– Visualization of binocular visual of four tone mapping operators
(a) Drago (b) Mantiuk
(c) Durand (d) Fattal
Fig. 13. Visualization of binocular visual differences of four tone
mapping operators for the HDR example in Fig. 20.
25/35
Results and Discussions
Framework adopted tone mapping operators
– Supporting four tone mapping operators
• Experimented with four operators
Fig. 14. Optimal LDR
image pairs generated by
our framework using four
tone mapping operators.
(a) Drago (b) Mantiuk
(c) Durand (d) Fattal 26/35
User study
– Visual richness evaluating effectiveness of framework
• Comparing bioptic image pairs to dichoptic image pairs
− Dichoptic image pairs showing more preference
Table 2. User study of visual richness.
27/35
– Binocular symmetry
• Binocular tone mapping symmetric to left and right eyes
− Comparing statistics from two sets of evaluations
Table 3. User study of binocular symmetry. L/R means the left/right
eye sees the images generated by our framework.
28/35
– Predictability of BVCP
• Discomfort limit of binocular vision
− Based on classical psychophysical methodology
» Method of adjustment
Fig. 15. One test sequence for evaluating BVCP predictability.
29/35
• Statistics of 20 test sequences
(a) Drago (b) Mantiuk
(c) Durand (d) Fattal
Fig. 16. Statistics of the predictability of BVCP.
30/35
Incorporating stereopsis
– Feasible to extend framework to incorporate stereopsis
Fig. 17. A stereo LDR image pair with left and
right images tone-mapped differently.
31/35
Limitation
– Human-tolerable image pairs may be rejected
– Human vision not equally sensitive to all pixel
• Due to visual attention
− Introducing importance map based on image content
– Contrast threshold function
• In modeling failure of rivalry
Fig. 18. A case with small improvement of
overall visual richness. 32/35
– Other resulting images
Fig. 19. LDR pair of “BrightonPierOld”
using Drago’s operator.
Fig. 20. LDR pair of “GGmusicians”
using Durand’s operator.
33/35
– Other resulting images
Fig. 21. LDR pair of “WreathBuilding”
using Fattal’s operator.
Fig. 22. LDR pair of “Alhambra” using
Mantiuk’s operator.
34/35
Conclusion
Binocular tone mapping framework
– Generating binocular LDR image pair
• Presenting more visual richness
– Developing novel BVCP metric
• Predicting discomfort threshold
35/35
