Binocular tone mapping

transcript

ACM Transactions on Graphics

vol. 31, No. 4, 2012

Xuan Yang, Linling Zhang, Tien-Tsin Wong, Pheng-Ann Heng

Presented by Ran Shu

School of Electrical Engineering and Computer Science

Kyungpook National Univ.

Binocular tone mapping

Abstract

Binocular single vision phenomenon

– Fusing two displaced images

• With difference in detail and contrast and luminance

– Proposed binocular tone mapping framework

• Generating binocular LDR image pair

− Preserving more human-perceivable visual content

− Left for tone-mapped LDR image

» Without loss of generality

− Right for proposed optimally synthesize image

• LDR image pair presenting more human-perceivable visual richness

− Without triggering visual discomfort

» Using novel binocular viewing comfort predictor

» Based on finding in vision science

– Binocularly tone-mapped image pair

Fig. 1. LDR image pairs. Left shows global contrast, right shows

more local details

Introduction

Two binocular vision phenomenon

– Stereopsis for existing binocular display

• 3D movies

– Binocular single vision

• Fusing images from two eyes as single percept

− Even two images with difference

» Complex non-linear neurophysiological process

Proposed method

– Utilizing existing LDR binocular display

• Simultaneously presenting contrast and details in HDR images

− By proposed binocular tone mapping framework

– Given an HDR image and tone-mapped LDR image

• Generated by existing tone mapping techniques

• Optimally synthesizing counterpart LDR image

− Using proposed framework BVCP

− Presenting more human-perceivable visual content

Proposed binocular viewing comfort predictor

– Guiding binocular tone mapping

• Based on findings in vision science and experimental results

− Avoiding visual discomfort

− Making more difference in two views

Binocular single vision

– Combing two different images from two eyes into single vision

• Occurring only in small volume of retinal area

− Around fixating area

− Non-linear combination of luminance and contrast and color

» Regarded as combination of binocular fusion and suppression

– Binocular fusion

• Process of superimposing

• Combining similar content from two views into one percept

− Under two similar or same image

– Binocular suppression

• Under submissive view

– Binocular rivalry

Related work

– Binocular fusion and suppression and rivalry

Fig. 2. Fusion, suppression and rivalry.

– Assessment for binocular viewing comfort

• Existing metrics considering visible difference

− Between two images looking with both eyes

» Without considering binocular vision

• Designing brand new metric

− Binocular viewing comfort predictor

» Based on theories and experimental results

Tone mapping

– Using previous tone mapping method

• Global operators

− Adaptive logarithmic mapping

• Local operators

− Bilateral filtering approach

− Gradient domain HDR compression

− Perceptual based contrast processing

Overview

Overview of binocular tone mapping

– Input HDR image and left tone-mapping LDR image

– Generating optimal right LDR image

• Using same tone-mapping operator as left

• Objective function composed of two metrics

− Visible difference predictor

− Binocular viewing comfort predictor

Fig. 3. System overview.

Fusional area

– Based on visual agreement of neighborhoods

• Panum’s fusional area

• Occupying constant solid angle subtended at eye

• Considering whole area for fusion stability

− In terms of contour and contrast and luminance and color difference

Fig. 4. Fusional area.

– Radius of fusional area

– Approximating fusional area by rectangle

• Go over all pixel pairs from LDR image pair

− Considering corresponding fusion areas

» For measuring fusion stability

tanfr d (1)

where is the maximal retinal disparity, which is around 60 to 70 arcmin

Contour fusion

– Contour difference more important than contrast or color difference

– Corresponding contour no need to exactly same

• Motor fusion

− Superimposing corresponding points or similar contour

» By movement of our eyes

• Sensory fusion

− Combing two views into one unified percept

Fig. 6. Motor fusion and sensory fusion. Fig. 5. Contour dominance.

– Contour has different definitions in different domains

• Applying scale representation to fusional area

− Constructing pyramid

» Fourier transform fusional area to frequency domain

» Applying pyramid of LPF in frequency domain

» Mesa filter as LPF

» Inversely Fourier transforming each low-passed frequency images

» Obtaining pyramid of low-passed fusional areas

Fig. 7. Mesa pyramid.

– Measuring color difference between adjacent pixels

• Defined as 2-norm distance of colors in LAB color space

• Color difference between a pair of pixels

1 2 1 2, ,k k k

CL p p E L p L p (2)

where is color difference of fusional area,

is color difference of two pixel.

Fig. 9. Contour matching.

• Looking up decision table

− With JND and obvious color difference

− Considering count of fusible pixels in fusional area

− Contour fusion state at higher levels override lower

− Final fusion state of two fusional areas

Table 1. Decision table for contour fusion where J=JND, O=OCD

0, , 0

1, , 0

cf p p

if S p p

B elseif S p p

otherwise

1 0k k k

cf cf cfB B if B (4)

cf cfB B (5)

Contour and regional contrasts

– Two types of contrast influencing binocular single vision

• Contour contrast

− Coexisting with contour if detected by human eye

− Matched contour pair generally helping fusion

» Except obviously inversed contrast

• Regional contrast

− Contrast between two regions by adopting restrictive constraint

1 2 1 2 1 2

1 2 1 2

, 1 , ,

k k k k k

k k k k

S p p if C L p L p OCD and C R p R p OCD

and C L p L p C R p R p

where computes the lightness difference between

the pixel pair

1 2,C c c

1 2,c c

if E L p R p DCDrB

otherwise

where are two corresponding pixels located at

position p in L and R, respectively.

,L p R p

Failure of rivalry

– Contrast below certain threshold

• Stable single percept forming regardless of contour fusion

− Happen in low-contrast regions

Fig. 8. Failure of Rivalry.

– Measuring contrast in log space

• As log percentage contrast between pixel pair

• Correlation between luminance and frequency and contrast

1 2 10

, logY Y

P c cY Y

where are the normalized luma of ,

maps the normalized luma Y in [0,1] to the physical

measurement in the unit of

1 2,Y Y 1 2,c c

2/cd m

(a) (b)

Fig. 10. Luminance and freq. vs. contrast. 20/35

• Obtaining two continuous curves

• Function of both luminance and spatial frequency

10 10min 2,1.999 0.362log 0.026logT l l l (9)

10min 2, log 3.557 1.334 1.881 0.108T f f f f (10)

where is average luminance in ,

is spatial frequency measured in cycles per degree,

represents log percentage contrast threshold.

2/cd m

10 10 10, min 2, log 3.557 1.334 1.881 0.108 0.514 0.362log 0.026logT l f f f f l l (11)

where is user parameter and is set to a value in [-0.15,0.15]

Fig. 11. The constructed log percentage contrast threshold.

– Contrast threshold for left fusional area

• Obtained by feeding parameters

• Take second revision to fusion state variable

L L kT T l f (12)

1 2 1 2 1 2 1 2, 0 , 0 , ,k k k k k k k kS p p if S p p and P L p L p T and P R p R p T (13)

The overall fusion predictor

– Predict ultimate viewing comfort

• For arbitrary pair of fusional areas

B otherwise

Fig. 12. Visualization of BVCP assessment.

Optimization

Adopt visible difference predictor

– Detecting visible difference

• With VDP and proposed BVCP

− Maximizing E without Violating BVCP

E H V i j (15)

where is a user-defined probability threshold and generally set as 75%,

is the Heaviside step function,

is the total number of pixels in the images.

– Visualization of binocular visual of four tone mapping operators

(a) Drago (b) Mantiuk

(c) Durand (d) Fattal

Fig. 13. Visualization of binocular visual differences of four tone

mapping operators for the HDR example in Fig. 20.

Results and Discussions

Framework adopted tone mapping operators

– Supporting four tone mapping operators

• Experimented with four operators

Fig. 14. Optimal LDR

image pairs generated by

our framework using four

tone mapping operators.

(c) Durand (d) Fattal 26/35

User study

– Visual richness evaluating effectiveness of framework

• Comparing bioptic image pairs to dichoptic image pairs

− Dichoptic image pairs showing more preference

Table 2. User study of visual richness.

– Binocular symmetry

• Binocular tone mapping symmetric to left and right eyes

− Comparing statistics from two sets of evaluations

Table 3. User study of binocular symmetry. L/R means the left/right

eye sees the images generated by our framework.

– Predictability of BVCP

• Discomfort limit of binocular vision

− Based on classical psychophysical methodology

» Method of adjustment

Fig. 15. One test sequence for evaluating BVCP predictability.

• Statistics of 20 test sequences

(c) Durand (d) Fattal

Fig. 16. Statistics of the predictability of BVCP.

Incorporating stereopsis

– Feasible to extend framework to incorporate stereopsis

Fig. 17. A stereo LDR image pair with left and

right images tone-mapped differently.

Limitation

– Human-tolerable image pairs may be rejected

– Human vision not equally sensitive to all pixel

• Due to visual attention

− Introducing importance map based on image content

– Contrast threshold function

• In modeling failure of rivalry

Fig. 18. A case with small improvement of

overall visual richness. 32/35

– Other resulting images

Fig. 19. LDR pair of “BrightonPierOld”

using Drago’s operator.

Fig. 20. LDR pair of “GGmusicians”

using Durand’s operator.

– Other resulting images

Fig. 21. LDR pair of “WreathBuilding”

using Fattal’s operator.

Fig. 22. LDR pair of “Alhambra” using

Mantiuk’s operator.

Conclusion

Binocular tone mapping framework

– Generating binocular LDR image pair

• Presenting more visual richness

– Developing novel BVCP metric

• Predicting discomfort threshold

Binocular tone mapping

Documents