Journal of Visual Communication and Image Representation
vol. 21, Issue. 5-6, 2010
Mehrdad Panahpour Tehrani, Akio Ishikawa, Shigeyuki Sakazawa and Atsushi Koike
Presented by Shu Ran
School of Electrical Engineering and Computer Science
Kyungpook National Univ.
Iterative Colour Correction of
Multicamera Systems Using
Corresponding Feature Points
Abstract
Color distortion in multicamera
– Proposing novel color correction method
• Advantage of proposed method
− Both working for dense and sparse multicamera
− Obtaining average color pattern among all cameras
• Starting from any camera on array sequentially
− Following certain path until reaching starting point and trigger iterations
• Color correction transformation based on energy minimisation
− Using dynamic programming
» By nonlinearly weighted Gaussian-based kernel density function
» From geometrically corresponding feature points
• Guarantee convergence of iteration procedure
− Without any visible color distortion
2/27
Introduction
3-D manner for human visual systems
– Changing our view position according interest
• Used as communication component
− FTV and 3D-TV
– Video sequences captured at same time from different viewpoints
• Dissimilarities between luminance and chrominance
− Appropriating color correction for compensating inconsistency
– Basic color correction in multicamera
• Adjusting parameters
− Self-calibrating each camera
» Using automatic gain and white balance adjustment
• Image processing
− Generally divided into two categories
3/27
– Using color pattern board
• Achieving average color intensity among viewpoints
− Useless for large areas and outdoor situations
– Popular by without color pattern board
• Performed after capturing and applied to indoor and outdoor
− Average color intensity cannot be achieved
• Classified into illumination based and feature-based approaches
− Easy to implement and basically linear transformation
» Without considering geometrical characteristics of multicamera
− Estimating nonlinear color transformation using geometrical characteristics
• Similar to one viewpoint as reference
− Far distances pairs well corrected
4/27
Proposed method
– Advantage of proposed method
• Performed for dense and sparse multicamera
• Achieving average color intensity among viewpoints
– Without color pattern board
• Feature based method
− Nonlinear transformation obtained from geometrical characteristics
– Assuming condition
• Lambertian condition and similarly manufactured cameras
− Similar color filters in multicamera
− Captured images obtained without RGB cross matrix calculation
– Aim goals
• Scalability and accuracy
• Average color intensities and adaptive
• Fully automatic
5/27
– By using iterative color correction
• Obtaining corresponding intensities for neighbour pair
− Reference and target cameras using modified SIFT
» Start correction from any location on camera array
• Suppress outliers from matched feature
− Collect more corresponding intensities
» From blurred intensities of suppressed corresponding locations
» From several frames of reference and target video
» With same procedure
• Using proposed nonlinearly weighted kernel density function
− Generating nonparametric statistical representation of collected correspondences
» For each color channel
− Nonlinear weighting approach
» Decreasing effect of corresponding intensities with large difference
» Causing error in correction transformation
6/27
Several previous approaches
– Two main processes within any color correction
• Showing list of corresponding intensities
• Generating transfer function for color correction
– Previous method of color pattern boards
• Generating linear transformation
− Based on linear least-square matching
» Simultaneously modifying RGB at pixels using 3x3 matrix
• Linear RGB-RGB transformation using 3x4 matrix
• Capturing whole hue and detecting corresponding intensities
− Useless for outdoor and wide-space coverage of multicamera
» Hard for providing color pattern board
Previous works
7/27
– Without color pattern board targeted specific application
• Histograms of two cameras obtaining corresponding intensities
− Generating LUT by using histograms of two views
− Generating linear transformation for YUV channels
» Insufficient for occlusion area
• Compensating illumination for increasing efficiency of coding
− Using macro-block-based illumination change
− Using average and variance in a block
− Adding offsets to compensate for blocks
– Using nonlinear approach without using color pattern board
• Content-adaptive color correction
− Using linear transformation in format of matrix
• Choosing one reference camera
− All target cameras have a color pattern in reference
8/27
Procedure of proposed method
– Choosing reference camera and nearest as target
• Reducing color pattern of target camera to reference
− Using proposed color correction algorithm
» Generating nonlinear transformation functions LUT
» By minimisation of correspondences energy function
» Through dynamic programming
Proposed color correction
9/27
Fig. 1. Flowchart of the proposed
colour correction method.
– Obtaining energy function as follows
10/27
Step1 For first frame of the pair, use SIFT to select feature points
locations and determine the corresponding feature points.
Step2 Suppress outliers and save the location of corresponding points
in set ‘‘Lf”.
Step3 Blurring with different orders (i.e., several Gaussian filters) are
applied to the pair. Then, corresponding intensities are collected
in set ‘‘P” from original and blurred pair in location of
corresponding points obtained in step 2.
Step4 For several further frames of the pair, using the same
approaches from step 1 to step 3, appended corresponding
feature points to set ‘‘P”.
Step5 A Gaussian-based kernel density function is generated using all
corresponding intensities.
Step6 Within the Kernel density function, the Gaussian function is
nonlinearly weighted using the distribution of corresponding
intensities in set ‘‘P”.
Step7 Finally, we generate an energy function for each colour channel
using the statistical model in step 6 and a regulation energy
function. Then, a lookup table for each colour channel is
generated for correction.
Algorithm of proposed method
– Step1: detection of corresponding feature points
• Using modified SIFT finding corresponding feature
• Modified Matching process
11/27
1 1 1 1= , , , , , , , , , , , , ,r r t t r r t t r r t t
f i i i i M M M ML x y x y x y x y x y x y (1)
where is the ith pair of corresponding locations in
target “t” and reference “r” camera. Symbol ‘‘ ” shows the
frame number where the corresponding points are
detected for the pair. The optimal value for ‘‘M = 300” is
estimated experimentally.
, , ,r r t t
i i i ix y x y
f
– Step 2: suppressing outliers
• Proposing new method for suppressing outliers
− Generating histogram of angles of lines connecting correspondence
− Calculating mean and standard deviation of histogram
− Choosing part of histogram
» Rest as outlier
12/27
1 1 1 1= , , , , , , , , , , , , ,f f f f
r r t t r r t t r r t t
f i i i i N N N NL x y x y x y x y x y x y
(2)
(a) (b) (c) (d)
Fig. 2. Suppression (a) detected by modified SIFT, (b) histogram
of lines’ angles in (a) that connect corresponding points before
suppression of outliers, (c) histogram after suppression of outliers,
and (d) remained corresponding points after suppression.
– Step 3: correspondences from blurred pairs
• Image containing high spatial resolution and noise
− Impairing color correction
• Generating several orders of blurred image pairs
− Applying Gaussian filter for G times
− Using all corresponding colors of filtered images at given time
− Blurring increasing total number of corresponding intensities
» Picking better corresponding in edge
13/27
0 1
1 1 1 1
= , , , , ,
= , , , , , , , , , , , , ,
i G
i f f f f
f f f f
f
f f r r f t t f r r f t t f r r f t t
i i i i i i i i i N N i N N
P P P P P
P P x y P x y P x y P x y P x y P x y
(3)
where “ ”is the set of corresponding intensities for a colour
channel that the pair is blurred with ‘‘ ”.
f
iP
i
(4)
• Procedure for three times of Gaussian filtering
14/27
Fig. 3. Gaussian filtering and picking up of corresponding
intensities in filtered images from the same location detected by
modified SIFT in the original images.
– Step 4: correspondences from several time instances
• Choosing more frames for increasing corresponding points
− Limit time range of frame
− Number of corresponding intensities is slightly changed for each time
» Total number of correspondences for each color channel
• Example of set P
15/27
1 1,...,1
,F
i ii S i Ni
P P r t
(5)
iN
11
1F
i Si
N G N
(6)
Table. 1. Example of set ‘‘P” for RGB channels.
– Step 5: generation of a Gaussian-based kernel density function
• Building statistical representation of corresponding intensities of P
− Using general nonparametric kernel density estimation
» Without any assumption about underlying distributions
− Estimate density at point
16/27
1
1 N
i
i
P x K x xN
(7)
where “ ”is a kernel function with bandwidth K
,n nr t
1
1, , ,
N
n n i J i J
i
P r t G r G tN
(8)
where the same kernel function is used with a suitable bandwidth
for each dimension J
– Step 6: nonlinear weighting of kernel density function
• Problem of corresponding intensities with large differences
− Deviate correction transformation
− Cause color distortion during iteration
• Correct target image with large intensity changes
− Adding weight coefficient during generation of statistical model
• Proposing nonlinearly calculating nonlinear weighting coefficient
− For each pair of corresponding intensities
• Nonlinearly weighted 2-D Gaussian kernel density function
17/27
1
1, , , ,
N
n n i i i J i J
i
J r t r t G r G tN
(9)
where the weighting coefficients are calculated as follows. ,i ir t
(10)
2
2, exp
2
i
i
i Q
i i
Q
xr t
where is ‘‘ ” is the number of corresponding intensities in
the histogram that are parallel to .
ix ,i ir t
ir t Q
• Produce of step 6
− Calculating histogram of set P
− Locally cut histogram along lines normal to line r = t
» Giving highest coefficient value to most emphasized corresponding
» Reducing effect of large intensity changes
18/27
Fig. 4. Procedure to calculate weighting coefficient of each
corresponding intensity for Gaussian-based weighted coefficient. For the
given set of corresponding intensities in a colour channel
– Step 7: calculation of lookup table and performing color correction
• Calculating nonlinear transformation as LUT for each color channel
• Example of LUT for RGB color channels
19/27
(11) i ir f t
(12) 0 1 2 255, , ,..., ,...,ir r r r r r
(13) 2550,1,2,..., ,..., 255it t t
Table. 2. Example of LUT for RGB colour channels.
• Defining energy function for calculating
− Global minimum as transfer function
− Consist of two parts
» Energy of corresponding intensities obtained from model
» Step-by-step regularization energy function
» Enforcing be along r=t line
− Energy value for
20/27
if t
(14) 1 2arg minrf E r E r
255
1 0,n nn
E r J r t
255 2
2 101n nn
E r t t
i ir f t
,i jr t
(15) 2
11,...,
, min , , 1i j i j i j i i kk j
e r t J r t e r t t t
where is the energy of corresponding intensities
obtained from the model and is the step-by-
step regularization energy..
,i je J r t2
1 1n nt t
– Iteration
• Proposing iteration based approach for color correction
− Architecture of iteration-based colour correction in general
» For different multicamera configurations
21/27
Fig. 5. Architecture of iteration-based colour correction.
• Step1: choosing a camera randomly as the reference camera
− Calculate the total amount of change ‘‘e1” for all colour intensities in LUT
• Step2: assigning target camera in last correction as reference camera
− Collecting e2 and add to e1
• Step3: decision on stopping iteration of colour correction
− Calculating average value of total change eave for iteration
» Less than a threshold to stop
Color correction of multiview video
– Color correction for all frames within proposed W1 window
• Using generated LUT
− Iteration LUT and window
22/27
(16) 255
1
0
1
3i
RGB i
e f t i
Experiments
Evaluating proposed method
– Using 5 kinds scene
– Using similarity of hue as measure
Adjustment of parameters
– Measure sensitivity of method in RGB channel to parameters
– Sensitivity to parameters
• Changing one parameter
23/27
(17)
1 2
1 2
0 1 2
1, 1
max ,
H
k
h k h ksimilarity h h
H h k h k
Fig. 6. Sensitivity of the proposed method to
parameters that decide the total number of
corresponding intensities (a)–(d), and
parameters during generation of LUT (e) and
(f).
• Sensitivity of proposed method to the different colour channels
24/27
Fig. 7. Comparison of the proposed colour correction in RGB, HSV, and YUV
colour domains applied to four multiview sequences.
• Resulting images of different colour channels
25/27
Fig. 8. Examples of the proposed colour correction in RGB, HSV, and YUV
colour domains applied to the ‘‘flamenco” sequence.
• Sensitivity of proposed method to the starting camera
26/27
Fig. 9. Performance of the proposed colour correction when starting camera
is changed.
– Estimation of parameters
• Value proportional to texture density of the given multiview images
− Measure texture density of the images
» Defining structure matrix
− Relation of with measured average of texture density
» For a pair of images for all five multiview sequences
27/27
(18) , ,yx
x y
p bp bT
r g g
x p b y p b
S x y x y
Fig. 10. Estimation curve of ‘‘k” (regulation coefficient during dynamic
programming) for optimal (i.e., no colour distortion and maximum similarity
value) colour correction vs. the average of texture density for a pair of images.
– Comparison and analysis
• Performance of proposed method against conventional method
− By applied to sparse multicamera system
• Compare performances of conventional and proposed architectures
and algorithms
28/27
Fig. 11. Comparison of the proposed method with the conventional method
• Compare performance of proposed colour correction with conventional
29/27
Fig. 12. Comparison of the proposed method with the conventional method
• Performance of proposed method in comparison with RTRT0100
30/27
Fig. 13. Examples of generated LUT using, (a) RTRT0100 [8,9]; and (b)
RTRT1111.
• Performance of proposed method in comparison with RTRT0100
− Showing on table
31/27
Table. 3. Example of LUT for RGB colour channels.
• Examples of generated model
32/27
Fig. 14. Examples of generated model.
– Resulting images
33/27
Fig. 15. Result of colour correction using the
proposed method ‘‘crowd” (iteration started
from the leftmost image).
Fig. 16. Result of colour correction using the
proposed method ‘‘flamenco” (iteration started
from the leftmost image).
– Another resulting images
34/27
Fig. 17. Result of colour correction using the
proposed method ‘‘object”(iteration started
from the leftmost image).
Fig. 18. Result of colour correction using the
proposed method ‘‘race” (iteration started from
the leftmost image).
– Resulting images
35/27
Fig. 19. Result of colour correction using the proposed
method ‘‘cheer”(iteration started from the leftmost image).
– Impact on stereo matching in a 3-D system
• Compare generated viewpoint between two stereo cameras
− with the actual image captured by an existing camera
» Using colour corrected and uncorrected sequences in different
disparities
36/27
Fig. 20. Comparison of block matching PSNR for free
viewpoint synthesis using colour-corrected
Discussion and conclusion
Proposed general-purpose method
– Iteration based color correction of mulitview camera
• Without using color pattern board
– Generating correction transformation
• Considering geometrical consistency of multicamera
– Using suppression and weighting processes
• Reducing wrong correspondence
37/27
