49
Automatic Color Automatic Color Gamut Calibration Gamut Calibration Cristobal Alvarez-Russell Cristobal Alvarez-Russell Michael Novitzky Michael Novitzky Phillip Marks Phillip Marks
Automatic Color Automatic Color Gamut CalibrationGamut Calibration
Cristobal Alvarez-RussellCristobal Alvarez-RussellMichael NovitzkyMichael Novitzky
Phillip MarksPhillip Marks
InspirationInspirationG. Klein and D. Murray, G. Klein and D. Murray, Compositing for Compositing for Small CamerasSmall Cameras, ISMAR'08, ISMAR'08
MotivationMotivationCalibrate and compensate for:Calibrate and compensate for:
Color distortions of a small video cameraColor distortions of a small video cameraLighting conditions of environmentLighting conditions of environment
Purpose:Purpose:Augmented RealityAugmented Reality
Matching the color gamut of virtual objects Matching the color gamut of virtual objects to video camera imageto video camera image
RoboticsRoboticsCalibrating a video camera for particle-Calibrating a video camera for particle-filter-based object tracking (i.e. orange filter-based object tracking (i.e. orange ball in robot soccer)ball in robot soccer)
ApproachApproach
GretagMacbeth ColorChartGretagMacbeth ColorChartDiffuse materialDiffuse material
Color samples under daylightColor samples under daylight
RGB values are knownRGB values are known
Approach (cont.)Approach (cont.)11 Start with picture of a scene with the Start with picture of a scene with the
chartchart
22 Locate the squares of the chart in the Locate the squares of the chart in the imageimage
33 Unproject and crop the chartUnproject and crop the chart
44 Sample the colors in the chartSample the colors in the chart
55 Adjust the color of the entire image Adjust the color of the entire image (and subsequent ones)(and subsequent ones)
Locating the Locating the Chart Chart
Failed AttemptsFailed AttemptsSwain’s Histogram Back-projectionSwain’s Histogram Back-projection
Color constancy a big problemColor constancy a big problemTried color constancy approximationsTried color constancy approximations
Not good for color chartNot good for color chartToo many histogram matches -> false Too many histogram matches -> false positivespositives
Only returned a point within the squareOnly returned a point within the squareWe hoped it would be an estimation of the We hoped it would be an estimation of the center of the chartcenter of the chart
No information useful for unprojectionNo information useful for unprojection
Locating the Locating the Chart (cont.)Chart (cont.)
Original imageOriginal image
Locating the Locating the Chart (cont.)Chart (cont.)
Color constancyColor constancyColor normalizationColor normalization
Locating the Locating the Chart (cont.)Chart (cont.)
False positivesFalse positivesRatios high because of wide chart Ratios high because of wide chart histogramhistogram
Locating the Locating the Chart (cont.)Chart (cont.)
Result not useful for feature extractionResult not useful for feature extractionNot a good estimate of the center of the Not a good estimate of the center of the chartchart
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Current approachCurrent approachFirst step: User interfaceFirst step: User interface
User clicks and labels squaresUser clicks and labels squares
Flood fillFlood fillUses histogramUses histogram
Create screen-aligned bounding boxCreate screen-aligned bounding box
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Second step: Connected componentsSecond step: Connected componentsSweep through the imageSweep through the image
Label neighboring pixels that are Label neighboring pixels that are activatedactivated
Choose the connected component with Choose the connected component with the highest votethe highest vote
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Third step: Recognize regionsThird step: Recognize regionsWe need to find the corners of the region We need to find the corners of the region within the bounding boxwithin the bounding box
First attempt: Draw lines from bounding box First attempt: Draw lines from bounding box corners and vote on likelihood of region edgecorners and vote on likelihood of region edge
Failed!Failed!
Second attempt: Look for region corner Second attempt: Look for region corner iteratively from bounding box corneriteratively from bounding box corner
Success!Success!
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Locating the Locating the Chart (cont.)Chart (cont.)
Unprojecting the Unprojecting the chartchart
Start with corners of some color regionsStart with corners of some color regions
Construct a matrix A composed of Construct a matrix A composed of image and world point correspondencesimage and world point correspondences
Compute homography matrix from null Compute homography matrix from null space of Aspace of A
SVD to compute itSVD to compute it
Use inverse homography to unproject Use inverse homography to unproject each pixeleach pixel
Unprojecting the Unprojecting the chart (cont.)chart (cont.)
Problems:Problems:OpenCV matrices are not good for OpenCV matrices are not good for numerical methodsnumerical methods
Switched to GSLSwitched to GSL
Noise in region corner positionsNoise in region corner positionsRemove smallest eigenvalue of singular Remove smallest eigenvalue of singular matrixmatrix
Squares in the middle of the chart betterSquares in the middle of the chart better
Unprojecting the Unprojecting the chart (cont.)chart (cont.)
Unprojecting the Unprojecting the chart (cont.)chart (cont.)
Sampling the Sampling the chartchart
We sample at square centersWe sample at square centersSquares centers estimated by Squares centers estimated by predefined, specific ratios of the chartpredefined, specific ratios of the chart
We assume the homography and the We assume the homography and the unprojection are good enoughunprojection are good enough
Stochastic samplingStochastic samplingWe average several samples to reduce We average several samples to reduce noise influencenoise influence
Sampling the Sampling the chart (cont.)chart (cont.)
Sampling the Sampling the chart (cont.)chart (cont.)
Sampling the Sampling the chart (cont.)chart (cont.)
Adjusting the Adjusting the color gamutcolor gamut
Step 1: Adjust white balance of the Step 1: Adjust white balance of the samplessamples
Simple linear scaleSimple linear scale
Using White 9.5 and Black 32 from color Using White 9.5 and Black 32 from color chartchart
Both in chart in image and known RGB Both in chart in image and known RGB valuesvalues
Adjusting the Adjusting the color gamut color gamut
(cont.)(cont.)
Adjusting the Adjusting the color gamut color gamut
(cont.)(cont.)
Adjusting the Adjusting the color gamut color gamut
(cont.)(cont.)Step 2: Adjust chromaticityStep 2: Adjust chromaticityUse color samples as a distributionUse color samples as a distribution
Linear scale of every pixel color according to Linear scale of every pixel color according to mean and standard deviation of distributionmean and standard deviation of distributionColor samples from chart do not map to Color samples from chart do not map to themselvesthemselves
Approach 1: Marginal DistributionApproach 1: Marginal DistributionThree 1D distributions (one per channel)Three 1D distributions (one per channel)Treat channels independently from each otherTreat channels independently from each other
Approach 2: Joint DistributionApproach 2: Joint DistributionTreat colors as 3D points in RGB cubeTreat colors as 3D points in RGB cubeStandard deviation is a 3D distance from the Standard deviation is a 3D distance from the mean colormean color
Adjusting the Adjusting the color gamut color gamut
(cont.)(cont.)
Adjusting the Adjusting the color gamut color gamut
(cont.)(cont.)
Adjusting the Adjusting the color gamut color gamut
(cont.)(cont.)
Future WorkFuture WorkLocating the color chartLocating the color chart
Use SIFT-like descriptors with point matching Use SIFT-like descriptors with point matching according to the color chart structureaccording to the color chart structureUse grid-pattern algorithms like the ones Use grid-pattern algorithms like the ones used in fiducial-based tracking (i.e. used in fiducial-based tracking (i.e. ARToolkit)ARToolkit)
Chart unprojectionChart unprojectionTry iterative homography estimationTry iterative homography estimation
Color gamut adjustment:Color gamut adjustment:Interpolate colors using a tetrahedral meshInterpolate colors using a tetrahedral meshTry using color spaces that separate Try using color spaces that separate chromaticity from intensity (HSV, YUV, etc.)chromaticity from intensity (HSV, YUV, etc.)
The end!The end!
Another exampleAnother example
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
Another example Another example (cont.)(cont.)
