Color Image Processing
Dr. Edmund Lam
Department of Electrical and Electronic EngineeringThe University of Hong Kong
ELEC4245: Digital Image Proessing(Second Semester, 2015–16)
http://www.eee.hku.hk/˜elec4245
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 1 / 46
Color images
Color = RGB (red, green, blue)?
Yes, and no.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 2 / 46
Physics of color and human vision
1 Physics of color and human vision
2 Colorimetry: the science of color
3 Representing colors in different color spaces
4 Capturing colors with photodetectors
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 3 / 46
Physics of color and human vision
Physics of color
Light: electromagnetic waves with a range of frequencies (or,equivalently, wavelengths)
c = 3.0 × 108 m s−1; 400 nm ≤ λ ≤ 700 nm for visible light.There is color because our eyes interpret those frequencies
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 4 / 46
Physics of color and human vision
Physics of color
We interpret colors based on wavelengthsλ ≈ 400 nm =⇒ blueλ ≈ 550 nm =⇒ greenλ ≈ 700 nm =⇒ redλ < 400 nm: ultra-violet (UV)λ > 700 nm: infra-red (IR)UV and IR are gaining importance in imaging, e.g., in medicalimaging and remote sensing respectively
Electromagnetic waves with a range of frequencies at various strength hit oureyes (or photodetectors). How is color perceived?
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 5 / 46
Physics of color and human vision
Illumination
We can see things because there are illuminations, or light sources.Virtually all illuminations consist of light with a range of wavelengths.
We have to go to extraordinary length to create lasers that are of a“single” wavelength (more precisely, a very narrow bandwidthlight source).Each illumination is described with a curve showing the strengthof the electromagnetic radiations at different values of λ.Its normalization is called the spectral power distribution ofillumination as a function of wavelength.Mathematically, we denote it as I(λ)
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 6 / 46
Physics of color and human vision
Illumination
Examples of spectral power distribution:
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 7 / 46
Physics of color and human vision
Object
When the electromagnetic radiations from the illumination reach anobject, they are partially absorbed, and partially reflected ortransmitted.
the proportion of the reflection or transmission varies withwavelengthsit is irrespective of illuminationspectral reflectance or spectral transmittanceMathematically: R(λ) or T(λ)
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 8 / 46
Physics of color and human vision
Object
Examples of object reflectance:
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 9 / 46
Physics of color and human vision
Object
A “Macbeth Color Checker”
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 10 / 46
Physics of color and human vision
Color stimulus
What reaches our eye: reflectance of object upon illumination
S(λ) = I(λ)R(λ). (1)
Color stimulus influences the color perceptionImportant corollary: Color is not an inherent feature of an objectWe can make a “red” T-shirt looks “blue” with careful change ofillumination
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 11 / 46
Physics of color and human vision
Human Visual System
We often don’t need the entire color stimulus curveWe summarize in 3 numbersPossibility of ambiguity due to information lossEnough for color reproduction due to human visual system (HVS)
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 12 / 46
Physics of color and human vision
Photoreceptors
Our eyes have “sensors” called photoreceptors. There are two types:cones and rods.
Cones Rods
6 to 7 million 75 to 150 million
Located together at fovea Distributed all over the retina
Highly sensitive to color Not for color vision
Help resolve fine details Provide broad field of view
Photopic (bright-light) vision Scotopic (dim-light) vision
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 13 / 46
Physics of color and human vision
Photoreceptors
Cones are responsible for color vision.
3 types of cones:L-cones (long-wavelength peak sensitivity)M-cones (middle-wavelength peak sensitivity)S-cones (short-wavelength peak sensitivity)
They have different spatial sampling distributionsL- and M-cones have similar wavelength sensitivityS-cone is significantly differentS-cone is relatively rare in the retina, and widely spaced
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 14 / 46
Physics of color and human vision
Photoreceptors
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 15 / 46
Physics of color and human vision
Spectral sensitivities
We use x(λ), y(λ), and z(λ) to denote the response of the L-, M-, andS-cones respectively.
Color is formed by integration!
X =
∫ 700
400x(λ)I(λ)R(λ) dλ
Y =
∫ 700
400y(λ)I(λ)R(λ) dλ (2)
Z =
∫ 700
400z(λ)I(λ)R(λ) dλ
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 16 / 46
Physics of color and human vision
Spectral sensitivities
If discretized, we can compute a summation:
XYZ
=
x(λ1) . . . x(λn)y(λ1) . . . y(λn)z(λ1) . . . z(λn)
I(λ1)R(λ1)...
I(λn)R(λn)
(3)
The triplet (X,Y,Z) is called trichromatic responses. These three arethe “magic numbers” sent to our brain for color interpretation.
It is possible that two objects with different stimuli produce the same color.We call them metamers.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 17 / 46
Physics of color and human vision
Color formation
Our eyes make these interpretations, e.g.,Strong L-cone response =⇒ redStrong L-cones and M-cones responses =⇒ yellowMore generally, we represent the addition of colors withred/green/blue (RGB) as they correspond to the peaks of theresponses of L-, M-, and S-cones.
RGB are often referred to as primary colors.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 18 / 46
Physics of color and human vision
Describing colors
Other than quantifying with the RGB “color components”, we oftenhave different ways to describe colors:
luminance: the “energy” of the light, and chrominance: the “color”hue: the “dominant wavelength” of the light, and saturation: the“purity” of the color (i.e., how much it is mixed with white)
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 19 / 46
Colorimetry: the science of color
1 Physics of color and human vision
2 Colorimetry: the science of color
3 Representing colors in different color spaces
4 Capturing colors with photodetectors
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 20 / 46
Colorimetry: the science of color
Mixing colors
We can mix the primary colors to form other colors.
Many displays make use of RGB “light sources” to simulate differentcolors, e.g., your television / monitor (experiment: put a magnifyingglass in front of your LCD screen). This is called additive color system.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 21 / 46
Colorimetry: the science of color
Mixing colors
Each color has its opposite, called color complement:
0◦30◦
60◦
90◦
120◦
150◦180◦
210◦
240◦
270◦
300◦
330◦
yellow
greenblue
magenta
red
cyanE. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 22 / 46
Colorimetry: the science of color
Mixing colors
Red’s complement is Cyan (C)Green’s complement is Magenta (M)Blue’s complement is Yellow (Y)
CMY are often referred to as secondary colors.
We often use them for subtractive color system, e.g., mixing inks, whichblock the reflection of that color. So colors subtract: if all colors arepresent, you perceive black.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 23 / 46
Colorimetry: the science of color
Mixing colors
We can assign different weights to the “primary colors” to generatedifferent colors. Let’s say we have X amount of “red”, Y amount of“green”, Z amount of “blue”: (X,Y,Z) are called tri-stimulus values.
CIE in 1931 standardizes the following: (CIE stands for “CommissionInternationale de l’Eclairage” — the International Commission onIllumination)
red = 700 nmgreen = 546.1 nmblue = 435.8 nm
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 24 / 46
Colorimetry: the science of color
Mixing colors
We can normalize the X, Y, Z values by dividing the sum of the weights
x =X
X + Y + Z
y =Y
X + Y + Z(4)
z =Z
X + Y + Z
Note that x + y + z = 1. So, (x, y) determines the color!How can we reduce 3 parameters to 2? We lose the normalization,i.e., the brightness (luminance) of the color.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 25 / 46
Colorimetry: the science of color
Mixing colors
Use (x, y) as the coordinates, and each point determines a color: theCIE chromaticity diagram. The boundaries are the “pure colors” atvarious wavelengths, known as the spectral locus.
y
x
z = 1 − (x + y)
380 nm
480 nm
490 nm
500 nm
510 nm
520 nm
540 nm
570 nm
600 nm
780 nmwhite
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 26 / 46
Colorimetry: the science of color
Mixing colors
The chromaticity diagram is useful to visualize mixing colors: supposewe have three light sources, “r”, “g”, and “b” respectively. The trianglerepresents all possible colors from mixing them: the color gamut
y
x
r
g
b
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 27 / 46
Colorimetry: the science of color
Mixing colors
Insight from the chromaticity diagram and color gamut:Bigger gamut means more color can be reproduced from the lightsourcesNo triangle can cover the entire chromaticity diagram: i.e., for anyfixed set of three light sources, there will always be colors thatcannot be reproduced. These colors are out of gamut.Having more light sources can generally increase the gamut, butstill cannot cover the entire space.
Color reproduction is a hard problem! You take a picture of an object,and you want the color to show up “correctly” on your computermonitor, your mobile phone, your photographic print, etc.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 28 / 46
Representing colors in different color spaces
1 Physics of color and human vision
2 Colorimetry: the science of color
3 Representing colors in different color spaces
4 Capturing colors with photodetectors
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 29 / 46
Representing colors in different color spaces
RGB color space
To represent a color image, each pixel should have three numbers. Thesimplest way is to let them represent red, green, and blue; this is calledthe RGB color space. We can visualize with a cube:
R
G
B
red
green
blue
yellow
cyan
magenta
K
W
K: black
W: whitegray
Each color is a linear combination from no (R,G,B) to full (R,G,B)E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 30 / 46
Representing colors in different color spaces
CMY color space
Remember that CMY are complements of RGB. Assume all values gofrom 0 to 1.
CMY
=
111
−RGB
. (5)
CMY are usually used in printers.In theory, C = M = Y = 1 gives black. For inkjet printers, however,mixing all inks usually give a rather muddy color.Black is used very frequently in printing =⇒most inkjet printersactually use at least 4 colors, i.e. CMYK. (K for black)
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 31 / 46
Representing colors in different color spaces
RGB to CMYK
Application: To print your digital photograph, a process known ascolor space conversion has to be done.
Often, this is RGB to CMYK.A non-linear mapping (one-to-many mapping theoretically).Has to be performed on every pixel =⇒ a time-consuming process!Often implemented in a look-up table (LUT).Often proprietary and specifically tuned for the printer engine.
A further complication: inks are high-density binary (ink or no ink),while digital images are usually lower-density 8-bit-per-color-channel.Printing involves a process known as halftoning.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 32 / 46
Representing colors in different color spaces
NTSC and PAL
To give you a flavor of the use of different color spaces, consider thoseused in our TVs. YIQ is used for NTSC. YUV is used for PAL.
YIQ
=
0.299 0.587 0.1140.596 −0.275 −0.3210.212 −0.523 0.311
RGB
(6)
YUV
=
0.299 0.587 0.114−0.147 −0.289 0.4360.615 −0.516 −0.100
RGB
(7)
Note that
I = −U sin 33◦ + V cos 33◦
Q = U cos 33◦ + V sin 33◦
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 33 / 46
Representing colors in different color spaces
NTSC and PAL
How to interpret YIQ and YUV color spaces?Y is the (positively) weighted sum of RGB. It describes theluminance of the signal. Technically it’s also called the luma.Weights are different for RGB because of our eyes: e.g. we aremost responsive to green.In the old days, we have monochrome TV =⇒ it only decodes theY signal!
How about I, Q, U, and V? {I,Q} or {U,V} describe the hue (“purecolor”) and saturation (“degree of pure color diluted by white”)collectively. Remember we need three components to describe a color.
They are designed such that we can limit their spatial bandwidthwithout noticeable image degradation.YUV was proposed first; YIQ was an improvement because it wasfound that it could reduce the bandwidth to a greater degree.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 34 / 46
Representing colors in different color spaces
HSI color space
In applications that deal with human more directly, shouldn’t we use amore direct description of color? Yes, we have HSI!
H = hueS = saturationI = intensity, a.k.a. “energy”
As with before, there should be a one-to-one mapping between RGBand HSI.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 35 / 46
Representing colors in different color spaces
HSI color space
white
black
IS HI = 0.5
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 36 / 46
Representing colors in different color spaces
RGB to HSI
H =
{θ if B ≤ G360 − θ otherwise
S = 1 − 3(R + G + B)
[min(R,G,B)] (8)
I =13
(R + G + B)
where
θ = cos−1
12 [(R − G) + (R − B)]
√(R − G)2 + (R − B)(G − B)
and R ∈ [0, 1], G ∈ [0, 1], and B ∈ [0, 1].
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 37 / 46
Representing colors in different color spaces
HSI to RGBFor 0 ≤ H < 120◦, use H = H.
B = I(1 − S)
R = I[1 +
S cos Hcos(60◦ − H)
](9)
G = 1 − (R + B).
For 120◦ ≤ H < 240◦, compute H = H − 120◦.
R = I(1 − S)
G = I[1 +
S cos Hcos(60◦ − H)
](10)
B = 1 − (R + G).
For 240◦ ≤ H < 360◦, compute H = H − 240◦.
G = I(1 − S)
B = I[1 +
S cos Hcos(60◦ − H)
](11)
R = 1 − (G + B).
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 38 / 46
Representing colors in different color spaces
Color processing
How to do engineering applications that involve color?
First decision: what color space / modelSeldom a good idea to operate on RGBUsually much better on a luminance-chrominance model
Second decision: operate on 3-planes together or separately?Separately: would results from the 3 “layers” in the color modelmis-align? e.g. edge detection finds edges at different locations?Together: how? May not be easy.
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 39 / 46
Representing colors in different color spaces
Color processing
Example: median filter
3 separate planes:Color may shift drastically! Let’s say we work on RGB. If R and Gdon’t change much but B changes a lot, then the color suddenlybecomes much more yellow!
Together:Each point is a vector of 3 intensity valuesWe don’t know how to order vectors!
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 40 / 46
Capturing colors with photodetectors
1 Physics of color and human vision
2 Colorimetry: the science of color
3 Representing colors in different color spaces
4 Capturing colors with photodetectors
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 41 / 46
Capturing colors with photodetectors
Sensor technology
Two dominant technology: CCD and CMOS
Charged-couple device (CCD):Most common; dominant technology; lowest costInvented at Bell Labs in 1970. Nobel prize in 2009.Optimized photodetectors→ good SNR; low nonuniformityDifficult to integrate other functions on the sensor; high powerconsumption; limited frame rate due to serial readout
Complementary metal-oxide semiconductor (CMOS):Integrated fabrication; cheapLow power consumption (∼ 1/10 of CCD)Very high frame rate can be achievedIntegrate processing with sensingNon-uniformity due to multiple levels of amplification (pixel,column, chip); fixed pattern noise and dark current
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 42 / 46
Capturing colors with photodetectors
Capturing in color
Both CCD and CMOS respond to the “amount of light” (number ofphotons), within the range of visible spectrum. In other words, they donot differentiate color.
To capture color images, each pixel needs to output at least threevalues (e.g. Red, Green, Blue)We can then do color space conversion, and other color processingA few possible architectures
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 43 / 46
Capturing colors with photodetectors
Dichroic mirror
R
GB
R
G
B
Red sensor
Green sensor
Blue sensor
Three separate sensorsHigh spatial resolution; little optical loss of the signal; costly
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 44 / 46
Capturing colors with photodetectors
Sequential imaging
R
G
B
R
G
B
Red sensor
Green sensor
Blue sensor
Time-varying color filterThree exposures; possibility of mis-alignment
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 45 / 46
Capturing colors with photodetectors
Color filter array
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
G
R
B
G
R
G
B
Red sensor
Green sensor
Blue sensor
Capture 1/3 information; interpolate with demosaicingLow cost; most common in practice
E. Lam (The University of Hong Kong) ELEC4245 Jan–Apr, 2016 46 / 46