CSE 803 Fall 2017 1

Color

n  Used heavily in human vision n  Color is a pixel property,

making some recognition problems easy

n  Visible spectrum for humans is 400nm (blue) to 700 nm (red)

n  Machines can “see” much more; ex. X-rays, infrared, radio waves

CSE 803 Fall 2017 2

Imaging Process (review)

CSE 803 Fall 2017 3

Factors that Affect Perception •  Light: the spectrum of energy that illuminates the object surface •  Reflectance: ratio of reflected light to incoming light

•  Specularity: highly specular (shiny) vs. matte surface

•  Distance: distance to the light source

•  Angle: angle between surface normal and light source * Sensitivity how sensitive is the sensor

CSE 803 Fall 2017 4

Some physics of color

n  White light is composed of all visible frequencies (400-700) n  Ultraviolet and X-rays are of much smaller wavelength

n  Infrared and radio waves are of much longer wavelength

CSE 803 Fall 2017 5

Coding methods for humans

•  RGB is an additive system (add colors to black) used for displays

•  CMY[K] is a subtractive system for printing

•  HSV is a good perceptual space for art, psychology, and recognition

•  YIQ used for TV is good for compression

CSE 803 Fall 2017 6

Comparing color codes

CSE 803 Fall 2017 7

RGB color cube

•  R, G, B values normalized to (0, 1) interval

•  human perceives gray for triples on the diagonal

•  “Pure colors” on corners

CSE 803 Fall 2017 8

Color palette and normalized RGB

CSE 803 Fall 2017 9

Color hexagon for HSI (HSV) Color is coded relative to the diagonal of the color cube. Hue is encoded as an angle, saturation is the relative distance from the diagonal, and intensity is height.

intensity

saturation hue

CSE 803 Fall 2017 10

Editing saturation of colors

(Left) Image of food originating from a digital camera;

(center) saturation value of each pixel decreased 20%;

(right) saturation value of each pixel increased 40%.

CSE 803 Fall 2017 11

Properties of HSI (HSV) n  Separates out intensity I from the coding n  Two values (H & S) encode chromaticity n  Convenient for designing colors

n  Hue H is defined by an angle n  Saturation S models the purity of the color S=1 for a completely pure or saturated color S=0 for a shade of “gray”

CSE 803 Fall 2017 12

YIQ and YUV for TV signals n  Have better compression properties n  Luminance Y encoded using more bits than

chrominance values I and Q; humans more sensitive to Y than I,Q

n  NTSC TV uses luminance Y; chrominance values I and Q

n  Luminance used by black/white TVs n  All 3 values used by color TVs n  YUV encoding used in some digital video and

JPEG and MPEG compression

CSE 803 Fall 2017 13

Conversion from RGB to YIQ

We often use this for color to gray-tone conversion.

CSE 803 Fall 2017 14

Colors can be used for image segmentation into regions

n  Can cluster on color values and pixel locations n  Can use connected components and an

approximate color criteria to find regions

n  Can train an algorithm to look for certain colored regions – for example, skin color

CSE 803 Fall 2017 15

Color Clustering by K-means Algorithm

Form K-means clusters from a set of n-dimensional vectors 1. Set ic (iteration count) to 1 2. Choose randomly a set of K means m1(1), …, mK(1). 3. For each vector xi, compute D(xi,mk(ic)), k=1,…K and assign xi to the cluster Cj with nearest mean. 4. Increment ic by 1, update the means to get m1(ic),…,mK(ic). 5. Repeat steps 3 and 4 until Ck(ic) = Ck(ic+1) for all k.

CSE 803 Fall 2017 16

K-means Clustering Example

Original RGB Image Color Clusters by K-Means

CSE 803 Fall 2017 17

Extracting “white regions”

n  Program learns white from training set of sample pixels.

n  Aggregate similar neighbors to form regions.

n  Components might be classified as characters.

n  (Work contributed by David Moore.)

(Left) input RGB image

(Right) output is a labeled image.

CSE 803 Fall 2017 18

Skin color in RGB space

Purple region shows skin color samples from several people. Blue and yellow regions show skin in shadow or behind a beard.

CSE 803 Fall 2017 19

Finding a face in video frame

n  (left) input video frame n  (center) pixels classified according to RGB space n  (right) largest connected component with aspect

similar to a face (all work contributed by Vera Bakic)

CSE 803 Fall 2017 20

Color histograms can represent an image

n  Histogram is fast and easy to compute.

n  Size can easily be normalized so that different image histograms can be compared.

n  Can match color histograms for database query or classification.

CSE 803 Fall 2017 21

Histograms of two color images

CSE 803 Fall 2017 22

Retrieval from image database

Top left image is query image. The others are retrieved by having similar color histogram (See Ch 8).

CSE 803 Fall 2017 23

How to make a color histogram n  Make 3 histograms and concatenate them

n  Create a single pseudo color between 0 and 255 by using 3 bits of R, 3 bits of G and 2 bits of B.

n  Can normalize histogram to hold frequencies

so that bins total 1.0

CSE 803 Fall 2017 24

Apples versus oranges

Separate HSI histograms for apples (left) and oranges (right) used by IBM’s VeggieVision for recognizing produce at the grocery store checkout station (see Ch 16).

CSE 803 Fall 2017 25

Swain and Ballard’s Histogram Matching for Color Object Recognition

Opponent Encoding: Histograms: 8 x 16 x 16 = 2048 bins Intersection of image histogram and model histogram: Match score is the normalized intersection:

•  wb = R + G + B •  rg = R - G •  by = 2B - R - G

intersection(h(I),h(M)) = ∑ min{h(I)[j],h(M)[j]}

match(h(I),h(M)) = intersection(h(I),h(M)) / ∑ h(M)[j]

j=1

numbins

j=1

numbins

Hair color matching

n  21-dim BoW features n  Distance Metric Learning

Σ min (f1-f2) A (f1-f2)T

A

Hist. from image 1 Hist. from image 2

Joseph Roth, Xiaoming Liu, "On Hair Recognition in the Wild by Machine,” AAAI 2014

Histogram of Oriented Gradient

CSE 803 Fall 2017 27

Input: Training data {xi; i 2 [1, K]} and theircorresponding class labels {yi; i 2 [1, K]}.

Output: A strong classifier F (x).1. Initialize weights wi = 1/K, and F (x) = 0.2. for m = 1, 2, ..., M do

(a) Fit the regression function fm(x) by weighted leastsquare of yi to xi with weights wi

fm(x) = argminf2F ≤(f) =PK

i=1 wi(f(xi)° yi)2.

(b) Update F (x) = F (x) + fm(x).(c) Update the weights by wi = wie

°yifm(xi) andnormalize the weights such that

PKi=1 wi = 1.

end3. Output the classifier sign[F (x)] = sign[

PMm=1 fm(x)].

Algorithm 1: The GentleBoost algorithm.

10 20 300

0.020.040.060.08(a) (b) (c)

(d)

Figure 2. (a) The parametrization of a cell; (b) The gradient map;(c) The HOG of a block; (d) The HOG features of positive andnegative samples.

incorporate spatial information into HOG, a 2£2 cell arrayis used to form a block. For each cell, the b-bin histogramof the gradient magnitude at each orientation is computed.The concatenation of the HOG for 4 cells within one blockforms a 4b-dimensional vector, as shown in Figure 2(c). Thelarge hypothesis space F , where (x0, y0, x1, y1) resides,is obtained via an exhaustive construction within the tem-plate coordinate system. For example, there are more than300, 000 such block features for a template with the size of30 £ 30. Hence, the exhaustive feature selection process,Step 2(a) in Algorithm 1, is the most computationally in-tensive step in the GentleBoost algorithm.

When a multi-dimensional feature vector, such as thehistogram, is used in the weak classifier, the conventionalmethod of computing the threshold in the decision stumpclassifier, which often works comfortably with a 1-D fea-ture, can not be directly applied. In this paper, we em-ploy the idea of boosted histogram proposed by Laptev [11].As shown in Figure 2(d), a weighted Linear DiscriminativeAnalysis (LDA) is applied to the histogram features of pos-itive and negative samples, and results in the optimal pro-jection direction Ø. Thus, all histograms can be convertedto 1-D features by computing the inner product with Ø.

In summary, we use the following weak classifier:

f(x;p) =2º

atan(ØT

h(x0, y0, x1, y1)° t), (1)

where Ø is the LDA projection direction, t is the thresh-old and p are the parameters of the weak classifier p =[x0, y0, x1, y1,Ø, t]T. Given a cell location (x0, y0, x1, y1),the histogram features h are computed from all training datavia the integral histogram. Weighted LDA is applied tocompute Ø, and finally t is obtained through binary searchalong the span of LDA projections of all training data, suchthat the weighted least square error (WLSE) is minimal.Similar to [15], we use the atan() function in the weakclassifier, instead of the commonly used decision stump, be-cause of its derivability with respect to the parameters p.

4. Gradient Feature Selection4.1. Problem definition

We define the feature selection as a process of updatingthe parameters of each weak classifier p

m

. As shown inStep 2(a) of Algorithm 1, the WLSE is used in selecting theweak classifier from the hypothesis space during the boost-ing iteration. Hence, it is natural to use the WLSE as theobjective function for the feature selection (updating). Thisleads to the following problem we are trying to solve

minp

≤(f(x;p)) = minp

KX

i=1

wi

(f(xi

;p)° yi

)2. (2)

In the context of feature selection, solving this problemmeans that given the initial parameters p

(0), we look forthe new parameters of the weak classifier that can lead tosmaller WLSE on the dataset {x

i

} with K samples in total.We choose to use the gradient descent method to solve thisproblem iteratively.

4.2. Algorithm derivation

Plugging Eq. 1 into Eq. 2, the function to be minimizedis

≤ =KX

i=1

wi

(2º

atan(ØT

h

i

(x0, y0, x1, y1)° t)° yi

)2. (3)

Taking the derivative with respect to p gives

d≤

dp=

KX

i=1

2wi

(f(xi

)° yi

)df

i

dp, (4)

where dfi

dp

= [ @fi

@x0

@fi

@y0

@fi

@x1

@fi

@y1

@fi

@Ø

@fi

@t

]T. Based on Eq. 1, wehave

@fi

@z=

2º

ØT

@hi@z

1 + (ØT

h

i

° t)2, z = x0, y0, x1, y1,

@fi

@Ø=

2º

h

i

1 + (ØT

h

i

° t)2,

@fi

@t=

2º

°11 + (ØT

h

i

° t)2.

(5)

Xiaoming Liu and Ting Yu, “Gradient Feature Selection for Online Boosting,” in Proceeding of International Conference on Computer Vision (ICCV) 2007, Rio de Janeiro, Brazil, October 14-20, 2007. https://www.youtube.com/watch?v=QaeJQTVnB-Y

CSE 803 Fall 2017 28

Models of Reflectance

We need to look at models for the physics of illumination and reflection that will 1. help computer vision algorithms extract information about the 3D world, and 2. help computer graphics algorithms render realistic images of model scenes.

Physics-based vision is the subarea of computer vision that uses physical models to understand image formation in order to better analyze real-world images.

CSE 803 Fall 2017 29

The Lambertian Model: Diffuse Surface Reflection

A diffuse reflecting surface reflects light uniformly in all directions

Uniform brightness for all viewpoints of a planar surface.

CSE 803 Fall 2017 30

Real matte objects

CSE 803 Fall 2017 31

Specular reflection is highly directional and mirrorlike.

R is the ray of reflection V is direction from the surface toward the viewpoint α is the shininess parameter

CSE 803 Fall 2017 32

Real specular objects n  Chrome car parts are

very shiny/mirror like n  So are glass or ceramic

objects n  And waxy plant leaves

CSE 803 Fall 2017 33

Phong reflection model n  Reasonable realism, reasonable computing n  Uses the following components (a) ambient light (b) diffuse reflection component (c ) specular reflection component (d) darkening with distance Components (b), (c), (d) are summed over all light

sources. n  Modern computer games use more complicated

models.

CSE 803 Fall 2017 34

Phong shading model uses

CSE 803 Fall 2017 35

Phong model for intensity at wavelength lambda at pixel [x,y]

ambient diffuse specular