Computer VisionMachine Learning
FeaturesPresented By
Dr. Keith Haynes
Introduction Appearance-Based Approach Features Classifiers Face Detection Walkthrough Questions
Outline
Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images.
What does that mean? What are some computer vision task?
Computer Vision
Detection
Are there any faces in this image?
Recognition
+ ClassLabel
Database of Classes
TestSubject
Pre p ro c e ssing
Se nsing
Fe a ture Extra c tio n
C la ssific a tio n
Po st-Pro c e ssing
Dec isio n
Pattern Recognition System
Computer Vision a difficult problem
Images vary due to the relative camera-object pose
Frontal, profile, etc.
Pose
Components may vary in:◦ Size◦ Shape◦ Color◦ texture
Structural Components
Some objects have the ability to change shape
Deformability
Computational Complexity There are many possible objects Scale Orientation
15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0
Crux of the Problem
As the dimensions increase, the volume of the space increases exponentially
The data points occupy a volume that is mainly empty.
Under these conditions, tasks such as estimating a probability distribution function become very difficult.
In high dimensions the training sets may not provide adequate coverage of the space.
Curse of Dimensionality
Machine learning is the science of getting computers to act without being explicitly programmed.
Applications◦ self-driving cars◦ speech recognition◦ effective web search◦ understanding of the human genome
Machine Learning
15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0
Humans Don’t Understand
Model-Based◦ Uses 3D models to generate images◦ Original and rendered images compared for
classification Appearance-Based
◦ Learns how to classify image via training examples
Computer Vision Approaches
S e t o f D iscrim in ato ry
F eature s
Tra in ing S e t
L ea rn
P e rfo rmF eatu re E x trac tion
Test S ub ject
F eatureR epresen tatio n o f
Test S ub ject
C lass ify
C lassL ab el
Q u ick ly & A c cu ra te ly
The Concept
Features are learned through example images, usually known as a training set
3D Models are not needed Utilizes machine learning and statistical
analysis
Appearance-Based Approach
Training Sets
ORL (Face recognition)
COIL-100 (Object Recognition)
A feature is a calculation performed on a portion of an image that yields a number
Features are used to represent the entity being analyzed.
Image Features
15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0
Image
Haar Features Computes the
difference between sums of two or more areas
Edge detector
A B
D
C
15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0
Image199 527
-328
414152
262
+ -
Feature representation is determined by:◦ the task being performed◦ performance constraints such as accuracy and
calculation time.
Two Groups◦ Global – feature uses the entire image◦ Local – feature uses parts of the image
Image Features
Local Features Attempts to identify the critical areas from
a set of images for class discrimination How are critical areas identified? Requires an exhaustive search of possible
sub-windows
Rectangular Features
2MP image has 922,944,480,000 possible features and took 16.45 min
Height Width Possible Features Time1 1 1 < ms2 2 9 < ms3 3 36 < ms4 4 100 < ms
24 24 90,000 0.2 ms128 128 68,161,536 0.1 sec256 256 1,082,146,816 1.19 sec512 512 17,247,043,584 18.48 sec
A single Haar feature is a weak classifier
A set of features can form a strong classifier
Features and Classification
Haar Feature Effectiveness
Features in Set Number of Sets1 90,0002 8,099,910,0003 7.28976E+144 6.56056E+195 5.90424E+246 5.31352E+297 4.78185E+348 4.30333E+399 3.87266E+44
10 3.48504E+49
Sets of Features
Exhaustive Search◦ For 5 features 5.9x1024 unique sets
Find best features one at a time.◦ Find the first best feature◦ Find the feature that works best with the first
feature, and so on◦ For 5 features 449,990 sets searched
Increase step size
Search Method
Example Features
Together they form a strong classifier
15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0
Feature Extraction
Original Image
47-229-498179106-157-3461124-99-257423
FeatureSet
Feature selection is important, is application dependent
Statistical methods very useful with high dimensionality
Local identify discriminating areas or features images
No universal solution Features can be combined
Summary
The Classifier
Types of Classifiers Linear Discriminant Analysis Fisher Discriminant Analysis Bayesian Classifier Neural Networks K-Nearest Neighbor Classifier
Features can used to form a coordinate space called the feature space.
Euclidean distance is used as the metric
Nearest Neighbor Classification
221
21211 )(...)( dd xxxxX
Feature Selection
The distance is not used directly for feature selection
The higher the ratio, the better the filter
In order to prevent one class from dominating, an exponential function was used
The sum of function for all test images was used for selection
[Liu, Srivastava, Gallivan]
Feature Space Examples
Separation and grouping Better Classification Low Classification Rates
Rapid Classification Tree
“Divide and Conquer” Instead of trying to solve a difficult problem
all at once, divide it into several parts Each of the resulting parts should be easier
to solve than the original problem Perform classifications fast
Rapid Classification Tree
Example RCT1,..,20
2,7,11,12,14,16,17,19
7,11,16,17
11,177,16
14,192,12
1,13,15,18
13,181,15
5,6,9,103,4,8,20
- Indicates that all children are leaf nodes
Principal Component Analysis Classical technique that is widely used for
image compression and recognition Produces features with a dimensionality
significantly less than that of the original images
Reduction is performed without a substantial loss of the data contained in the image
Analysis is based on the variance of dataset◦ Variance implies a distinction in class
PCA
47-229-498179106-157-3461124-99-257423
FeatureSet
478-367206-358386
LowerDimensional
Space
FeatureSet
PCAMatrix×
In many cases, the PCA reduction was not sufficient
Improving the performance of the reduction matrix is necessary
Four methods were implemented◦ Gradient Search◦ Random or Vibration Search
Variation of the Metropolis Algorithm◦ Neighborhood Component Analysis◦ Stochastic Gradient Search
3. Reduction Optimization
Optimization Search Data reduction occurs via a matrix
multiplication◦ x′ = xA
Optimization is achieved by ◦ defining F as a function A, F(A)◦ Changing A
Gradient Search
Gradient Search Can be computationally expensive Does not provide a means to escape a
local maximum
x
f(x )
lo c a lm a x im u m
g lo b a lm a x im u m
Stochastic Search Makes a guess Guesses are fast There is a possibility of escaping a local
maximum
x
f(x )
lo c a lm a x im u m
g lo b a lm a x im u m
Stochastic Search Restricting the search area increases the
probability of finding an increasing path
If all data at the node can be classified accurately◦ The classification decision is stored as leaf nodes◦ No further processing occurs down this branch of the
tree. If data cannot be classified accurately
◦ The problem then becomes a clustering one.◦ Accuracy is now defined in terms of clusters, not classes.
The accuracy achieved through class level clustering will always be no worse than that of individual classes and in most cases, will be higher.
Clustering
F e a tu re S p a c e D e c is io n B o u n d a ry P rio r to C lu s te r in g
D e c is io n B o u n d a ry A f te r C lu s te r in g
Clustering Process
Decomposition
2
4
105
1
3
796
8
R
AB C
D
E
Generated RCT
F e a ture S pac e D e c is ion B ou nd ary A fte r C lu ster in g
A
R o o t
B
Decomposition2
4
105
1
3
796
8
R
AB C
D
E
Decomposition
F ea tu re S pa ce D e c isio n B ou nd a ry A fte r C lu sterin g
C
D
E
Decomposition2
4
105
1
3
796
8
R
AB C
D
E
Classifier Dataset Accuracy ThroughputKNN 100% 16,735Neural Network 92.50% 53,522SVM 86.25% 7,251RCT 97% 982,110
KNN 100% 19,204Neural Network 89% 53,709SVM 97.75% 3,790RCT 100% 3,781,933
SVM 96.96% 141,240RCT 97.35% 8,949,638
ORL
COIL
Breast Cancer
Classifier Comparisons
X. Liu, A. Srivastava, K. Gallivan, Optimal linear representations of images for object recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, May 2004, pp. 662-666.
Haynes, K., Liu, X., Mio, W., (2006). Object Recognition Using Rapid Classification Trees. 2006 International Conference on Image Processing
Haynes, K. (2011). Using Image Steganography to Establish Covert Communication Channels. International Journal of Computer Science and Information Security, Vol. 9, No. 9
Duda, R., Hart, P., Stork, D. (2001). Pattern Classification. Wiley-Interscience Publications, NY
References
Questions?