Date post: | 01-Apr-2015 |
Category: |
Documents |
Upload: | edward-grisson |
View: | 221 times |
Download: | 1 times |
Exploiting Big Data via Attributes(Offline Contd.)
Recap - Attributes
What are attributes?
Slide Credit: Devi Parikh
Recap - Attributes
Rich Understanding
Image Credit: Ali Farhadi
Recap - Annotations
Zero-shot learning
Frogs are green, have heads and legs. What is this?
Image Credit: Olga Russakovsky
Recap - Annotations
Attributes help in getting richer description from Annotators.
Image Credit: Devi Parikh
Understanding Single Image Or
Learning a Classifier (w/ Human Feedback)
Single or Few images to Big Data
Slide Credit: Abhinav Gupta
Big Data
90% of web data is visual!
142 Billion Images6 Billion added
monthly
6 Billion Images
72 hours of videouploaded every
minute
How attributes can help in learning from big-data?Slide Credit: Abhinav Gupta
What this part is about
Semi-supervised Learning
Slide Credit: Abhinav Gupta
What this part is about
Before the start of the debate, Mr. Obama and Mrs. Clinton met with the moderators,
Charles Gibson, left, and George Stephanopoulos, right, of ABC News.
A officer on the left of car checks the speed of other cars on the road.
Weakly-labeled Learning
Slide Credit: Abhinav Gupta
Key-insight
Attributes can help in coupling the learning and hence provide constraints for joint learning
Amph
ithea
tre
Audi
toriu
m
Goal: Learn multiple classifiers simultaneously. Ba
nque
tBe
droo
m
Slide Credit: Abhinav Gupta
Slide Credit: Abhinav Gupta
Semi-supervised Learning
Shrivastava et al., 2012
SEMI-SUPERVISED
[Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009] Slide Credit: Abhinav Gupta
Labeled Seed Examples
Amphitheatre
Unlabeled Data
Select Candidates
TrainModels
Add to Labeled Set
RetrainModels
Amphitheatre
BOOTSTRAPPING
Slide Credit: Abhinav Gupta
BOOTSTRAPPING
RetrainModels
Labeled Seed Examples
Amphitheatre
Unlabeled Data
Select Candidates
Add to Labeled Set
Amphitheatre
25th Iteration
[Curran et al., PACL 2007]
Semantic Drift
Amphitheatre + Auditorium
Slide Credit: Abhinav Gupta
GRAPH-BASED METHODS
[Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009] Slide Credit: Abhinav Gupta
Amphitheatre Amphitheatre
CONSTRAINED BOOTSTRAPPINGAmphitheatre
Auditorium
Amphitheatre
Auditorium
Slide Credit: Abhinav Gupta
Amphitheatre
Auditorium
Amphitheatre
Auditorium
Joint Learning
[Carlson et al., NAACL HLT Workshop on SSL for NLP 2009]
Share Data
CONSTRAINED BOOTSTRAPPING
Slide Credit: Abhinav Gupta
AmphitheatreAmphitheatre
AuditoriumAuditorium
BanquetHall
BanquetHall
Conference Room
Conference Room
Binary Attributes (BA)
Indoor Man-madeTables and Chairs Large Seating CapacityIndoor Man-madeTables and Chairs Large Seating Capacity
[Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009] Slide Credit: Abhinav Gupta
Slide Credit: Abhinav Gupta
Tables and Chairs
Conference Room
BanquetHall
Auditorium
Amphitheatre
Indoor
Large Seating Capacity
Man-made
[Patterson and Hays, CVPR 2012]
Tables and Chairs
Conference Room
BanquetHall
Auditorium
Amphitheatre
Indoor
Large Seating Capacity
Man-made
Binary Attributes (BA)
Slide Credit: Abhinav Gupta
AuditoriumIndoor Has Seat Rows
✗
Sharing via Dissimilarity
Amphitheatre Auditorium
Has Larger Circular Structures
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] Slide Credit: Abhinav Gupta
Amphitheatre AuditoriumHas Larger
Circular Structures
?Slide Credit: Abhinav Gupta
✗
Amphitheatre AuditoriumHas Larger
Circular Structures
Slide Credit: Abhinav Gupta
Dissimilarity
Has Larger Circular Structures
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
COMPARATIVE ATTRIBUTES
Slide Credit: Abhinav Gupta
• Similar to Relative Attributes.
• Uses pair of images as data-points during learning.
• Instead of predicting a real number, it uses binary classifier.
COMPARATIVE ATTRIBUTES
Slide Credit: Abhinav Gupta
DissimilarityCOMPARATIVE ATTRIBUTES
Has Larger Circular
Structures
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
……
……
……
……
Features• GIST• RGB (Tiny Image)• Line Histogram of:
Length Orientation
• LAB histogram
Slide Credit: Abhinav Gupta
……
……
DissimilarityCOMPARATIVE ATTRIBUTES
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
……
…… Has Larger
Circular Structures
ClassifierBoosted Decision Tree[Hoiem et al., IJCV 2007]
✗or
Has Larger Circular
Structures
Slide Credit: Abhinav Gupta
Comparative Attributes
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
Amphitheatre > Barn
Amphitheatre > Conference Room
Desert > Barn
Is More Open
Church (Outdoor) > CemeteryBarn > Cemetery
Has Taller Structures
Slide Credit: Abhinav Gupta
Amphitheatre
Auditorium
Amphitheatre
Auditorium
Labeled Seed Examples Bootstrapping
Slide Credit: Abhinav Gupta
Labeled Seed Examples
Amphitheatre
Auditorium
Amphitheatre
Auditorium
Bootstrapping
Amphitheatre
Auditorium
Constrained Bootstrapping
Indoor
Has Seat Rows
Attributes
Has Larger Circular
Structures
ComparativeAttributes
Slide Credit: Abhinav Gupta
Banq
uet
Bedr
oom
Labeled Data
Unlabeled Data
has more space
has larger structures
Training Pairwise Data
Promoted InstancesConference Room Banquet Hall
[Gupta and Davis, ECCV 2008]
Comparative Attribute Classifiers
mor
e sp
ace
larg
er st
ruct
ures
Attribute Classifiersin
door
has
gras
s
Scene Classifiers
bedr
oom
banq
uet h
all
Slide Credit: Abhinav Gupta
Boot
stra
ppin
gBA
Con
stra
ints
AmphitheatreC-
Boot
stra
ppin
gSe
ed Im
ages
BA C
onst
rain
ts
BridgeSe
ed Im
ages
Boot
stra
ppin
gC-
Boot
stra
ppin
g
Slide Credit: Abhinav Gupta
Attributes help improve Recall
Slide Credit: Abhinav Gupta
1
40
Banquet Hall
10
Itera
tions
Seed
Imag
es
Slide Credit: Abhinav Gupta
Itera
tion-
1Ite
ratio
n-60
Boot
stra
ppin
gC-
Boot
stra
ppin
gIte
ratio
n-1
Itera
tion-
60Se
ed Im
ages
Bedroom
Scene Classification
Eigen Functions: [Fergus et al., NIPS 2009] Slide Credit: Abhinav Gupta
Co-training (large Scale)
• 15 Scene Categories 25 Seed images / category
• Unlabeled Set 1Million (SUN Database + ImageNet) >95% distractors
SUN Database: [Xiao et al., CVPR 2010]ImageNet: [Deng et al., CVPR 2009]
Improve 12 out of 15 scene classifiers
Slide Credit: Abhinav Gupta
LIMITATIONS
C-bootstrapping uses semantic attributes and needs manually specified relationships
Amphitheatre > Barn
Amphitheatre > Conference Room
Desert > Barn
Is More Open
Can we learn the relationships?
Slide Credit: Abhinav Gupta
Choi et al., Adding Unlabeled Samples to Categories by Learned Attributes , CVPR 2013
Framework for jointly learning visual classifiers and noun-attribute mapping.
Formulation• A joint optimization for
– Learning classifier in visual feature space (wca)
– Learning classifier in attribute space (wcv)– With finding the samples (I)
• Non-convex– Mixed integer program: NP-complete problem– Solution: Block coordinate-descent
Learning a classifier on visual feature space
Learning a classifier on attribute spacewith a selection criterion
Mutual ExclusionNot convex
discrete continuous
Slide Credit: Junghyun Choi
Overview Diagram
Initial Labeled-Samples
Build Attribute Space
Project
Find Useful Attributes
Unlabeled Samples
Project
Choose Confident Examples To Add
Auxiliary data
Slide Credit: Jonghyun Choi
Example Qualitative Results
• Categorical: common traits of a categorySelected by Categorical Attributes
Initial Labeled Training Examples
Dotted
Animal-like shape
…
Slide Credit: Jonghyun Choi
Slide Credit: Abhinav Gupta
Weakly-Labeled Learning
Gupta et al., 2008
Captions - Bag of Nouns
Learning Classifiers involves establishing correspondence.
road.A officer on the left of car checks the speed of other cars on the
officercar
road
officer
car
road
Slide Credit: Abhinav Gupta
Correspondence - Co-occurrence Relationship
Bear
Water
Bear
FieldWater
Bear
Field
Slide Credit: Abhinav Gupta
Co-occurrence Relationship (Problems)
RoadCar RoadCar RoadCarRoadCar RoadCar RoadCarCar Road RoadCar
Hypothesis 1
Hypothesis 2
Car Road
Slide Credit: Abhinav Gupta
Beyond Nouns – Exploit Relationships
Use annotated text to extract nouns and relationships between nouns.
road.officer on the left of car checks the speed of other cars on theA
On (car, road)Left (officer, car)
car officer road
Constrain the correspondence problem using the relationships
On (Car, Road)
Road
Car
Road
Car
More Likely
Less Likely
Key insight: Solve the correspondence problem jointly using constraints!
Slide Credit: Abhinav Gupta
Relationships• Prepositions – A preposition usually indicates the temporal, spatial or
logical relationship of its object to the rest of the sentence
• The most common prepositions in English are "about," "above," "across," "after," "against," "along," "among," "around," "at," "before," "behind," "below," "beneath," "beside," "between," "beyond," "but," "by," "despite," "down," "during," "except," "for," "from," "in," "inside," "into," "like," "near," "of," "off," "on," "onto," "out," "outside," "over," "past," "since," "through," "throughout," "till," "to," "toward," "under," "underneath," "until," "up," "upon," "with," "within," and "without” where indicated in bold are the ones (the vast majority) that have clear utility for the analysis of images and video.
• Comparative attributes – relating to color, size, movement- “larger”, “smaller”, “taller”, “heavier”, “faster”………
Goal: Learn models of nouns, prepositions, comparative attributes simultaneously from weakly-labeled data.
Slide Credit: Abhinav Gupta
Learning the Model – Chicken Egg Problem
Chicken-Egg Problem: We treat assignment as missing data and formulate an EM approach.
Road
Car
Car
Road
Assignment Problem Learning Problem
On (car, road)
Slide Credit: Abhinav Gupta
EM Approach- Learning the Model
• E-Step: Compute the noun assignment for a given set of object and relationship models from previous iteration.
• M-Step: For the noun assignment computed in the E-step, we find the new ML parameters by learning both relationship and object classifiers.
• For initialization of the EM approach, we can use any image annotation approach with localization such as the translation based model described in [1].
[1] Duygulu, P., Barnard, K., Freitas, N., Forsyth, D.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. ECCV (2002)
Relationships modeled
• Most relationships are learned “correctly”– Above, behind, below, left, right, beside, bluer, greener,
nearer, more-textured, smaller, larger, brighter
• But some are associated with the wrong features– In (topological relationships not captured by color, shape
and location)– on-top-of– taller (most tall objects are thin and the segmentation
algorithm tends to fragment them)
Slide Credit: Abhinav Gupta
Resolution of Correspondence Ambiguities
[2] Barnard, K., Fan, Q., Swaminathan, R., Hoogs, A., Collins, R., Rondot, P., Kaufold, J.: Evaluation of localized semantics: data, methodology and experiments. Univ. of Arizona, TR-2005 (2005)
Duygulu et. al [1] Our Approach
[1] Duygulu, P., Barnard, K., Freitas, N., Forsyth, D.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. ECCV (2002)
below(birds,sun) above(sun, sea) brighter(sun,sea) below(waves,sun)
above(statue,rocks);ontopof(rocks, water); larger(water,statue)
below(flowers,horses); ontopof(horses,field); below(flowers,foals)
Slide Credit: Abhinav Gupta
Summary
• Attributes can help in exploiting big-data.
• Attributes represent how class A is similar to class B, and how class B is different from class A…
• These relationships can help in formulating joint-learning problem and improve learning from large unlabeled and weakly labeled data.
Slide Credit: Abhinav Gupta
Slide Credit: Abhinav Gupta