Relative Attributes

Mingxia Liumingxialiu@nuaa.edu.cn

1. Introduction 2. Relative Attributes[1]

3. Discussion 4. Our Intent Work

Outline

[1] Devi Parikh and Kristen Grauman.Relative Attributes. ICCV 2011 Best Paper.

Traditional Recognition

Dog Chimpanzee Tiger ???Tiger

3

Attributes-based Recognition

FurryWhite

BlackBig

StripedYellow

StripedBlackWhite

Big

Attributes provide a mode of

communication between humans and

machines!

[Lampert 2009][Farhadi 2009][Kumar 2009][Berg 2010][Parikh 2010]…

Zero-shot learningDescribing objectsFace verificationAttribute discoveryNameable attributes…

4

Dog Chimpanzee Tiger

Binary or Relative Attributes?

“Smiling”=1 “Smiling”=0

???

6

Smiling less than Gao, more than Ge

Binary or Relative Attributes?

Donkey

Mule

Horse

Binary or Relative Attributes?

Attributes for Mule

8[Oliva 2001] [Ferrari 2007] [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Wang 2009] [Wang 2010] [Berg 2010] [Branson 2010] [Parikh 2010] [ICCV 2011] …

Is furryHas four-legs

Has tail

Tail longer than donkeys’

Legs shorter than horses’

Binary

9

Is furryHas four-legs

Has tail

Tail longer than donkeys’

Legs shorter than horses’

Relative

10

Tail longer than donkeys’

Legs shorter than horses’

Is furryHas four-legs

Has tail

Relative Attributes

11

Enhanced human-machine communication

More informative

Natural for humans

1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent Work

Outline

Contents

Relative attributes◦ Allow relating images and categories to each other◦ Learn ranking function for each attribute

Novel applications◦ Zero-shot learning from attribute comparisons◦ Automatically generating relative image

descriptions

13

1) Relative Attributes Annotation

14

coast (C), forest (F), highway (H), inside-city (I), mountain (M), open-country(O), street (S) and tall-building (T)8 categories, 6 attributes

Attri

bute

2) Learning Relative Attributes

For each attribute , supervision is“open”

15

2) Learning Relative Attributes

Learn a scoring function

16

Learned parameters

Image features

that best satisfies constraints:

2) Learning Relative Attributes

17

Ranking SVM

Based on [Joachims 2002]

Image Relative Attribute Score

12

34

56

Rank Margin

2) Learning Relative Attributes

Wm

N : total categories ◦ N = S + U

S : ‘seen’ categories ◦ training images are provided relative attribute

relation such as “lions are larger than dogs, as large as tigers, but less large than elephants”

U : ‘unseen’ categories ◦ training images are not provided

3) Relative Zero-shot Learning

How to relate Seen categories and Unseen categories through relative attributes?

3) Relative Zero-shot Learning

For attribute m

If attribute m is not used to describe to be the mean of all training image

3) Relative Zero-shot Learning

-

Step1: Compute the attribute rank score:

Step2: Compute the mean and covariance of according to relative attribute description.

Step 3: Assign class label for using the Maximum Likelihood Method:

3) Relative Zero-shot Learning

3) Relative Zero-shot Learning

Need not use all attributes, or all seen categories.

23

Age:ScarlettCliveHugh Jared Miley

Smiling:

JaredMileyScarlett Clive Hugh

3) Relative Zero-shot Learning

Clive

Infer image category using max-likelihood

Can predict new classes based on their relationships to existing classes – without training images

24

Age:ScarlettCliveHugh

Jared Miley

HughCliveScarlettSmiling:

JaredMileySm

iling

Age

Miley

S

J H

4) Automatic Relative Image Description

Density

Conventional binary description: not dense

Dense: Not dense:

Novel image

25

more dense than less dense than

DensityNovel image

26

4) Automatic Relative Image Description

Experiments

Datasets

28

Outdoor Scene Recognition (OSR)[Oliva 2001]

8 classes 2700 images 6 attributes: open, natural, etc.

Public Figures Face (PubFig)[Kumar 2009]

8 classes 800 images11 attributes: white, chubby, etc.

Ranker vs. Classifier

++

+

–

– –Percentage correctly ordered

pairs Classifier Ranker

Outdoor scenes 80% 89%

Celebrity faces 67% 82%

29

Zero-shot learning◦ Binary attributes:

Direct Attribute Prediction [Lampert 2009]

◦ Relative attributes via classifier scores

Automatic image-description◦ Binary attributes

30

Baselines

Relative Zero-shot Learning

An attribute is more discriminative when used relatively

Binary attributes

Rel. att. (classifier)

31

Rel. att.(ranker)

Relative zero-shot learning

ProposedBinary attributes

Classifier score

32

Relative (ours):

More natural than insidecity Less natural than highway

More open than street Less open than coast

Has more perspective than highway Has less perspective than insidecity

Binary (existing):

Not natural

Not open

Has perspective

33

Automatic Relative Image Description

Relative (ours):

More natural than tallbuilding Less natural than forest

More open than tallbuilding Less open than coast

Has more perspective than tallbuilding

Binary (existing):

Not natural

Not open

Has perspective

34

Automatic Relative Image Description

Relative (ours):

More Young than CliveOwenLess Young than ScarlettJohansson

More BushyEyebrows than ZacEfron Less BushyEyebrows than AlexRodriguez

More RoundFace than CliveOwenLess RoundFace than ZacEfron

Binary (existing):

Not Young

BushyEyebrows

RoundFace

Automatic Relative Image Description

35

(Viggo)

1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent work

Outline

Relative attributes representation

Attribution relation learning

Discussion

1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent work

Outline

Learning attributes’ relation automatically

Learning attributes’ relation with noise

Attribute selection

Instance selection for attributes

Our Intent Work

1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent work

Outline

Thank You