3D Shape AttributesDavid F. Fouhey1, Abhinav Gupta1, Andrew Zisserman2
Overview
Sculpture Data
PASCAL VOC Results
Mental Rotation
CMU Robotics Institute1, University of Oxford2
Pla
nar
ity
Most Least
…
…
…
Pt/
Lin
e C
on
t.
…
PlanarSurfaces
Point/Line Contact
ThinStructures
Rough Surfaces
HasHoles
Positives NegativesImage Samples
Base Network
…
Attr.Only
FullModel
or ……
Idea: do two images show the same object but rotated?
Evaluate as classification over ~100M pairs of imagesCosine similarity = classification prediction for featurePositive = same work; Negative = different work
Easy setting: all pairs (ignore same viewpoint cluster)Hard setting: remove “easy” positives via BOW+SIFT
Sculpture Attribute Results
Curvature Contact
Planar !Planar Cylinder Rough Pt/Line Multiple
82.8 77.2 56.9 76.0 74.4 76.4
Occupancy
Empty 2+ Pieces Holes Is Thin Mirr. Sym Cubic Ratio
87.0 60.4 69.3 85.8 60.8 60.3
ZPlanar = YHoles = Y
…2+ Contacts = N
Is Empty = N…
Indirect Baselines
KDES
Eigen ‘14
58.5
HHA
Barron ‘15
59.461.2 62.5 72.3
End-to-end
KDES HHA
…
Thin Structures
Point/Line ContactMost Least
…
Rough Surfaces
…
No Planar Surfaces
…
Qualitative Quantitative (AUROC)
Learning
12D Shape
Attributes
Conv. Layers FC LayersInput
VGG-M
1024D Shape
Embedding
a p
n
Label: {0, 1, invalid}
Prediction [0,1]
Anchor’s feature
Positive’s feature
Negative’s feature
Anchor Positive Negative
Objective: learn features such thattwo views of same work are closer than two views of different works by a margin
Objective: describe 3D shape of generic objects interms of qualitative, higher-order properties
- -
0 +
+ +
0 -
- +
0 0
Historical motivation
Not planarSmooth surface1 point of contactNot point contactHas holeNot thin structures…
Contrast objective with
Category Specific Metric 2.5D Maps (Depth, Normals)
Gathering Data
H. Moore
…
A. Calder
…
Two Forms
Sheep Piece
Locking Piece
5 Swords
Eagle
…
242 Artists 2197 Works
Sculpture list from
…
…
…
143K Images in 9352 Viewpoint Clusters
…
Princeton Columbus Toronto
YorkshireMalagaLondon
Mountains & Clouds
Five Swords
Red Feather
By Alexander Calder
Artist/Shape Correlation
Diverse Materials, Locations
…
Modern Sculpture
Great shape diversity
Categories we can’t describe
Ordinary Objects
Limited shape diversity
Category/shape correlation
Attributes
Curvature Properties
Contact Properties
Volumetric Properties
Cubic Asp. RatioHas Hole Thin Structures Mirror Symm.
Has RoughnessHas Planarity No Planarity Has Cylindrical
Multiple PiecesPoint/line contact Multiple Contacts Mainly Empty
Data source possibilities
We investigate twelve 3D shape attributes inspired by past work in vision
Shape Embedding
Images from
False Positive Rate
Tru
e P
osit
ive
Rat
e
False Positive Rate
Easy Setting (0.9% Positives) Hard Setting (0.3% Positives)
Tru
e P
osit
ive
Rat
e
Evaluation done on 35K images by held-out artists