3D ShapeNets: A Deep Representation for Volumetric Shape...

3D ShapeNets: A Deep Representation for

Volumetric Shape Modelingby Wu, Song, Khosla, Yu, Zhang, Tang, Xiao

presented by Abhishek Sinha

1

3D Shape Prior

2Slide Credit: Wu, Song et al. 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling, CVPR 2015

3D Shape Prior


3D Shape Prior


3

Outline

3

Outline

• Problem

3

Outline

• Problem• Motivation

3

Outline

• Problem• Motivation• Desirable Properties for Representation

3

Outline

• Problem• Motivation• Desirable Properties for Representation• Architecture

3

Outline

• Problem• Motivation• Desirable Properties for Representation• Architecture• Dataset

3

Outline

• Problem• Motivation• Desirable Properties for Representation• Architecture• Dataset• Applications

3

Outline

• Problem• Motivation• Desirable Properties for Representation• Architecture• Dataset• Applications• Extensions

3

Outline

• Problem• Motivation• Desirable Properties for Representation• Architecture• Dataset• Applications• Extensions• Discussion Points

4

Problem

Learn ‘Useful’ 3D shape representations from

images

6

Motivation

7

3D Shape Representation

Slide Credit: Wu, Song et al. 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling, CVPR 2015

7


Shape Synthesis


7


Shape Synthesis

Shape Completion


7


Shape Synthesis

Shape Completion

2.5D Object Recognition

person

tricycle


7


Shape Synthesis

Feature ExtractorShape Completion

2.5D Object Recognition

person

tricycle


8

Desirable Properties

What is a Desirable 3D Shape Representation?

9



9

Data-driven

Generic

Compositional

Versatile


10

Data-driven

Generic

Compositional

Versatile


Data-driven


11

Simple Shapes Complex Shapes

Data-driven

Generic

Compositional

Versatile

Generic



12

building blocks full object

Data-driven

Generic

Compositional

VersatileCompositional



13

mesh classification shape completion

shape generation2.5D object recognition

person

tricycle

Data-driven

Generic

Compositional

Versatile


Versatile Slide Credit: Wu, Song et al. 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling, CVPR 2015

14

Architecture

15

3D Deep Learning


15

3D Deep Learning3D Shape as Volumetric Representation


15

3D Deep Learning

mesh

3D Shape as Volumetric Representation


15

3D Deep Learning

mesh



15

3D Deep Learning

mesh binary voxel



16

3D ShapeNets

Convolutional Deep Belief Network


16

3D ShapeNets


A Deep Belief Network is a generative graphical model that describes the distribution of input x over class y.


16

3D ShapeNets

• Convolution to enable compositionality• No pooling to reduce reconstruction

error




16

3D ShapeNets

• Convolution to enable compositionality• No pooling to reduce reconstruction

error

Layer 1-3 convolutional RBM

Layer 4 fully connected RBM

Layer 5 multinomial label + Bernoulli feature form an associate memory

configurations




17

3D ShapeNets



17

3D ShapeNets


3D ShapeNets ≠ CNNs


17

3D ShapeNets




17

3D ShapeNets




17

3D ShapeNets



generative process

discriminative process


17

3D ShapeNets



* 3D ShapeNets can be converted into a CNN, and discriminatively trained with back-propagation.

generative process

discriminative process


18

TrainingMaximum Likelihood Learning



18

Training

Layer-wise pre-training:

Lower four layers are trained by CD

Last layer is trained by FPCD[1]

Fine-tuning:

Wake sleep[2] but keep weights tied

[2] Hinton, et al "A fast learning algorithm for deep belief nets." Neural computation[1] Tijmen, et al. "Using fast weights to improve persistent contrastive divergence.”

Maximum Likelihood Learning



18

Training

Layer-wise pre-training:

Lower four layers are trained by CD

Last layer is trained by FPCD[1]

Fine-tuning:

Wake sleep[2] but keep weights tied

[2] Hinton, et al "A fast learning algorithm for deep belief nets." Neural computation[1] Tijmen, et al. "Using fast weights to improve persistent contrastive divergence.”

Maximum Likelihood Learning



generation process:Gibbs Sampling

Convolutional Deep Belief Network 19

Sampling


Sampling


20

object label


Sampling


20

object label


Sampling


20

object label


Sampling


20

object label

••••


Sampling


20

object label

••••


21

Dataset

Big 3D Data


Big 3D DataQuery Keyword: common object categories from the SUN database that

contain no less than 20 object instances per category


Big 3D Data

23

Query Keyword: common object categories from the SUN database that

contain no less than 20 object instances per category


Big 3D Data

151,128 models 660 categories Slide Credit: Wu, Song et al. 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling, CVPR 2015

25

Applications

2.5D Completion & Recognition

26

Slide Credit: Wu et al


26



26



26



26



26



26



26



26

Gibbs sampling with clamping Slide Credit: Wu et al


26

Gibbs sampling with clamping Slide Credit: Wu et al

27[29] R. Socher, B. Huval, B. Bhat, C. D. Manning, and A. Y. Ng. Convolutional-recursive deep learning for 3d object classification. In NIPS 2012.


Training on CAD models and no discriminative tuning!


View Planning for Recognition

28



Volumetric representation

28




sofa?

bathtub?What is it?

?dresser?

28




sofa?

bathtub?What is it?

?dresser?

Not sure. Look from

another view?

28




sofa?

bathtub?What is it?

?dresser?

Not sure. Look from

another view? Where to look next?

Next-Best-View

28



Volumetric representation New depth map

sofa?

bathtub?What is it?

?dresser?

Not sure. Look from


Next-Best-View

28




sofa?

bathtub?What is it?

?dresser?

Aha! It is a sofa!

Not sure. Look from


Next-Best-View

28




sofa?

bathtub?What is it?

?dresser?

3D ShapeNets

Aha! It is a sofa!

Not sure. Look from


Next-Best-View

28


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


Deep View Planning

Mathematically, this is equivalent to evaluate the conditional mutual information:

0.8

0.5

0.2

0.3 0.4

0.8 0.3

0.8

0.80.3

0.7

0.3

0.4

0.8

0.80.3

29


30

Recognition Accuracy from Two Views.

Deep View Planning

Based on the algorithms’ choice, we obtain the actual depth map for the next view and recognize the objects using two views by our 3D ShapeNets.


30

Recognition Accuracy from Two Views.

Deep View Planning

Based on the algorithms’ choice, we obtain the actual depth map for the next view and recognize the objects using two views by our 3D ShapeNets.

Our algorithm stands out as the uncertainty of the first view increasesSlide Credit: Wu, Song et al. 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling, CVPR 2015

Back Propagation Fine-tuning

31

48 filters of stride 2



30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

4000

object label




30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

4000

object label




30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

object label




30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

object label




30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

object label




30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

object label




30

13

5

1200

2

4000

object label

3D ShapeNets



31




30

13

5

1200

2

object label




30

13

5

1200

2

4000

object label

3D ShapeNets 3D CNN


As a 3D Feature Extractor

32



32

Mesh Classification & Retrieval



32

Mesh Classification & Retrieval

[29] R. Socher, B. Huval, B. Bhat, C. D. Manning, and A. Y. Ng. Convolutional-recursive deep learning for 3d object classification. In NIPS 2012.

2.5D object recognition


33mesh retrieval



Extensions

34

• Include RGB information in representation

• 3D Segmentation

• Improve for non-rigid 3D objects

35

Discussion Points

36

• Is the network deep enough?

• 30x30x30 = 27000 vs 256x256 = 65000 for Image Net

• 150K training examples vs millions for Image Net

37

• Won’t removal of max-pooling layers hurt performance on classification tasks?

http://3dshapenets.cs.princeton.edu/38

http://3dshapenets.cs.princeton.edu/

• Any other systems that use binary units with approximate training and inference techniques rather than standard back-prop?

• Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18.7 (2006): 1527-1554

• Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.

39

• Are there better ways for representing 3D Shapes. In particular, doesn’t the voxel representation have the bottleneck of cubic dependency on grid size?

• Yes. Su, Majhi et al that tries to recognize 3D shapes from multiple 2D views instead of voxel representation and get better results for classification .

H. Su, S. Maji, E. Kalogerakis, E. Learned-Miller. Multi-view Convolutional Neural Networks for 3D Shape Recognition. ICCV2015.40

• Are there other 3D CAD model datasets

• 3D Warehouse. https://3dwarehouse.sketchup.com/

• Manually removing clutter from 3D CAD models a problem

• Did not address non-rigid objects sufficiently.

• Even the 40 model classification dataset seemed to contain only 4 non-rigid categories — persons, plant, sofas, curtains.

41

https://3dwarehouse.sketchup.com/

Appendix

42

Contrastive divergence learning: A quick way to learn an RBM

0>< jihv 1>< jihv

i

j

i

j

t = 0 t = 1

)( 10 ><−><=Δ jijiij hvhvw ε

Start with a training vector on the visible units. Update all the hidden units in parallel Update all the visible units in parallel to get a “reconstruction”. Update all the hidden units again.

This is not following the gradient of the log likelihood. But it works well. It is approximately following the gradient of another objective function.

reconstructiondata

12

Slide Credit: Geoffery Hinton

43

The wake-sleep algorithm for an SBN

• Wake phase: Use the recognition weights to perform a bottom-up pass.

– Train the generative weights to reconstruct activities in each layer from the layer above.

• Sleep phase: Use the generative weights to generate samples from the model.

– Train the recognition weights to reconstruct activities in each layer from the layer below.

h2

data

h1

h3

2W

1W1R

2R

3W3R

Slide Credit: Geoffery Hinton

44

Date post:	03-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

3D ShapeNets: A Deep Representation for Volumetric Shape...

Documents