SSD: Single Shot MultiBox Detector

Wei Liu(1), Dragomir Anguelov(2), Dumitru Erhan(3), Christian Szegedy(3), Scott Reed(4), Cheng-Yang Fu(1), Alexander C. Berg(1)

UNC Chapel Hill(1), Zoox Inc.(2), Google Inc.(3), University of Michigan(4)

VGGNet Titan X Pascal

VGGNet Titan X Pascal

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

All with VGGNet pretrained on ImageNet, batch_size = 1 on Titan X

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30074% mAP / 46 fps

6.6x faster

All with VGGNet pretrained on ImageNet, batch_size = 1 on Titan X

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30074% mAP / 46 fps

6.6x faster

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30074% mAP / 46 fps

SSD51277% mAP / 19 fps

11% better

All with VGGNet pretrained on ImageNet, batch_size = 1 on Titan X

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30074% mAP / 46 fps

6.6x faster

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30074% mAP / 46 fps

SSD51277% mAP / 19 fps

11% better

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps SSD300

74% mAP / 46 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD51277% mAP / 19 fps

SSD30077% mAP / 46 fps

SSD51280% mAP / 19 fps

All with VGGNet pretrained on ImageNet, batch_size = 1 on Titan X

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30077% mAP / 46 fps

SSD51280% mAP / 19 fps

10 20 30 40 50Speed (fps)

70

80VO

C20

07 te

st m

AP

R-CNN, Girshick 201466% mAP / 0.02 fps

Fast R-CNN, Girshick 201570% mAP / 0.4 fps

Faster R-CNN, Ren 201573% mAP / 7 fps

YOLO, Redmon 201666% mAP / 21 fps

SSD30077% mAP / 46 fps

SSD51280% mAP / 19 fps

Two-Stage

box proposal + postclassify Single Shot

Classical sliding windows

Bounding Box Prediction

Classical sliding windows

Bounding Box Prediction

Is it a cat? No

Is it a cat? No

Discretize the box space densely

Classical sliding windows

Bounding Box Prediction

SSD and other deep approaches

Is it a cat? No

Discretize the box space densely

Classical sliding windows

Bounding Box Prediction

SSD and other deep approaches

Is it a cat? No

Discretize the box space densely

Classical sliding windows

Bounding Box Prediction

SSD and other deep approaches

cat: 0.8 dog: 0.1Is it a cat? No

Discretize the box space densely

Classical sliding windows

Bounding Box Prediction

SSD and other deep approaches

Is it a cat? No

Classical sliding windows

Bounding Box Prediction

Discretize the box space densely

SSD and other deep approaches

Is it a cat? No

Classical sliding windows

Bounding Box Prediction

Discretize the box space densely

SSD and other deep approaches

Is it a cat? No

Classical sliding windows

Bounding Box Prediction

Discretize the box space densely

SSD and other deep approaches

dog: 0.4 cat: 0.2Is it a cat? No

Classical sliding windows

Bounding Box Prediction

Discretize the box space densely

SSD and other deep approaches

dog: 0.4 cat: 0.2Is it a cat? No

Classical sliding windows

Bounding Box Prediction

Discretize the box space more coarselyRefine the coordinates of each boxDiscretize the box space densely

ConvNet

feature map

SSD Output Layer

ConvNet

feature map

SSD Output Layer

small (e.g. 3x3) conv kernel

ConvNet

feature map

SSD Output Layer

small (e.g. 3x3) conv kernel

default box

feature map

box regression

multiclass probabilities

SSD Output Layer

ConvNet

ConvNet

feature map

box regression

multiclass probabilities

SSD Training• Match default boxes to ground truth boxes to determine true/false positives.

• Loss = SmoothL1(box param) + Softmax(class prob)

Smooth L1 loss Softmax loss

Related Work

Related WorkMultiBox [Erhan et al. CVPR14]

P(objectness) for K boxes

Fully connected

Offsets for K boxes

Related WorkMultiBox [Erhan et al. CVPR14]

P(objectness) for K boxes

Fully connected

Offsets for K boxes

+ post classify boxes

Related WorkYOLO [Redmon et al. CVPR16]

multiclass probfor K boxes

Fully connected

Offsets for K boxes

MultiBox [Erhan et al. CVPR14]

P(objectness) for K boxes

Fully connected

Offsets for K boxes

+ post classify boxes

Related WorkYOLO [Redmon et al. CVPR16]

multiclass probfor K boxes

Fully connected

Offsets for K boxes

Faster R-CNN [Ren et al. NIPS15]

Convolutional

P(objectness) Box offsets

MultiBox [Erhan et al. CVPR14]

P(objectness) for K boxes

Fully connected

Offsets for K boxes

+ post classify boxes

Related WorkYOLO [Redmon et al. CVPR16]

multiclass probfor K boxes

Fully connected

Offsets for K boxes

Faster R-CNN [Ren et al. NIPS15]

Convolutional

P(objectness) Box offsets

+ post classify boxes

MultiBox [Erhan et al. CVPR14]

P(objectness) for K boxes

Fully connected

Offsets for K boxes

+ post classify boxes

Related WorkYOLO [Redmon et al. CVPR16]

multiclass probfor K boxes

Fully connected

Offsets for K boxes

Faster R-CNN [Ren et al. NIPS15]

Convolutional

P(objectness) Box offsets

SSD

Convolutional

Box offsetsmulticlass prob

+ post classify boxes

MultiBox [Erhan et al. CVPR14]

P(objectness) for K boxes

Fully connected

Offsets for K boxes

+ post classify boxes

Contribution #1:Multi-Scale Feature Maps

ConvNet

box regression

multiclass scores

Contribution #1:Multi-Scale Feature Maps

ConvNet

box regression

multiclass scores

stride 2 convolution

Contribution #1:Multi-Scale Feature Maps

ConvNet

box regression

multiclass scores

box regression

multiclass scores

stride 2 convolution

8⇥ 8 feature map 4⇥ 4 feature map

vs.

8⇥ 8 feature map

SSD

Multi-Scale Feature Maps

8⇥ 8 feature map 4⇥ 4 feature map

vs.

8⇥ 8 feature map

SSD

Multi-Scale Feature Maps

Faster R-CNN Objectness Proposal, Ren 2015

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

boundary boxes

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Prediction source layers from:

mAP

use boundary boxes?

# Boxes

38⇥ 38 19⇥ 19 10⇥ 10 5⇥ 5 3⇥ 3 1⇥ 1 Yes No

4 4 4 4 4 4 74.3 63.4 8732

4 4 4 70.7 69.2 9864

4 62.4 64.0 8664

Multi-Scale Feature Maps Experiment

Contribution #2:Splitting the Region Space

ConvNet convolution

SSD300include { 1

2 , 2} box? 4 4include { 1

3 , 3} box? 4number of Boxes 3880 7760 8732

VOC2007 test mAP 71.6 73.7 74.3

Contribution #2:Splitting the Region Space

ConvNet convolution

Contribution #2:Splitting the Region Space

ConvNet convolution

Use 38x38 feature map : +2.5 mAP (conv4_3)

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

GT

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

GT DETECTION

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

• SmoothL1 or L2 loss for box shape averages among likely hypotheses

GT DETECTION

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

• SmoothL1 or L2 loss for box shape averages among likely hypotheses

• Need to have enough default boxes (discrete bins) to do accurate regression in each

GT DETECTION

Why So Many Default Boxes?Faster R-CNN YOLO SSD300 SSD512

# Default Boxes 6000 98 8732 24564Resolution 1000x600 448x448 300x300 512x512

• SmoothL1 or L2 loss for box shape averages among likely hypotheses

• Need to have enough default boxes (discrete bins) to do accurate regression in each

• General principle for regressing complex continuous outputs with deep nets

GT DETECTION

Handling Many Default Boxes

• Matching ground truth and default boxes

Handling Many Default Boxes

• Matching ground truth and default boxes

Handling Many Default Boxes

`

`

GT

• Matching ground truth and default boxes

Handling Many Default Boxes

`

`

GT Default box

• Matching ground truth and default boxes

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

• Matching ground truth and default boxes

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

?

• Matching ground truth and default boxes‣ Match each GT box to closest default box

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

?

• Matching ground truth and default boxes‣ Match each GT box to closest default box

‣ Also match each GT box to all unassigned default boxes with IoU > 0.5

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

?

• Matching ground truth and default boxes‣ Match each GT box to closest default box

‣ Also match each GT box to all unassigned default boxes with IoU > 0.5

• Hard negative mining

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

?

• Matching ground truth and default boxes‣ Match each GT box to closest default box

‣ Also match each GT box to all unassigned default boxes with IoU > 0.5

• Hard negative mining• Unbalanced training: 1-30 TP, 8k-25k FP

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

?

• Matching ground truth and default boxes‣ Match each GT box to closest default box

‣ Also match each GT box to all unassigned default boxes with IoU > 0.5

• Hard negative mining• Unbalanced training: 1-30 TP, 8k-25k FP

• Keep TP:FP ratio fixed (1:3), use worst-misclassified FPs.

Handling Many Default Boxes

`

`

GT Default box

TP

TP

FP

?

SSD Architecture

300

VGG16

Det

ectio

ns:8

732

per

Cla

ss

Classifier : Conv: 3x3x(3x(Classes+4))

Non

-Max

imum

Sup

pres

sion

74.3mAP 46FPS

Classifier : Conv: 3x3x(6x(Classes+4))

SSD

Extra Convolutional Feature Maps

Conv: 3x3x(4x(Classes+4))

38 19 10

1910

300 38

5

5

31

image

Contribution #3:The Devil is in the Details

Data Augmentation

Data Augmentation

`

`

Data Augmentation

`

`

`

` `

`

Data Augmentation

`

`

`

` `

`

data augmentation SSD300horizontal flip 4 4

random crop & color distortion 4VOC2007 test mAP 65.5 74.3

Data Augmentation

`

`

Data Augmentation

`

`

`

`

``

Random expansion creates more small training examples

Data Augmentation

`

`

`

`

``

Random expansion creates more small training examples

Data Augmentation

data augmentation SSD300horizontal flip 4 4 4

random crop & color distortion 4 4random expansion 4

VOC2007 test mAP 65.5 74.3 77.2

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

6.6x

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

10%

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

Results on VOC2007 test

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

Results on VOC2007 test

77.2

77.279.8

79.8

Method mAP FPS batch size # Boxes Input resolution

Faster R-CNN (VGG16) 73.2 7 1 ⇠ 6000 ⇠ 1000⇥ 600

Fast YOLO 52.7 155 1 98 448⇥ 448YOLO (VGG16) 66.4 21 1 98 448⇥ 448

SSD300 74.3 46 1 8732 300⇥ 300SSD512 76.8 19 1 24564 512⇥ 512SSD300 74.3 59 8 8732 300⇥ 300SSD512 76.8 22 8 24564 512⇥ 512

Results on More Datasets

Results on More Datasets

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/A

Results on More Datasets

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/ASSD300 74.3 72.4 23.2 43.4

Results on More Datasets

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/ASSD300 74.3 72.4 23.2 43.4

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/ASSD300 74.3 72.4 23.2 43.4SSD512 76.8 74.9 26.8 46.4

Results on More Datasets

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/A

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/ASSD300 74.3 72.4 23.2 43.4

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/ASSD300 74.3 72.4 23.2 43.4SSD512 76.8 74.9 26.8 46.4

Method VOC2007test

VOC2012test

MS COCOtest-dev

ILSVRC2014val2

Fast R-CNN 70.0 68.4 19.7 N/AFaster R-CNN 73.2 70.4 21.9 N/A

YOLO 63.4 57.9 N/A N/ASSD300* 77.2 75.8 25.1 N/ASSD512* 79.8 78.5 28.8 N/A

COCO Bounding Box precision

COCO Bounding Box precision

mAP @ IoU 0.5 0.75 0.5:0.95

Faster R-CNN 45.3 23.5 24.2SSD512* 48.5 30.3 28.8

gain +3.2 +6.8 +4.6

Future Work

• Object detection + pose estimation

Future Work

• Object detection + pose estimation

Figure 2. Two-stage vs. Proposed. (a) The two-stage approach separates the detection and pose estimation steps. After object detection,the detected objects are cropped and then processed by a separate network for pose estimation. This requires resampling the image at leastthree times: once for region proposals, once for detection, and once for pose estimation. (b) The proposed method, in contrast, requires noresampling of the image and instead relies on convolutions for detecting the object and its pose in a single forward pass. This offers a largespeed up because the image is not resampled, and computation for detection and pose estimation is shared.

3. ModelFor an input RGB image, a single evaluation of the

model network is performed and produces scores for cat-egory, bounding box offset directions, and pose, for a con-stant number of boxes. These are filtered by non-max sup-pression to produce the final output. The network is a vari-ant of the single shot detection (SSD) network from [10]with additional outputs for pose. Here we present the net-work’s design choices, structure of the outputs, and training.

An SSD-style detector [10] works by adding a sequenceof feature maps of progressively decreasing spatial resolu-tion to an image classification network such as VGG [17].These feature layers replace the last few layers of the imageclassification network, and 3x3 and 1x1 convolutional fil-ters are used to transform one feature map to the next alongwith max-pooling. See Fig. 3 for a depiction of the model.

Predictions for a regularly spaced set of possible detec-tions are computed by applying a collection of 3x3 filtersto channels in one of the feature layers. Each 3x3 filterproduces one value at each location, where the outputs areeither classification scores, localization offsets, and, in ourcase, discretized pose predictions for the object (if any) in abox. See Fig. 1. Note that different sized detections are pro-duced by different feature layers instead of taking the moretraditional approach of resizing the input image or predict-ing different sized detections from a single feature layer.

We take one of two different approaches for pose predic-tions, either sharing outputs for pose across all the objectcategories (share) or having separate pose outputs for eachobject category (separate). One output is added for each of

N

✓

possible poses. With N

c

categories of objects, there areN

c

⇥ N

✓

pose outputs for the separate model and N

✓

poseoutputs for the share model. While we do add a 3x3 filter foreach of the pose outputs, this added cost is relatively smalland the original SSD pipeline is quite fast, so the result isstill faster than two stage approaches that rely on a (oftenslower) detector followed by a separate pose classificationstage. See Fig. 2 (a).

3.1. Pose Estimation Formulation

There are a number of design choices for a joint detectionand pose estimation method. This section details three par-ticular design choices, and Sec. 4.1.1 shows justificationsfor them through experimental results.

One important choice is in how the pose estimation taskis formulated. A possibility is to train for continuous poseestimation and formulate the problem as a regression. How-ever, in this work we discretize the pose space into N

✓

dis-joint bins and formulate the task as a classification problem.Doing so not only makes the task feasible (since both thequantity and consistency of pose labels is not high enoughfor continuous pose estimation), but also allows us to mea-sure the confidence of our pose prediction. Furthermore,discrete pose estimation still presents a very challengingproblem.

Another design choice is whether to predict poses sepa-rately for the N

c

object classes or to use the same weights topredict poses for all classes. Sec. 4.1.1 assess these options.

The final design choice is the resolution of the input im-age. Specifically, we consider two resolutions for input:

[Poirson et al, coming out at 3DV, 2016]

Future Work

• Object detection + pose estimation

Figure 2. Two-stage vs. Proposed. (a) The two-stage approach separates the detection and pose estimation steps. After object detection,the detected objects are cropped and then processed by a separate network for pose estimation. This requires resampling the image at leastthree times: once for region proposals, once for detection, and once for pose estimation. (b) The proposed method, in contrast, requires noresampling of the image and instead relies on convolutions for detecting the object and its pose in a single forward pass. This offers a largespeed up because the image is not resampled, and computation for detection and pose estimation is shared.

3. ModelFor an input RGB image, a single evaluation of the

model network is performed and produces scores for cat-egory, bounding box offset directions, and pose, for a con-stant number of boxes. These are filtered by non-max sup-pression to produce the final output. The network is a vari-ant of the single shot detection (SSD) network from [10]with additional outputs for pose. Here we present the net-work’s design choices, structure of the outputs, and training.

An SSD-style detector [10] works by adding a sequenceof feature maps of progressively decreasing spatial resolu-tion to an image classification network such as VGG [17].These feature layers replace the last few layers of the imageclassification network, and 3x3 and 1x1 convolutional fil-ters are used to transform one feature map to the next alongwith max-pooling. See Fig. 3 for a depiction of the model.

Predictions for a regularly spaced set of possible detec-tions are computed by applying a collection of 3x3 filtersto channels in one of the feature layers. Each 3x3 filterproduces one value at each location, where the outputs areeither classification scores, localization offsets, and, in ourcase, discretized pose predictions for the object (if any) in abox. See Fig. 1. Note that different sized detections are pro-duced by different feature layers instead of taking the moretraditional approach of resizing the input image or predict-ing different sized detections from a single feature layer.

We take one of two different approaches for pose predic-tions, either sharing outputs for pose across all the objectcategories (share) or having separate pose outputs for eachobject category (separate). One output is added for each of

N

✓

possible poses. With N

c

categories of objects, there areN

c

⇥ N

✓

pose outputs for the separate model and N

✓

poseoutputs for the share model. While we do add a 3x3 filter foreach of the pose outputs, this added cost is relatively smalland the original SSD pipeline is quite fast, so the result isstill faster than two stage approaches that rely on a (oftenslower) detector followed by a separate pose classificationstage. See Fig. 2 (a).

3.1. Pose Estimation Formulation

There are a number of design choices for a joint detectionand pose estimation method. This section details three par-ticular design choices, and Sec. 4.1.1 shows justificationsfor them through experimental results.

One important choice is in how the pose estimation taskis formulated. A possibility is to train for continuous poseestimation and formulate the problem as a regression. How-ever, in this work we discretize the pose space into N

✓

dis-joint bins and formulate the task as a classification problem.Doing so not only makes the task feasible (since both thequantity and consistency of pose labels is not high enoughfor continuous pose estimation), but also allows us to mea-sure the confidence of our pose prediction. Furthermore,discrete pose estimation still presents a very challengingproblem.

Another design choice is whether to predict poses sepa-rately for the N

c

object classes or to use the same weights topredict poses for all classes. Sec. 4.1.1 assess these options.

The final design choice is the resolution of the input im-age. Specifically, we consider two resolutions for input:

[Poirson et al, coming out at 3DV, 2016]

Future Work

• Single shot 3D bounding box detection

• Object detection + pose estimation

Figure 2. Two-stage vs. Proposed. (a) The two-stage approach separates the detection and pose estimation steps. After object detection,the detected objects are cropped and then processed by a separate network for pose estimation. This requires resampling the image at leastthree times: once for region proposals, once for detection, and once for pose estimation. (b) The proposed method, in contrast, requires noresampling of the image and instead relies on convolutions for detecting the object and its pose in a single forward pass. This offers a largespeed up because the image is not resampled, and computation for detection and pose estimation is shared.

3. ModelFor an input RGB image, a single evaluation of the

model network is performed and produces scores for cat-egory, bounding box offset directions, and pose, for a con-stant number of boxes. These are filtered by non-max sup-pression to produce the final output. The network is a vari-ant of the single shot detection (SSD) network from [10]with additional outputs for pose. Here we present the net-work’s design choices, structure of the outputs, and training.

An SSD-style detector [10] works by adding a sequenceof feature maps of progressively decreasing spatial resolu-tion to an image classification network such as VGG [17].These feature layers replace the last few layers of the imageclassification network, and 3x3 and 1x1 convolutional fil-ters are used to transform one feature map to the next alongwith max-pooling. See Fig. 3 for a depiction of the model.

Predictions for a regularly spaced set of possible detec-tions are computed by applying a collection of 3x3 filtersto channels in one of the feature layers. Each 3x3 filterproduces one value at each location, where the outputs areeither classification scores, localization offsets, and, in ourcase, discretized pose predictions for the object (if any) in abox. See Fig. 1. Note that different sized detections are pro-duced by different feature layers instead of taking the moretraditional approach of resizing the input image or predict-ing different sized detections from a single feature layer.

We take one of two different approaches for pose predic-tions, either sharing outputs for pose across all the objectcategories (share) or having separate pose outputs for eachobject category (separate). One output is added for each of

N

✓

possible poses. With N

c

categories of objects, there areN

c

⇥ N

✓

pose outputs for the separate model and N

✓

poseoutputs for the share model. While we do add a 3x3 filter foreach of the pose outputs, this added cost is relatively smalland the original SSD pipeline is quite fast, so the result isstill faster than two stage approaches that rely on a (oftenslower) detector followed by a separate pose classificationstage. See Fig. 2 (a).

3.1. Pose Estimation Formulation

There are a number of design choices for a joint detectionand pose estimation method. This section details three par-ticular design choices, and Sec. 4.1.1 shows justificationsfor them through experimental results.

One important choice is in how the pose estimation taskis formulated. A possibility is to train for continuous poseestimation and formulate the problem as a regression. How-ever, in this work we discretize the pose space into N

✓

dis-joint bins and formulate the task as a classification problem.Doing so not only makes the task feasible (since both thequantity and consistency of pose labels is not high enoughfor continuous pose estimation), but also allows us to mea-sure the confidence of our pose prediction. Furthermore,discrete pose estimation still presents a very challengingproblem.

Another design choice is whether to predict poses sepa-rately for the N

c

object classes or to use the same weights topredict poses for all classes. Sec. 4.1.1 assess these options.

The final design choice is the resolution of the input im-age. Specifically, we consider two resolutions for input:

[Poirson et al, coming out at 3DV, 2016]

Future Work

• Single shot 3D bounding box detection

• Joint object detection + tracking model

Check out the code/models

https://github.com/weiliu89/caffe/tree/ssd

Thank you! Come by our poster O-1A-02