Instance Segmentation

Min Bai

University of Toronto

April 4, 2016

Min Bai (UofT) Instance Segmentation April 4, 2016 1 / 21

Segmentation Review

Task in computer vision

Assign object class label to each pixel in image

Source: Taegyu Lim

Min Bai (UofT) Instance Segmentation April 4, 2016 2 / 21

What is instance segmentation?

Problem:

How many cows are there?How many cars are there?

Source: Taegyu Lim

Min Bai (UofT) Instance Segmentation April 4, 2016 3 / 21

Why do we care?

Richer information about world

Object localization, tracking

Interactions with objects

Source: Japan TimesMin Bai (UofT) Instance Segmentation April 4, 2016 4 / 21

Current Approaches

Convolutional Neural Networks + Conditional Random Fields fordepth ordering

Z. Zhang et al, Monocular Object Instance Segmentation and DepthOrdering with CNNs (ICCV 2015)Z. Zhang et al, Instance-Level Segmentation with Deep DenselyConnected MRFs (CVPR 2016)

Recurrent Neural Networks

Parades et al, Recurrent Instance Segmentation

Min Bai (UofT) Instance Segmentation April 4, 2016 5 / 21

Monocular Object Instance Segmentation and Depth

Ordering with CNNs

Instance Segmentation and Depth Ordering Network

MRF for Patch Merging

Reasons about a globally consistent depth ordering instances

Min Bai (UofT) Instance Segmentation April 4, 2016 6 / 21

Monocular Object Instance Segmentation and Depth

Ordering with CNNs

Instance Segmentation and Depth Ordering Network

Source: Zhang et al

Min Bai (UofT) Instance Segmentation April 4, 2016 7 / 21

MRF Patch Merging

Total energy function to be minimized

Source: Zhang et al

Min Bai (UofT) Instance Segmentation April 4, 2016 8 / 21

MRF Patch Merging

Total energy function to be minimized

First term: Global ordering should always ≥ ordering within patch

seen by CNN

Min Bai (UofT) Instance Segmentation April 4, 2016 9 / 21

MRF Patch Merging

Total energy function to be minimized

Second term: Connected components are ordered vertically. Depth

label should ≥ vertical ordering

Min Bai (UofT) Instance Segmentation April 4, 2016 10 / 21

MRF Patch Merging

Total energy function to be minimized

Third term: depth labeling for pixels belonging to di�erent connected

components should be di�erent

Sparse, random connectivity

Min Bai (UofT) Instance Segmentation April 4, 2016 11 / 21

MRF Patch Merging

Total energy function to be minimized

Fourth term: depth labeling of neighboring pixels in same connected

component should be the same

Min Bai (UofT) Instance Segmentation April 4, 2016 12 / 21

MRF Patch Merging

Total energy function to be minimized

Minimized via quadratic pseudo-boolean optimization and graph cut

Min Bai (UofT) Instance Segmentation April 4, 2016 13 / 21

MRF Patch Merging

Results - Left: input, middle: ground truth, right: result

Source: Zhang et al

Min Bai (UofT) Instance Segmentation April 4, 2016 14 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Extends previous project

Instead of locally connected MRF, uses fully connected MRF within

each patch and between connected components of di�erent patches

Source: Zhang et alMin Bai (UofT) Instance Segmentation April 4, 2016 15 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Similar instance segmentation and depth ordering network

Densely connected MRF

Does not reason about depth ordering - only instance identitiesLonger range smoothnessInnovative method for MRF energy minimization

Min Bai (UofT) Instance Segmentation April 4, 2016 16 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Complete MRF energy term

Source: Zhang et al

Min Bai (UofT) Instance Segmentation April 4, 2016 17 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Complete MRF energy term

First term: encourages smoothness of instance label assignments

Min Bai (UofT) Instance Segmentation April 4, 2016 18 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Complete MRF energy term

Second term: encourages global instance assignments of pixels to besame if CNN assigns them to be the same, and di�erent otherwise

Min Bai (UofT) Instance Segmentation April 4, 2016 19 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Complete MRF energy term

Third term: encourages assignments of labels to pixels belonging todi�erent inter-connected-components to be di�erent

Min Bai (UofT) Instance Segmentation April 4, 2016 20 / 21

Instance-Level Segmentation with Deep Densely Connected

MRFs, Zhang et al

Results, from left to right: input, ground truth, previous paper, this

paper

Source: Zhang et al

Min Bai (UofT) Instance Segmentation April 4, 2016 21 / 21