3D-Based Reasoning with Blocks, Support, and Stability

transcript

School of Electrical and Computer EngineeringCornell University

Zhaoyin Jia

Computer Vision with RGB-D

Jia, Gallagher, Saxena and Chen

Activity DetectionJ. Sung et al. 2012; H. Koppula et

al. 2013.

Object Recognition K. Lai et al. 2011; A. Janoch et al. 2011

3D Scene Labeling H. Koppula, et al. 2011; N. Silberman et al 2011, 2012.

Pose RecognitionJ. Shotton et al. 2011; G. Girshick et al.

RGB-D Images

3D Reasoning on RGB-D Images Free Space:

objects can be placed in empty spaces.

Physical Stability: one book is

supported by the table and wall.

Foresee Consequences: the camera and the

book will fall if the box moves.

Reasoning with Blocks, Support, & Stability

Input: RGB-D

Segmentation

Input: RGB-D

Blocks, Support, and Stability

Input: RGB-D

Final 3D representation

Algorithms

Overview

3D Block FittingInput Segmentation*

* "Indoor Segmentation and Support Inference from RGBD Images," N. Silberman et al. ECCV, 2012.

Support and Stability

Evaluate Energy

Function

3D Block FittingInput Segmentation

Evaluate Energy

Function

Overview

3D Block Fitting

Single Block Fitting 3D orientated bounding box on depth data Partially observed. Minimum volume may fail * Minimum surface distance (Min-surf)

* "Fast oriented bounding box optimization on the rotation group SO(3, R)," C. Chang et al, ACM Transactions on Graphics, 2011.

Overview

Evaluate Energy

Function

Support Relations

Supporting Area

Stability

Support Relation

Surface On-topSupport

Partial On-topSupport

SideSupport

15 Jia, Gallagher, Saxena and Chen

SideSupport

Separate axis is parallel to y

Separate axis is perpendicular to y

16 Jia, Gallagher, Saxena and Chen

SideSupport

From Support To Stability Supporting Area

Stability

Stable

Stability

Stable Unstable

Overview

Evaluate Energy

Function

Evaluate Energy

Function

Reasoning Through an Energy Function

F(S) =

φ(si)i∑ +

ψ (si,sφ)i, φ∑Segmentatio

n Energy Function

Use Support Relations, Stability, Other Box-based/RGB-D info as features.

RGB-DBetter SegmentationSmaller F(S)

Worse SegmentationLarger F(S)

Energy Function: Single Box Potential

Features: minimum surface distance, visibility, single box stability, etc.

F(S) =

φ(si)i∑ +

ψ (si,sφ)i, φ∑

WorseBox

BetterBox

Energy Function: Pairwise Box Potential

Features: box intersection, support, supporting area distance etc.

F(S ) =

φ(si)i∑ +

ψ (si,sφ)i, φ∑

WorseBoundar

BetterBoundar

Segmentation at one

F(S ) =

φ(si)i∑ +

ψ (si,sφ)i, φ∑

Segmentation Energy Function:

1.2……

……

Summary

Evaluate Energy

Function

Experiments

Experiments: Block dataset

Cornell Support Object dataset (SOD) 300 RGB-D images with ground-truth segments and support

relations

NYU-2 RGB-D datasetJia, Gallagher, Saxena and Chen

Experiment: Segmentation Results Pixel-wise object segmentation

accuracy:

Cornell Dataset

NYU Dataset

ECCV-12’ 60.2% 60.1%Ours 70.0% 61.7%

Input RGB-D images

Experiment: Segmentation Results

Experiments: Support Inference Neighbor: object is

supported by its neighbors Stability: trim unnecessary

support after reasoning

Block Dataset

CornellDataset

Neighbor 80.6% 52.9%Stability 91.7% 72.9%

Blocks world revisited A. Gupta et all, ECCV, 2010.

Indoor Segmentation & Support

N. Silberman et al. ECCV 2012.Jia, Gallagher, Saxena and Chen

Semantic 3D Labeling H. Koppula et. al. NIPS 2011.

Object Placement Y. Jiang et al. IJRR, 2012.

Color SegmentationD. Hoiem et al. ICCV, 2007;

P. Arbelaez et al. CVPR, 2012.……

Conclusion 3D support and stability

Based on box representations

Object segmentation in 3D scene Learning algorithm.

Future work Non-uniform density Semantic classification on blocks Occluded supports

3D-Based Reasoning with Blocks, Support, and StabilityZhaoyin Jia, Andrew Gallagher, Ashutosh Saxena, Tsuhan Chen

Thanks. Questions?

Cornell University

3D-Based Reasoning with Blocks, Support, and Stability

Documents