Welcome to the First Workshop on

RGB-D: Advanced Reasoning with Depth Cameras!

Xiaofeng Ren: Intel Labs SeattleDieter Fox: UW and Intel Labs SeattleKurt Konolige: Willow GarageJana Kosecka: George Mason University

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 2

Workshop Schedule

9:00am - 09:10am Welcome 09:10am - 09:50am Overview and RGB-D Research at Intel Labs and UW

D. Fox and X. Ren; University of Washington, Intel Labs Seattle 09:50am - 10:30am Invited Talk (Vision and Graphics)

C. Theobalt; Max Planck Institute 10:30am - 11:00am Semantic Parsing in Indoor and Outdoor Scenes

J. Kosecka; George Mason University 11:00am - 11:30am Coffee Break 11:30am - 11:50am 3D Pose Estimation, Tracking and Model Learning of Articulated Objects from

Dense Depth Video using Projected Texture Stereo J. Sturm, K. Konolige, C. Stachniss, W. Burgard; Univ. of Freiburg and Willow Garage

11:50am - 12:10pm Learning Deformable Object Models for Mobile Robot Navigation using Depth Cameras and a Manipulation Robot B. Frank, R. Schmedding, C. Stachniss, M. Teschner, W. Burgard; Univ. of Freiburg

12:10pm - 12:30pm 3D Indoor Mapping for Micro-UAVs Using Hybrid Range Finders and Multi-Volume Occupancy Grids W. Morris, I. Dryanovski, J. Xiao; City College of New York

12:30pm - 01:40pm Lunch Break 01:40pm - 02:20pm Invited Talk (Robotics and Vision)

P. Newman; Oxford University 02:20pm - 03:00pm 3D Modeling and Object Recognition at Willow Garage

R. Rusu, K. Konolige; Willow Garage 03:00pm - 04:00pm Poster Session and Wrap-Up

RSS RGB-D Workshop

RGB-D Overview and Work at UW and Intel Labs Seattle

Dieter FoxXiaofeng Ren

Intel Labs Seattle

University of WashingtonDepartment of Computer Science & Engineering

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 4

Outline

RSS RGB-D Workshop

RGB-D: adding depth to color Dense 3D mapping Object recognition and modelingDiscussion

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 5

3D Scanning in Robotics

Panning 2D scanner, Velodyne, time of flight cameras, stereo

Still very expensive, substantial engineering effort, not dense

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 6

RGB-D: Recent Developments Soon we’ll have cheap depth cameras

with high resolution and accuracy (>640x480, 30 Hz)

Key industry drivers: Gaming, entertainment

Two main techniques: Structured light with stereo Time of flight

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 7

Hands Free Gaming

RSS RGB-D Workshop

Microsoft Natal promo video

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 8

RGB-D: Raw Data

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 9

Outline

RSS RGB-D Workshop

RGB-D: adding depth to color Dense 3D mapping Object recognition and modelingDiscussion

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 10

RGB-D Mapping

Visual odometry via frame to frame matching

Loop closure detection via 3D feature matching

Optimization via TORO, SBARSS RGB-D Workshop

[Henry-Herbst- Krainin-Ren-F]

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 11

Kitchen Sequence

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 12

Visual Odometry

Standard point cloud ICP not robust enough Limited FOV, lack of features for data

association Add sparse visual features (SIFT,

Canny edges) Improved data association, might fail in

dark areas

RSS RGB-D Workshop

Point-to-plane Point-to-point Point-to-edge

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 13

RBG-D Mapping Algorithm

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 14

3D Mapping

RSS RGB-D Workshop

Data processing: 4 frames / sec

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 15

Intel Lab Flythrough

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 16

Allen Center Flythrough

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 17

Mapping Accuracy

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 18

Surfel Representation

“Surface Elements” – circular disks representing local surface patches

Introduced by graphics community [Pfister ‘00], [Habbecke ‘07], [Weise ‘09]

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 19

Surfel Representation

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 20

Outline

RSS RGB-D Workshop

RGB-D: adding depth to color Dense 3D mapping Object recognition and modelingDiscussion

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 21

Toward a Robotic Object Database

Enable robots to autonomously learn new objects Robot picks up objects and builds models of them Models can be shared among robots Models can contain meta data

(where to find, how to grasp, material, what to do with it …)

RSS RGB-D Workshop

[Krainin-Henry-Lai-Ren-F]

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 22

Encoders for Object Modeling Commonly used but requires high

accuracy e.g. [Sato ‘97], [Kraft ’08]

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 23

Articulated ICP for Arm Tracking

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 24

Simultaneous Tracking and Object Modeling

RSS RGB-D Workshop

Builds object model on-the-fly Jointly tracks hand and object ICP incorporates dense points, SIFT

features, and color gradients

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 25

Tracking Results

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 26RSS RGB-D Workshop

Object Models

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 27

Handling Multiple Grasps

Switching Kalman filter Examining object Moving to or from table Grasping or releasing Between grasps

Second grasp should be computed from partial model

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 28

Object Recognition

RSS RGB-D Workshop

159 objects31 classes12,554 video frames

Shape based segmentation

[Lai-Bo-Ren-F: RSS-09, IJRR-10]

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 29

Early Results

Learn local distance function for each object

Sparsification via regularization

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 30

Outline

RSS RGB-D Workshop

RGB-D: adding depth to color Dense 3D mapping Object recognition and modelingDiscussion

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 31

Conclusion

New breed of depth camera systems can have substantial impact on mapping (3D, semantic, …) navigation (collision avoidance, 3D path planning) manipulation (grasping, object recognition) human robot interaction (detect humans,

gestures, …)

Currently mostly constrained to indoors, but outdoors possible too

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 32

Some RGB-D Questions

Which problems become easy? Gesture recognition? Grasping? Segmentation? 3D

mapping? Object modeling?

Which problems become (more) tractable? Dense 3D mapping? Object recognition?

What are the new research areas / opportunities generated by RGB-D? Graphics, visualization, tele-presence HRI, activity recognition

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 33

More Questions

What’s the best way to combine shape and color? depth just an additional dimension? interest points, feature descriptors,

segmentation How to take advantage of geometric

info? on top of, next to, supports, …

Is depth always necessary? vision often seems more efficient can we use RGB-D to train fast RGB

systems?

RSS RGB-D Workshop

Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington 34

Some RGB-D Questions

Hardware: What can we expect in the near future?

Real-time dense 3D reconstruction / mapping Representation: planes, meshes, surfels, geometric

primitives, texture, articulation Registration: 3D points vs. visual features

Semantic mapping / object recognition What does 3D add: interest points, feature descriptors,

segmentation, spatial information Humans

Detection, tracking, pose estimation Gesture and activity recognition

RSS RGB-D Workshop

UW Robotics and State Estimation LabIntel Labs SeattleBrian Ferris, Peter Henry, Evan Herbst, Jonathan Ko, Michael Krainin, Kevin Lai, Cynthia Matuszek

Post-docs: Liefeng Bo, Marc Deisenroth

Intel research: Matthai Philipose, Xiaofeng Ren, Josh Smith