8/14/2019 rss-main-09

1/28

Milestone 2

Autonomous Door Opening and Plugging inwith a Personal Robot

Sachin Chitta, Eric Berger, Gary Bradski, Tully Foote, Brian Gerkey,Stuart Glaser, John Hsu, Kurt Konolige, Eitan Marder-Eppstein, Bhaskara Marthi,

Conor McGann, Wim Meeussen, Patrick Mihelich, Marius Muja, Vijay Pradeep,Radu Rusu, Melonee Wise, Keenan Wyobrek

8/14/2019 rss-main-09

2/28

About Us

Based in Menlo Park, CA

Focused on open source personal robotics

ROS (Robot Operating System)

PR2 (Personal Robot) Look us up

http://www.willowgarage.com

http://sourceforge.net/projects/personalrobots

http://sourceforge.net/projects/ros

Coming soon

The PR2 (beta)

http://www.willowgarage.com/http://sourceforge.net/projects/personalrobotshttp://sourceforge.net/projects/roshttp://sourceforge.net/projects/roshttp://sourceforge.net/projects/personalrobotshttp://www.willowgarage.com/

8/14/2019 rss-main-09

3/28

About Us

We are open source you can take everything we build

We build robots that can be punished

8/14/2019 rss-main-09

4/28

Goal

Demonstrate the ability to reliably navigate indoors, open doorsand plug into regular outlets

Robustness and reliability of hardware and software platform

26.2 mile marathon

Over multiple days

10 doors and 10 plugs

No failures

8/14/2019 rss-main-09

5/28

The Robot 2 compliant 7DOF arms

counterbalanced

Omnidirectional base, Pan tilt head

Parallel jaw grippers

Fingertip sensors

Sensors

Laser scanners (2)

Base, head

Stereo camera

Projective stereo

Regular cameras

Mounted on head and forearms

Computing

4 Core 2 Duo boxes

8/14/2019 rss-main-09

6/28

Modular Platform

8/14/2019 rss-main-09

7/28

Tasks

Indoor navigation

Planning, sensing, localization and control in crowded 3Dindoor environments

Door opening Deal with doors in different states (open, closed, half-open)

Plugging in

Find regular outlets with no special modifications High-level planning

Robustness to failure

8/14/2019 rss-main-09

8/28

Indoor Navigation

Indoor map

Generated offline using gmappingand logged data

8/14/2019 rss-main-09

9/28

Indoor Navigation

AMCL (Adaptive Monte

Carlo Localization)

Fuses laser scans with

wheel odometry and IMU

information

8/14/2019 rss-main-09

10/28

Control

Goals in map frame, control in odometry frame

map

t = 1 t = 2

localization jump resulting obstacle

odom

t = 1 t = 2

localization jump resulting obstacle

8/14/2019 rss-main-09

11/28

Navigation

Dealing with 3D obstacles- voxel grid

3D is expensive

Create a 2D array of integers- each bit represents a height value

Bresenham ray-trace and mark in the grid using bit-wise & and | operations with

a mask

x, y, z3, 3, 4

x, y

| =

Adding obstacles

& =

Clearing obstacles

0, 0

5, 5

z-mask z-maskselect column

8/14/2019 rss-main-09

12/28

Navigation

Dealing with 3D obstacles- voxel grid

8/14/2019 rss-main-09

13/28

Indoor Navigation

Global Navigation

Grid based planner

Local Navigation

Dynamic window ortrajectory rollout

8/14/2019 rss-main-09

14/28

Navigation Milestone

Reliably traversed 26.2 miles

32 hours of actual robot navigation time

Spread over 4 days

No failures

8/14/2019 rss-main-09

15/28

Doors

ADA compliant

Smaller search space for width of door, height of handle, shape of thehandle

Two tasks Detection

Opening

Deal with half-open, closed, open, locked doors.

8/14/2019 rss-main-09

16/28

Topological Map

Derived from metric map and door info

Divide map into door and openregions

Have a connector between each pair ofregions

Graph over connectors

Double-click to add graphics

8/14/2019 rss-main-09

17/28

Door detection

Laser based

Tilting laser provides 3Dview of the world

Prior location estimatefrom the map

Look for vertical planesthat match shape and sizeof doors

Pick best candidate match

8/14/2019 rss-main-09

18/28

Handle Detection

Laser based + vision based

Door handle detected from imageusing a boosted cascadedclassifier of haar features

Handle locations clusteredacross several frames for a morerobust detection

Stereo information

Used to eliminate false positives basedon position and size priors

Used to compute final handle position

8/14/2019 rss-main-09

19/28

Door Opening

Caging grasp on the handle

Armpushes forwardon the door (nostiff position control)

Base attracted to goal on other side

of door + reactive controller forcollision avoidance

Constraint on shoulder keeps elbowtucked in

8/14/2019 rss-main-09

20/28

Door Opening

Partially open doors

8/14/2019 rss-main-09

21/28

8/14/2019 rss-main-09

22/28

Finding the outlet

Far outlet detection

Outlet candidates obtained by segmentingthe disparity image

Wall pose computed from the base laser

scan List of outlet candidates is filtered using

position, size and orientation priors

Each image patch is rectified to obtain afrontal view of the outlet

Outlet identity confirmed using a templatematching algorithm

8/14/2019 rss-main-09

23/28

Near Outlet detection

Camera

5 Megapixel camera on the head with 16mmfocus

View angle is ~60 degrees, distance is ~1 meter

Aim for less than 1.5mm error in hole

location

Algorithm

Run a hole detector across the image

Look for close pairs of holes

Filter by a single outlet template Group by LePetit's one way descriptor

Compute detailed homography using holes andknown geometry

8/14/2019 rss-main-09

24/28

Near Outlet detection

8/14/2019 rss-main-09

25/28

Plug Localization

Checkerboard target forpose estimation

At 1m from camera,relative distance accurateto ~0.5mm

Region of interest

Centered at gripper position

Hardware ROI gives

consistent lighting from auto-exposure

Large speed boost

8/14/2019 rss-main-09

26/28

Plugging in

Laser based detection of plugon base

Outlet detection

Plug localization Visual differencing to reduce

error between plug and outlet

Search for outlet

Plug in

8/14/2019 rss-main-09

27/28

TREX - Executive

Generative Planning

8/14/2019 rss-main-09

28/28

Conclusion

Milestone 2 was a success

Milestone 3 focuses on you (potential users of the robot)

Have users successfully develop interesting applications onthe robot

Future Work

Doors (opening away from the robot, more handle types,automatically finding doors)

Plugging in (different types of outlets, better accuracy)

Navigation (faster speeds)

Other tasks

Tabletop manipulation, dynamic obstacles, human-robotinteraction, object recognition, grasping, etc. etc.