6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using

a Monocular Camera

Authors:

Carlos Jaramillo[a]

Ivan Dryanovski[a]

Roberto Valenti[b]

Jizhong Xiao[b]

Presenter: Dr. Jizhong Xiao

City University of New York

The Graduate Center[a] and

The City College of New York[b]

Presentation Outline

1. Problemdescription

2. Existingapproaches

a) MonocularSLAM

b) RGB-DSLAM

3. Proposedmethod

a) Initialposeestimation

b) System’spipeline

4. Results

a) Experiments

b) Performance

5. Futurework

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 2

1. Problem Description

GOAL:6-degree-of-freedom(6-DoF)poselocalization

bysimplyusingamonocularcamera

insidea3Dpoint-clouddensemap

“prebuilt”withdepthsensors

(e.g.,RGB-Dsensor,laserscanner,etc.)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 3

+

+

1. Problem Description

APPLICATIONEXAMPLES:unconstrainedmotionof

monocularcamerassuchasinsmartphonesormountedin

smallrobots

• Augmentedreality

– Showcases

– Games

–Museumtours

• Mobilerobotnavigation

– Swarmnavigation(SearchandRescue)

1. Aleaderequippedwithpowerfulsensor(s)createsamap

2. Followers(withsimplecameras)localizethemselvesinmap

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 4

http://augmentedpixels.com

Jaramillo’sDREU2009

2. Existing approaches

VisualSLAM:VisualSimultaneous Localization andMapping

a) MonocularSLAM

– MonoSLAM

» [2007,Davisonet.al.]

– PTAM(ParallelTrackingandMapping)

» [2007,Williamset.al.]

– Structurefrommotion(Sfm)

» [1981,Longuet-Higgins]

b) RGB-DSLAM

- Visual3DSLAM

- [2011,Engelhardet.al.]

- Fast3DMapping+VisualOdometry

- [2013,Dryanovski et.al.]

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 5

Resourceintensive:

Needtokeepa

historyoffeatures

inthemap

3. Proposed method

1. Userinitiallymapsoutthescene

(3Ddensepointcloud)

– Avoidsresource-intensive

VisualSLAMtechniques

2. Ourlocalizationmethod:

– Usesdense point-cloud(map)

– Usessingleimagesfroma

monocularcamera

– Wedon’ttrackpoints

– Wegeneratevirtualimages

(usingpreviouspose)ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 6

MONOCULARLOCALIZATIONWITHINA3DMAP

3. Proposed method

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 7

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

Initialposeestimation(firsttimeonly!):

1. Inthefirstinputimage,I1,wedetectSURF.Also,extractSURFfromallthemap’s frameimages.

2. WetrainadescriptormatcherfromalltheSURFfeatures.

3. Foreachfeatureintherealimage,wefindn nearestfeatureneighborsusingthematcher.

4. EachfeatureinI1 maypointtoavectorofdescriptormatches.Wetakethetopn candidates

5. TheinitialposeisfoundfromarobustPnPmatchingbetweenthen pointsfromtherealimage

andtheircorresponding3Dpointsinthemapobtainedfromthetopnmatches.

3. Proposed method

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 8

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

3. Proposed method

1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 9

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

3. Proposed method

1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.

2) 2Dfeaturesarematchedbetweentherealandvirtualimages.

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 10

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

3. Proposed method

1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.

2) 2Dfeaturesarematchedbetweentherealandvirtualimages.

3) 2D-to-3Dpointcorrespondencesareobtainedbetweentherealcamera’s2Dfeaturesand

associated3Dpointsinthemap.

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 11

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

3. Proposed method

1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.

2) 2Dfeaturesarematchedbetweentherealandvirtualimages.

3) 2D-to-3Dpointcorrespondencesareobtainedbetweentherealcamera’s2Dfeaturesand

associated3Dpointsinthemap.

4) AfterPerspective-n-Point(PnP)+RANSAC,therelative6-DoFtransformationbetweenthereal

andvirtualcamerasisfound.

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 12

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

3. Proposed method

1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.

2) 2Dfeaturesarematchedbetweentherealandvirtualimages.

3) 2D-to-3Dpointcorrespondencesareobtainedbetweentherealcamera’s2Dfeaturesand

associated3Dpointsinthemap.

4) AfterPerspective-n-Point(PnP)+RANSAC,therelative6-DoFtransformationbetweenthereal

andvirtualcamerasisfound.

5) Afinalframetransformationlocalizesthe6-DoFposeofthecamerawithrespecttothemap.

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 13

[World]

CamaxisC

MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)

4. Results

Baby’sroomexample(1)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 14

4. Results

Baby’sroomexample(2)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 15

4. Results

Baby’sroomexample(3)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 16

4. Results

Baby’sroomexample(4)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 17

4. Results

Baby’sroomexample(5)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 18

4. Results

Officeroomexample(Video)

ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 19

4. Results

• AtQVGA resolution(320x240pixels),theworst-case

executiontimesrunningona1.7GHzIntelCorei5

processor(insideavirtualmachine)were:

• Bearinmindthatthesetimevaluesincludethe

visualizationoverheadofthe3Dmapandtheimages.

• Intheworstcase,itcanprocess3FPSROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 20

Process(Perimageframe) Worst-casetime(ms)

VirtualImageGeneration 70

SURFfeaturedetectionanddescription 100

SURFFeaturematchingwithFLANN 8

PnPwithRANSAC

(1000iters,50inliers,10px reprj.error)

200

Total 378

5. Discussion & Future Work

1. Computingtheinitialposeofthecameraaddsaninitialdelaybefore

theliveimage-feedcanenterthepipeline.

2. Wemustimprovequalityofthevirtualimages

– Affectsthefeaturecorrespondenceprocedure.

3. Improvequalityof3Dmaps

– Virtualimagesdependonmodeldensity(Trymeshedmodels)

4. Wehavetovalidateourmethodbyexperimentingwithbiggermaps

5. Wehavetoperformingerroranalysiswithgroundtruthdatasets.

– Existingdatasetsdon’tproducedensemaps

6. Otherenhancements:

1. AidtherotationestimationwithIMUsensors(phoneshaveit)

2. Usewiderfield-of-viewreal(andvirtual)imagesinorderto

toleratedrasticmotion.

3. Supportdynamicenvironments(onlystaticenvironmentstoday).ROBIO2013 226-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera

Thank you!

6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCameraROBIO2013 23

Jaramillo,

Carlos

Dryanovski,

Ivan

Valenti,

Roberto

Xiao,

Jizhong