Nov 3rd 2013 – IROS 2013 – Cloud Robotics Workshop

Life-long Learning Perceptionusing Cloud Database Technology

Tim Niemueller, Stefan Schiffer, and Gerhard LakemeyerKnowledge-based Systems Group, RWTH Aachen University

Safoura Rezapour-LakaniIntelligent and Interactive Systems, University of Innsbruck

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Motivation

Identify specific objects of interest in sensor range.

Describe objects by attributes instead of classesLearn objects over timeIntegrate new perception approaches over timeAccommodate various sensor modalities

Distributed Robot Perception DatabaseExtend perception system and object database over time

and share it among robots.

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 1 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Motivation

Identify specific objects of interest in sensor range.

Describe objects by attributes instead of classesLearn objects over timeIntegrate new perception approaches over timeAccommodate various sensor modalities

Distributed Robot Perception DatabaseExtend perception system and object database over time

and share it among robots.

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 1 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Perception Classifier Cascades

Find a red apple.

Candidates

apple color:redYes Yes

NoMatches?

NoResult

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 2 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Architecture

Classifiers

Base

Attribute

Meta

Data

Objects

Descriptors

Attributes

Queries

. . .

SURF Color · · · Haar VFH Shape

pepper color:red color:yellow apple . . .

Query: {color : red, apple}

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 3 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Perception Database Requirements

Flexible Data Structuresvarying/evolving data structures

Data Managementunified storage architecturereplication, backup and restore

Flexible and Efficient Retrievalquery for specific datalow-overhead retrieval of diverse and large data

MongoDB, the document-oriented,schema-less database, is particularly

well-suited to fulfill these criteria.

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 4 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Perception Database Requirements

Flexible Data Structuresvarying/evolving data structures

Data Managementunified storage architecturereplication, backup and restore

Flexible and Efficient Retrievalquery for specific datalow-overhead retrieval of diverse and large data

MongoDB, the document-oriented,schema-less database, is particularly

well-suited to fulfill these criteria.

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 4 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

MongoDB and its Building Blocks

Document-orientedGrouped key-value pairs

Schema-lessNo declaration or enforcement ofparticular structure by DB

CollectionsSimilarly structured documentsIndexing reference frame

QueriesJavaScript based query languageSelect based on document fields

{ // attributes/classifiers// for specific object"_id" : ObjectId("50e..."),"data_id" : "apple_1_1_10_c""scene_id" : "apple_1_1_10""attributes" : {"apple" : true,"color" : "red"

}"classifiers" :["SIFT", "SURF", "Gabor","Haar", "Color", "VFH" ...]

}

{ // classifier info excerpt// for attribute doc"_id" : ObjectId("52..."),"data_id" : "apple_1_1_10_c""VFH" : {"model_file" :

"apple_1_1_10_c_vfh.txt""extract_time" : 20

},// [...]

}

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 5 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

MongoDB in our Perception System

DataSharing

Scenes Objects Raw Data

Classifiers

Attributes

Scene

Soda

Apple

Cup

Image, Point

Cloud, ...

SIFT, VFH, ...

attributes:apple, color:red

classifiers:sift, vfh, ...

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 6 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Query Example

Candidates

apple color:redYes Yes

NoMatches?

No Result

Q = {color : red, cup}

{attributes: {

"apple": true,"color": "red"

}}

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 7 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Query Example

Candidates

apple color:red

Yes Yes

NoMatches?

No Result

CDq = D.classifiers,

where q ∈ D.attributes

Cq =⋂

CDq 6=∅

CDq

docs = db.attributes.aggregate({$match: { "attributes.color": "red"}},{$project: {classifiers: 1}},{$group: {_id: "$classifiers"}})

}

set_intersect(docs, "classifiers");

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 7 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Query Example

Candidates

apple color:redYes Yes

NoMatches?

No Result

Oq = Cq(input)

O =⋂

q∈QOq

Apply classifiersFilter through cascade

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 7 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Why a Cloud Database?

Flexible Data StructuresEasily accommodate various data typesQuickly grow w/o tedious specification

Query CapabilitiesFormulate queries deep into data structuresMajor benefit over traditional file system storage

Distributed DataReplicate for off-line trainingQuick boot-strapping for new robots/methodsSharding for multi-host robots

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 8 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Full Retraining

0

50

100

150

200

250

300

350

400

450

500

0 500 1000 1500 2000

Tim

e [m

s]

# of iterations

DB write

DB read

SIFT

SURF

Gabor

Shape

Color

VFH

Cylinder

Sphere

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 9 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Full Retraining

0

50

100

150

200

250

300

350

400

450

500

0 500 1000 1500 2000

Tim

e [m

s]

# of iterations

DB write

DB read

SIFT

SURF

Gabor

Shape

Color

VFH

Cylinder

Sphere

0

50

100

150

200

250

300

350

400

450

500

0 500 1000 1500 2000

Tim

e [m

s]

# of iterations

DB write

DB read

SIFT

SURF

Gabor

Shape

Color

VFH

Cylinder

Sphere

Gabor training magnified 4xTime [ms]

50

DB Read

DB Write

Training

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 9 / 10

Life-long Learning Perception using Cloud Database TechnologyMotivation Perception Database Evaluation Conclusion

Conclusion and Questions

Cloud database for perception which is extensiblein terms of known objects and perception methods.

Attribute-based perceptionIncreasing number of objectsEvolve perception methods

Flexible storage w/ MongoDBCapable query featuresShare data among robots

www.fawkesrobotics.org

Niemueller, Schiffer, Lakemeyer, Lakani Nov 3rd 2013 @ IROS 2013 10 / 10