iCuba shared platform for research in robotics & AI

Woods Hole, MA

Giorgio Metta & the iCub teamIstituto Italiano di TecnologiaVia Morego, 30 - Genoa, Italy

1

we have a dream

© Source Unknown. All rights reserved. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

2

the iCub

price: 250K€30 iCubdistributed since 2008about 3-4 iCub’s/year

© RobotCub Consortium. License GPL v2.0. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

3

why is the iCub special?• hands: we started the design from the hands

– 5 fingers, 9 degrees of freedom, 19 joints

• sensors: human-like, e.g. no lasers– cameras, microphones, gyros, encoders, force, tactile…

• electronics: flexibility for research– custom electronics, small, programmable (DSPs)

• reproducible platform: community designed− reproducible & maintainable yet evolvable platform − large software repository (~2M lines of code)

© RobotCub Consortium. License GPL v2.0. This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

4

why humanoids?• scientific reasons

– e.g. elephants don’t play chess

• natural human-robot interaction

• challenging mechatronics

• fun!

.Image removed due to copyright restrictions

Please see the video.

.Image removed due to copyright restrictions

Please see the video.

Image removed due to copyright restrictions.Please see the video.

.Image removed due to copyright restrictions

Please see the video.

5

why open source?• repeatable experiments

• benchmarking

• quality

this resonates with industry-grade R&D in robotics

.Image removed due to copyright restrictions

Please see the video.

.Image removed due to copyright restrictions

Please see the video.

.Image removed due to copyright restrictions

Please see the video.

6

Yet Another Robot Platform• YARP is an open-source (LGPL)

middleware for humanoid robotics

• history

– an MIT / Univ. of Genoa collaboration

– born on Kismet, grew on COG, under QNX

– with a major overhaul, now used by the iCub project

• C++ source code (some 400K lines)

• IPC & hardware interface

• portable across OSs and developmentplatforms

2000-2001

2001-2002

2003

2004-Today

© Bryan Adams. All rights reserved. Thiscontent is excluded from our CreativeCommons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.

Courtesy of Laboratory forIntegrated Advanced Robotics.Used with permission.

© RobotCub Consortium. License GPL v2.0.This content is excluded from our CreativeCommons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.

Image of Tibi Robot removeddue to copyright restrictions.

Image is in the public domain.

7

exploit diversity: portability• operating system portability:

– Adaptive Communication Environment, C++ OS wrapper: e.g. threads, semaphores, sockets

• development environment portability:– CMake

• language portability:– via Swig: Java (Matlab), Perl, Python, C#

C/C++library

C/C++library

C/C++library

C/C++library

Projectdescription

(.txt)

LINUX:Makefiles,Kdevelopfiles, ...

WINDOWS:MSVC files,Borland files,

...

OSX:Makefiles,

Xcode files, ...

8

wiki & manual

SVN & GIT

part lists

drawings

© RobotCub Consortium. License GPL v2.0. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

9

iCub sensors

© RobotCub Consortium. License GPL v2.0. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

10

the robot skin

crot

apac

i

electrodes: etched on a flexible PCBparameters: shape, folding, etc.

soft material: e.g. siliconeparameters: dielectric constant, mechanical stiffness, etc.

ground plane: e.g. conductive fabricparameters: mechanical properties, impedance, etc.

© Source Unknown. All rights reserved.

This content is excluded from our Creative

Commons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.

11

© Source Unknown. All rights reserved.

This content is excluded from our Creative

Commons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.

© Source Unknown. All rights reserved.

This content is excluded from our Creative

Commons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.

© Source Unknown. All rights reserved.

This content is excluded from our Creative

Commons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.

© RobotCub Consortium. License GPL v2.0. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

12

Fabric, conductive+protective

Inner support for PCB

PCB

fingernailCDC chip

Plastic interface transforming to curved shape

12 sensors14.5 x 13 mm

13

© RobotCub Consortium. License GPL v2.0. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

learning dynamics

Six axis F/T sensor

Inertial sensor

• learning body dynamics– compute external forces– implement compliant control

• so far we did it starting from e.g. the CAD models– but we’d like to avoid it

Courtesy of World Scientific Publishing Company. Used with permission.

14

incremental experiments

15

temperature compensation

16

dataset

Human“Teacher”

MotionDetection

Verbal (weak) Supervision

sprinkler

© Journal of Machine Learning Research. All rights reserved. This content is excluded from our

Creative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

17

methodsLocal Coding-Pooling Coding Pooling

e.g. SIFT, HOG,SURF

e.g. VQ, SC,LLC

e.g. SpatialPyramid

ImageConvolution, NL, Pooling

Mainstream Object Recognition Pipelines:

e.g. SVM, RLS

ClassifierImage

Convolution, NL, Pooling

Convolutional Neural Networks:

Representation

e.g. SVM, RLS

ClassifierRepresentation

© Source Unknown. All rights reserved. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

18

exploiting continuity in time

sprinkler sprinkler sponge sprinkler sprinkler sprinkler

0.5 sec

19

© Source Unknown. All rights reserved. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

incremental learning

• Present: test on current day• Causal: test on current and

past days• Future: test on future days

(current not included)• Independent: train & test on

current day only

Cumulative learning on the 4 days of acquisition. Tested on:

daya

ccu

racy (

avg

. o

ve

r cla

sse

s)

1 2 3 4

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

day

accu

racy

(avg

. ove

r cla

sses

)

OF presentOF causalOF futureOF Independent

20

3D vision for grasping

Input Segmentation Disparity

© Source Unknown. All rights reserved. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

21

force reconstruction

22

© Source Unknown. All rights reserved. This content is excluded from our Creative

Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.

Photo of researchers removed due to copyright restrictions. Please see the video.

23

MIT OpenCourseWarehttps://ocw.mit.edu

Resource: Brains, Minds and Machines Summer CourseTomaso Poggio and Gabriel Kreiman

The following may not correspond to a p articular course on MIT Openprovided by the author as an individual learning resource.

CourseWare, but has been

For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.