Smarter Humanoid

Companion

Embedded GPUs can make your robotic companion

more alive

Alexandre Mazel Innovation Software Director

Oct 2017

Hi Alex!

Agenda

● Overview○ Innovation Team Presentation○ Mummer Research Project

● Problematics

● Proposed solution

● Live Demo

● Question

Team Presentation

Part of the Innovation Department, which includes hardware,

electronics, collaborative projects and design.

AI Lab

Fundamental Research on Developmental Robotics

3 Permanent3 PhD student

3 Intern

Protolab

Applied Research

4 Permanent1 PhD student

Innovation Software

Goal: Prospection

Enhance our Humanoid Robots for more natural Human-Robot Interaction

● Explore future uses ● Test and embed new algorithms● Hardware improvement● Provide versatile platforms for research

Goals:● Make Pepper navigates in malls● Entertain visitors/customers

Experimentation field: ● Ideapark mall in Lempäälä (Finland)● Huge: More than 150 stores, restaurants and cafes

within 100.000 m2● Crowdy: 7 million visitors (2013)

SoftBank Robotics Europe

VTT TechnicalResearch Center of Finland

Ideapark

University of Glasgow

Heriot-Watt University

Idiap Research Institute

LAAS-CNRS

Challenges

● Obstacle avoidance

● Quick person detection (<1s)

● Self Localization

● Data confidentiality

Navigation Sensors

Laser (45 points, up to 3m)

RGB Camera (55°H, 44°V)

Sonar

Depth Camera (58°H, 45°V)RGB Camera (55°H, 44°V)

Current limitations

Current limitations

Proposed solution: case study

RGB Fish Eye (100°H, 180°V)

Robot POV

ConvNet Learning for obstacle avoidance

● pretrained AlexNet using the LSVRC-2010 ImageNet (1.3M Images)

● learning FC7, FC8 and binary classification

Passable

Non

passable

Collection of training data

Region of Interest

Results of the learning process

● 3400 images

● Learning rate: 0.001

● dropout rate: 0.5

● batches of size: 40

● duration of one epoch: 70 sec (using a Geforce GTX 1070)

Proposed solution: case study

RGB Fish Eye (100°H, 180°V)

JetsonTM TX2

DC-DC

29V-12V

Embedding JetsonTM TX2

USBEthernet

(or wifi)

Video Technical Demo

NB: Using the trained tensorflow model as is

Action Time (s)

Acquire image 0.016

Computing difference 0.005

Undistort and rotation (numpy) 0.049

Inference 0.033

Total Time 0.103 (9.7 fps)

Embedding JetsonTM TX2

Battery Draining Measure

NB: based on one test only

Standard Pepper - no movement 11h32

Pepper with gpu processing and infering every

frame - no movement10h21

Advantages

● Dodges obstacles

● Fully autonomous (no cloud, no wifi)

● Quick training - can be done multiple times

● Can be learned directly on site

● Confidentiality is preserved

To be continued

Next steps:

● add more classes (left/right/center)

● optimisation (int8, tensorRT, …)

● autonomous & continuous learning on the fly

Future work:

● Navigation: Localisation/VSlam

● Skeleton estimation (2D) (Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh)

● Face features extraction

● Speech Recognition (Caldi)

Live Demo

Conclusion

Acknowledgement

Based on work from:

● Abdelhak Loukkal (2017)

● Michael Guerzhoy and Davi Frossard (2016)

Reference:

● Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks (2015)

Questions Time

More questions:

Alexandre Mazel

amazel@softbankrobotics.com