LagadicVisual Servoing in Robotics,
Computer Vision, and Augmented Reality
François ChaumetteIRISA / INRIA Rennes
http://www.irisa.fr/lagadic
2
The Lagadic group
Spin-off of the Vista project in January 2004Created as an Inria project in December 2004
Currently 13 people:
François Chaumette, DR 2 Éric Marchand, CR 1, HDR 2004 Alexandre Krupa, CR 2, recruited in Sep. 2004 (LSIIT, Strasbourg) Fabien Spindler, IR 2 1 temporary research scientist: C. Collewet from Cemagref 1 temporary assistant prof.: A. Remazeilles, INSA Rennes 5 Ph. D. students: Master in Rennes (2), Strasbourg (2) and Grenoble (1) 1 post-doc: S. Segvic from Croatia 1 temporary engineer: F. Dionnet from LRP Paris
3
Research field
Visual servoing : vision-based control of a dynamic system
Modeling:
Control law:
Usually, highly nonlinear and coupled potential problems
Objective: cook so that is as linear as possible
4
Objectives Modeling visual features
for usual cameras (perspective projection) for omni-directional cameras for 2D ultrasound images
Considering high level tasks in complex environments Robot navigation Additional constraints (occlusions, joint limits avoidance, etc.)
Visual tracking real-time accurate for 6 dof robust mono-object geometrical structure
5
Application fields Robotics
Manipulating/grasping objects, target tracking Nuclear/submarine/space/medical, etc. Eye-in-hand/eye-to-hand systems Robot arms, mobile robots, UAV
Augmented reality Insert virtual objects in real images
Virtual reality Viewpoint generation Virtual cinematography Control of virtual humanoid
Cogniscience
6
Experimental platforms
Eye-in-hand, eye-to hand systems, mobile robot, medical robot Experimental validation, tests before transfer, demonstrations
Experimental results very time consuming (same image never acquired, and useless after 40 ms)
8
Modeling image moments Determination of the analytical form of
the interaction matrix for any moment Determination of combinations of
moments (from invariants) for decoupling and linearizing properties
with moments
usual choice
9
Visual servoing from ultrasound images Modeling features
No observation outside B-scan corresponding to the current 2D ultrasound image
Automation of spatial calibration procedure Adaptive visual servoing to position B-scan on
a cross-wire phantom
Robotized 3D «free-hand» ultrasound imaging Conventional 2D ultrasound probe moved by a
medical robot Thanks to calibration step, B-Scans positioned
in a 3D reference frame (collaboration with Visages)
Application field: remote examination
10
Navigation from an image database Appearance-based representation
Topological description of the environment with key images
(no 3D reconstruction) Image path retrieval from indexing
techniques (collaboration with Texmex)
Qualitative visual servoing Navigation expressed as visual features
to be seen (and not successive poses to be reached)
Confident interval for features
Automatic update of features used for navigation (by imposing a progress within the visibility corridor)
from to
11
Tasks sequencing Idea : to give as much freedom as possible to take
constraints (joint limits, occlusions, obstacles) into account Scheme more reactive than reactive path planning Scheme more versatile than classical visual servoing
Redundancy framework revisited: directional redundancy non linear projection operator
to increase the free spacewhere secondary tasks are applied
Visual elementary task managed by a stack Remove the good task for ensuring the constraints Put the task back when possible
12
3D model-based tracking Virtual visual servoing scheme for pose computation
Virtually moves a camera so that the projection of the 3D model of the object corresponds to the observed image
Statistically robust pose estimation to deal with outliers and occlusions (M-estimation)
Real-time capabilities
Application to visual servoing and augmented reality Extension to articulated object tracking
13
Texture and contours-based tracking 2D model-based tracking
Estimation of an homography Consider both edges and image intensities
3D model-based tracking Introducing spatio-temporal constraints in model-based tracking Joint estimation of pose and displacement
14
Collaborations Inside Inria : Visages (medical imaging)
Icare (Predit Mobivip, Robea Bodega)
In France : 5 Robea projects Omni-directional vision: Lasmea, Crea, Lirmm Small helicopters: I3S, CEA Mobile robot navigation (Lasmea, UTC)
Outside France : ANU Canberra: modeling, helicopters ISR Lisbon: jacobian learning KTH Stockholm, CSIRO Melbourne, Urbana-Champaign
15
Publications
Main journals : IEEE TRA(O): 6, IJRR: 5 Main conferences: ICRA:18, IROS:14
Best paper award : IEEE TRA 2002, RFIA’2004 Finalist papers : IROS’2004, AMDO’2004, ICRA’2004, IROS’2005
16
Transfert Marker-less: 3D model-based tracker
transferred to Total-Immersion for augmented reality (RIAM SORA)
France Télécom R&D: Augmented reality in urban environment
ESA: vision-based manipulation on the ISS with Eurobot
17
Software ViSP: Open source software environment for visual servoing
Currently available for Linux and Mac OS with QPL license Written in C++ (~ 100 000 lines of code)
Library of canonical vision-based tasks through many visual features
Suitable for 2D, 2½ D, 3D control laws Eye-in-hand / eye-to-hand Redundancy framework
Visual tracking algorithms
Independence wrt. the robotics platform, frame grabber Simulator included (interface with OpenGL)
18
Positioning wrt. INRIA & French labs INRIA scientific and technological challenges:
(4): Coupling models and data to simulate and control complex systems
(5): Combining simulation, visualization and interaction (real-time, augmented reality)
(7): Fully integrating ICST into medical technology (medical imaging, medical robotics)
Inside INRIA: Icare (Num A: Control and complex systems): visual servoing and control Vista, Movi, Isa: visual tracking
Other French labs: LASMEA: visual tracking, position-based visual servoing LSIIT: visual servoing for medical robotics LRP, I3S
19
Worldwide positioning Pioneering lab: CMU (1984 – 1994, no more active) Main labs:
USA: (S. Hutchinson, G. Hager) Australia (P. Corke), Japan (K. Hashimoto) Europe: KTH (more recently)
Other labs : almost everywhere (Italy, Spain, Portugal, Germany, Canada, Mexico, Brazil, South Korea, China, etc.)
Visual tracking: Cambridge, EPFL
Lagadic: High visibility in the robotics community AE IEEE TRA(O) Look for “visual servoing” ∪ “visual servo” in Google Scholar
20
Evolution wrt. past objectives From the 2001 Vista evaluation experts report: “Vista is
planning to split off its activities in visual servoing and active vision as a separate project. This is an excellent decision”
Evolution wrt. scientific objectives: 80 % well done
Complex objects of unknown shape: image moments Outliers: M-estimator integrated in the control loop Applications in robotics: underwater, space, flying robots Applications outside robotics: virtual reality, augmented reality
Visual servoing directly on image intensity: future objective
21
Objectives: modeling visual features Spherical projection:
same model for perspective projection and omni-directional cameras
nice geometrical properties
Modeling directly the image intensity(no image processing, many unknown parameters,
cooking very challenging)
Enclosing volume for 3D objects (global and sufficient information)
Mobile/flying robots: non holonomic or underactuated systems (modeling and control)
22
Objectives: medical robotics Modeling adequate ultrasound features and their interaction Automatic control of the probe motion to assist medical examination
Automatically follow an organ of interest along the patient skin Hybrid force/vision control schemes Remote examination without using haptic device
Robot control combining ultrasound images, force measurement and visual data of the patient provided by a remote camera
Autonomous exploration of a given area (organ, tumor)
control flow
2D trajectory
visual data
2D ultrasound images
site A site B
forcemulti-sensor control
23
Objectives: real-time visual tracking New camera models
Omnidirectional cameras (3D model-based tracking)
Model-based vs model-free approaches Structure estimation
Joint estimation of pose and structure “a la Debevec” Model with some degrees of freedom following the work with
articulated object On line structure estimation during visual servoing
Joint estimation of depth and displacement (controlled SLAM)
Initialization Object detection, recognition and localization Image-based model of the considered object(collaboration with Vista and EPFL through FP6 Pegase proposal)