1
Lakmal Seneviratne
Professor of Mechatronics Professor of Mechanical Eng.
Kings College London Khalifa Univeristy, Abu Dhabi.
Tactile Interactions During Robot Assisted
Surgical Interventions
2
Overview
1. Surgical Robotics – Soft Robots
2. Tactile Interactions - Learning
3. MRI Compatible Force Sensing
4. Haptic Interfaces
5. In Hand Manipulations
4
Khalifa University
• Established in 2009
• Vision - To be a leading international
center of higher education and research
7
Surgical Robotics – Robotic Catheterisation
Hansen Medical: Robot-steered catheterization
tool for cardiac ablation procedures.
8
• Tools enter the body through narrow openings and manipulate soft organs that can move, deform, or change in stiffness.
• Teleoperated
• Restricted access (through Trocar ports),
minimal haptic feedback,
rigid robot tools,
confined space,
safety-critical, real-time, bio-compatible, sterilized tools, MRI compatible.
MIS
11
STIFF-FLOP
• Biological inspiration taken from octopus
• Soft, highly redundant manipulation device,
Embedded distributed sensing
cognitive development and intelligent control
Learning and cognitive reasoning - learn from physical interactions with environment,
15
Tissue Palpation
Ex-Vivo Test Rig for Indentation
Ex-vivo indentation tests on bovine liver
- Measure Force Vs Displacement characteristics
16
Bovine Liver Static indentation (6mm probe)
Dual Maxwell Model for Palpation
Talal M. Al-ja'afreh, Yahya H. Zweiri, Lakmal D. Seneviratne, and Kaspar Althoefer, A New Soft Tissue Indentation Model for Estimating
Force-Displacement' Characteristics using Circular Indenters. Proc IMechE, 2008.
17
Rolling Palpations
Fz (Normal Force)
Fx (Drawbar Pull)
Force/Torque Sensor
Horizontal Motion
Vertical
Motion
6-DOF robotic
Manipulator
WheelTissue
Ex-Vivo Test Rig (Rolling Device)
18
Rolling Palpations – Soft Tissue
•Test results on pork kidney embedded with simulated tumor
•Kidney sample with an embedded nodule
Liu, H, Noonan, D. P., Challacombe, B. J., Dasgupta, P., Seneviratne, L. D., Althoefer*, K, Rolling Mechanical Imaging for Tissue Abnormality Localization During Minimally Invasive Surgery, IEEE Transactions on Biomedical Engineering, 2010.
23
•23
K. Sangpradit, H. Liu, P. Dasgupta, K. Althoefer, and L. Seneviratne, “Finite Element Modelling of Rolling Indentation” IEEE
Transactions on Biomedical Engineering, 2011.
• 3 nodules, 10 mm diameter
- Arruda-Boyce model. FE model parameters from unixial tests.
• FE results are in good agreement with the corresponding experimental data, RMS range 0.02-0.15 %
Material μ ,shear Modulus (kPa)
λm, Locking stretch
Mass Density (kg/m3)
Type of
mesh
Rubber (N1) 73.40 1.01 1000 CPS4R
Silicone (RTV6166 gel)
4.98 1.05 980
CPS4R
25
Fibre Optic Uni-Axial Force Sensor
P. Puangmali, H. Liu, K. Althoefer, and L. D. Seneviratne, “Optical Fibre Sensor for Soft Tissue Investigation during Minimally Invasive Surgery”, ICRA 2008.
26
Fibre Optic 3 Axis Force Sensor
3-Axis Force Sensor
P Puangmali, H Liu, L D Seneviratne, P Dasgupta, K Althoefer. Miniature 3-Axis Distal Force Sensor for Minimally Invasive Surgical
Palpation. IEEE/ASME Transactions on Mechatronics. 2011,
27
The miniaturized sensor (11 mm diameter) consists of a force sensor and four displacement sensors
Fibre optic technique is applied , MRI-compatible
Force sensor
Displacement sensor
Fibre Optic Stiffness Sensor
Panagiotis Polygerinos, Lakmal D. Seneviratne, and Kaspar Althoefer, Triaxial Catheter-Tip Force Sensor for MRI-Guided Cardiac Procedures,
IEEE/ASME Transactions on Mechatronics. 2012
28
Calibration results of fibre-optic
force sensor
Force Sensing
(a) (b)
(c)
Indentation Depth sensing (different Orientations)
Silicone
Mini FID θz
Fibre Optic Stiffness Sensor
Hongbin Liu, Jichun Li, Xiaojing Song, Lakmal Seneviratne, Kaspar Althoefer. "Rolling Indentation Probe for Tissue Abnormality
Identification during Minimally Invasive Surgery", IEEE Transactions Robotics. 2011.
.
29
Fibre Optic Force Sensing for Cardiac Catheters
Wei Yao, Tobias Schaeffter, L Seneviratne. K Althoefer,
MR-compatible Catheter for Cardiac Catheterization,
ASME Journal of Medical Devices, 2012
30
3-Axis Catheter Force Sensor
Fibre-optic catheter-tip force sensor.
12Fr catheter-tip integrated with tri-axial force sensor.
Panagiotis Polygerinos, Asghar Ataollahi, Tobias Schaeffter, Reza Razavi, Lakmal D. Seneviratne, and Kaspar Althoefer.
MRI-Compatible Intensity-Modulated Force Sensor for Cardiac Catheterization Procedures. IEEE Transactions on Biomedical
Engineering, 58 (3), pp. 721-726. 2011.
31
• Constant pressure on 9mm
Sphere.
• Displacement of the ball
indicates a change in stiffness
of the surface tissue.
Airflow Force Sensor
32
Indika Wanninayake, Lakmal Seneviratne, Kaspar Althoefer, Novel Indentation Depth Measuring System for Stiffness Characterization in Soft Tissue Palpation. IEEE ICRA 2012
– Direct stiffness indication
– Tuneable force range (depend on the inlet air pressure)
– Friction free rolling over the tissue
– Array of Tactile Element (sphere diameter – 4mm)
– Sensitivity -0.008, diameter 18mm
Air Flow Tactile Probe
34
The location of the 3 nodules on in the silicone phantom correspond to
peaks.
Airflow Force Sensor – Experimental Evaluation
35
Airflow Force Sensor
• Low friction sensor motion
• Simple design with potential for miniaturisation
• Can be built from MR-compatible materials
36
Pseudo-Haptic Feedback
Force matrices of different indentation depths
Palpation input device
Robot arm
Rolling probe Force
sensor
Soft tissue
•Real soft tissue
•Virtual soft tissue
•Tissue
properties
•Haptic
feedback
•RMIS system
• Rolling
indentation probe
•Tissue
properties
38
Palpation with Pseudo-Haptic Feedback
Cursor Real position
Tumor
Virtual force
v
Soft tissue
x
y
z
39
Pseudo-Haptic Feedback
• (a) cursor speed (b) cursor size
Min Li, Lakmal Seneviratne, Kaspar Althoefer. Tissue Stiffness Simulation and Abnormality
Localization using Pseudo-Haptic Feedback. IEEE ICRA 2012
40
HANDLE – EU FP7 IP UPMC (France), Shadow (UK), UC3M (Spain), FCTUC (Portugal), KCL (UK) ORU (Sweden), UHAM (Germany), CEA (France), IST (Portugal)
Project Objectives
• Characterization of object affordances • Learning and imitation of human strategies in handling tasks • Improving skills through 'babbling' • Autonomous in-hand dextrous manipulation
41
Learning Through Touch
Hongbin Liu, Lakmal Seneviratne, Kaspar Althoefer. Real-Time Local Contact Shape and
Pose Classification using a Tactile Array Sensor. IEEE ICRA 2012
42
Learning Through Touch
Right angle edge
λ1=121.9
λ2=22.5
λ3=7.5
flat surface
λ1=123.6
λ2=48.8
λ3=3.7
concave surface
(r = 45 mm)
λ1=123.6
λ2=53.3
λ3=7.5
sphere
(r = 6 mm)
λ1=28.7
λ2=18.1
λ3=15.6
vertex (corner)
λ1=30.1
λ2=8.45
λ3=6.23
cylinder
(r = 6 mm)
λ1=50.7
λ2=20.6
λ3=20.5
square
(a = 8 mm)
λ1=35.1
λ2=16.7
λ3=15.7
ring
(router= 10 mm,
rinner= 4 mm )
λ1=42.2
λ2=36.3
λ3=37.7
e1
e3
e3
e2
e1
e2
e3
e1
e2 e3
e1
e2 e3
e1
e1
e2 e3
e1
e2
e3
e2
e3
e1 e2
2
43
Challenges
Human-Robot system – Human in loop to deal with uncertainties. Monitoring, error recovery
Perception – Multi-modal (Tactile, Vision, etc)
Grasping and Manipulations
Tactile Interactions – Control, learn
Soft systems