l b kHaptic PlaybackG.K. Ananthasuresh M h i l E i iMechanical EngineeringIndian Institute of ScienceBangaloreBangalore
Work done withShanthanu ChakravarthyP. MuddukrishnaAvinash KumarAvinash KumarSantosh D. Bhargav
2
Outline
• About rbccps• Haptics and CPS• The contextThe context
– Tele‐operation with force feedback– Micro to macro– Micro to macro– From here to there
• Ha ti layba k• Haptic playback– Endoscopy simulation and playback
G. K. Ananthasuresh, IISc., Mar., 2013
• The details• Main points
3
Robert Bosch CentrefforCyber Physical Systems• A philanthropic grant from the Robert Bosch Foundation
(Rs. 11.5 crores per year for 10 years)• Five verticals
– Agriculture; Buildings; HealthcareTransportation; Water– Transportation; Water
• 20 projects Centre staff 50 faculty Students
• Liaison with industries Internships!
rbccps.iisc.ernet.in [email protected]
Internships!
G. K. Ananthasuresh, IISc., Mar., 2013
Send e‐mail for summer and other internships with “Internship” as the subject of the e‐mail.
4
InvestigatorsCyber Surgery and Remote Patient CareAshitava Ghosal, ME Dept., IISc, BangaloreG K Ananthasuresh, ME Dept., IISc BangaloreV Natarajan, CSA Dept., IISc BangaloreCh d Sh k S l l EE D IIS B lChandra Shekar Seelamantula, EE Dept., IISc BangaloreB Gurumoorthy, CPDM, IISc BangaloreDibakar Sen, CPDM, IISc Bangalore
Dr. Pradeep Rebala, Asian Institute of Gastroenterology, HyderabadDr. D Nageswara Reddy, Asian Institute of Gastroenterology, Hyderabad
Project Staff & StudentsDr. Parama Pal – RBCCPS, IISc BangaloreAditya K – ME Dept., IISc Bangalorey p , gAbin Jose – EE Dept., IISc BangaloreAvinash Kumar – ME Dept., IISc BangaloreSarvesh Kolekar – CPDM, IISc Bangalore
G. K. Ananthasuresh, IISc., Mar., 2013
S. Sagar – CPDM, IISc BangaloreRahul Sharma – CSA Dept., IISc BangaloreShanthanu Chakravarthy – ME Dept., IISc Bangalore
5
Cyber Surgery and Remote Patient Care\ Virtual exploration of objects WP3
oscope
\
ator
ination Virtual exploration of objects
Haptic simulation
WP3
WP4
ble endo
c simula
nt exami
Haptic control WP5
ke flexib
doscop
ic
e pa
tien
Interactive visualization
Heterogeneous material
WP6
Snake‐lik
obot En
d
Remote Heterogeneous material
representation
I i
WP7
G. K. Ananthasuresh, IISc., Mar., 2013
S r Image processing
WP1 WP2 WP9
WP8
6
\oscope
\
ator
ination
SMA‐activated
ble endo
c simula
nt exami
ke flexib
doscop
ic
e pa
tien
Snake‐lik
obot En
d
Remote
Silicone tube with nylon braid
G. K. Ananthasuresh, IISc., Mar., 2013
S r
WP1 WP2 WP9
7
\oscope
\
ator
ination
ble endo
c simula
nt exami
ke flexib
doscop
ic
e pa
tien
Snake‐lik
obot En
d
Remote
G. K. Ananthasuresh, IISc., Mar., 2013
S r
WP1 WP2 WP9
8
\
Patient in the ambulance
oscope
\
ator
ination
ble endo
c simula
nt exami
ke flexib
doscop
ic
e pa
tien
Doctor in the hospital
Snake‐lik
obot En
d
Remote hospital
G. K. Ananthasuresh, IISc., Mar., 2013
S r
WP1 WP2 WP9
9
Virtual exploration of objects WP3Virtual exploration of objects
Haptic simulation
WP3
WP4
Haptic control WP5
Interactive visualization
Heterogeneous material
WP6
Heterogeneous material representation
I i
WP7
G. K. Ananthasuresh, IISc., Mar., 2013
Image processing WP8
10
Virtual exploration of objects WP3Virtual exploration of objects
Haptic simulation
WP3
WP4
Haptic control WP5
Interactive visualization
Heterogeneous material
WP6
Heterogeneous material representation
I i
WP7
G. K. Ananthasuresh, IISc., Mar., 2013
Image processing WP8
11
Virtual exploration of objects WP3Position Virtual exploration of objects
Haptic simulation
WP3
WP4
Position
Haptic control WP5User
Interactive visualization
Heterogeneous material
WP6Haptic Device
Heterogeneous material representation
I i
WP7ForceInput Path
O t t P th
G. K. Ananthasuresh, IISc., Mar., 2013
Image processing WP8Output Path
12
Virtual exploration of objects WP3Virtual exploration of objects
Haptic simulation
WP3
WP4
Haptic control WP5
Interactive visualization
Heterogeneous material
WP6
Heterogeneous material representation
I i
WP7
G. K. Ananthasuresh, IISc., Mar., 2013
Image processing WP8
13
Virtual exploration of objects WP3Virtual exploration of objects
Haptic simulation
WP3
WP4
Haptic control WP5
Interactive visualization
Heterogeneous material
WP6
Heterogeneous material representation
I i
WP7
G. K. Ananthasuresh, IISc., Mar., 2013
Image processing WP8
14
Virtual exploration of objects WP3Virtual exploration of objects
Haptic simulation
WP3
WP4
Haptic control WP5
Interactive visualization
Heterogeneous material
WP6
Heterogeneous material representation
I i
WP7
G. K. Ananthasuresh, IISc., Mar., 2013
Image processing WP8
15
The Centre also has teeny weeny ψφ projects.
Desalination bottle Mobile diagnostic tool Breath sensor Breath sensor Touch‐screen anywhere Granular flow sensorGranular flow sensor Cognitive jewellery
G. K. Ananthasuresh, IISc., Mar., 2013
16
CognitiveCognitive Jewellery
Technology + Aesthetics + FashionN l t i t f ith th h i l ldNovel ways to interface with the physical world.One can wear biomedical sensors in style.
With …Anand P.Pragathi M.Dhruv S.Deepika M. S.
G. K. Ananthasuresh, IISc., Mar., 2013
17
Outline
• Haptics and CPS• The context
– Tele‐operation with force feedbackp– Micro to macro– From here to thereFrom here to there
• Haptic playbackEndoscopy simulation and playback– Endoscopy simulation and playback
• The details
G. K. Ananthasuresh, IISc., Mar., 2013
• Main points
18
Micromanipulation with haptics
I
I
Cell grasped with two contacts for
x xO OI ・ Input ports
actuated by fine‐motion stages
Compliant micro mechanism
Miniature grippers to hold and manipulate cells.Haptic interfaceI
I I
Cell probed with a single contact
contacts for manipulation
x
x O
Light source
Compliant micro mechanism
O ・ Output ports in contact with cellx ・ Observation ports for tracking and force computation
mechanism
x y z
Haptic interface.Mechanical characterization by solving inverse problems in mechanics.
I I
I I
Gross motion stage
Circular motion t
Fine motion stage
Tiltable armm
xyz
x y z
Controller
stage
Microscope
Haptic interfacefor human operator
x
CCD camera
Image processingComputation of forces and displacements
PC
G. K. Ananthasuresh, IISc., Mar., 2013
I/O to controller and haptic interface
19
Micromanipulation station
G. K. Ananthasuresh, IISc., Mar., 2013
20Miniature grippersthat also sense forcesthat also sense forces Reddy, A. N., Sahu, D. K., Maheswari, N., and Ananthasuresh, G. K., “Miniature Compliant Grippers with Vision‐based Force‐sensing,” IEEE Transactions on Robotics, Vol. 26, No. 5, pp. 867‐877, 2010.
Miniature gripper
Zebrafish egg in jaws
G. K. Ananthasuresh, IISc., Mar., 2013Squeezing HeLa cellsGripper small than “1” in the one rupee coin
21
Piercing into a zebrafish egg cell
G. K. Ananthasuresh, IISc., Mar., 2013
22
Master‐slave motion with force‐f db k i ti ttifeedback in tissue‐cutting
G. K. Ananthasuresh, IISc., Mar., 2013
23Setup
Visual
display
Slave device
Compliant cutting toolPh i
p gPhantom tissue
G. K. Ananthasuresh, IISc., Mar., 2013
24
MP 285MP 285
CMOS
camera
G. K. Ananthasuresh, IISc., Mar., 2013
Scalpel
25
Passive control with a compliant end-effector
Output displacement AnchoredAnchored
Rigid-body translation of the
l t (X )
21
plate (Xa) Rigid plate of end-effector casing
Y
X Ensuring cutting force (Fc)Phantom tissue
G. K. Ananthasuresh, IISc., Mar., 2013
26
Phantom tissue with a hard inclusion
Gelatin-based hydrogel
PDMS
G. K. Ananthasuresh, IISc., Mar., 2013
27Comparing rigid end-effector with one that is compliantp
Saturation of the load-cell
G. K. Ananthasuresh, IISc., Mar., 2013
28
Haptic PlaybackHaptic Playback
G. K. Ananthasuresh, IISc., Mar., 2013
29
Vision can be captured by camera , which can be viewed l tlater. Audio can be captured using sound recorder and replayed later using a speaker
What about other senses?Photo courtesy Santosh
Haptic replay
Environment
Force
Position
Haptic replay is difficult because of the duality of force and displacement
G. K. Ananthasuresh, IISc., Mar., 2013
30
Endoscopy PlaybackPlaying back a procedure done in the past…
Snapshot from endoscope image sensorimage sensor
Case for haptic replay Palpation is one of the important modes of diagnosis and in current
endoscopy procedure, no force information is stored for later replay.
G. K. Ananthasuresh, IISc., Mar., 2013
py p p y Important information like stiffness from earlier examination can be
replayed to the doctor to help in diagnosis.
31
Haptic playback with an endoscopy simulator
Step one Step two Step three
Endoscopic session
Step one Step two Step three
Endoscopic session carried out using a modified endoscope.
Rough estimation of tissue properties
User
Data acquisition from the instrumented endoscope
Creating a reality‐based haptic model
Haptic Device
instrumented endoscope haptic model
Doctors can explore the virtual model to recall what was experienced
G. K. Ananthasuresh, IISc., Mar., 2013
earlier.
32
Virtual haptic model
Position
Collision Detection
Object DBC
HA Collision information
SensorO/PData Physical properties
Objectgeometry
AI
3D
Physically based simulation(FEM)
Haptic
Collision information
Actuato
User of Object
D (FEM)
Force
DeviceHaptics Rendering
engine
r I/P Data
Force gInput Path
Output Path“The Haptics Loop”
G. K. Ananthasuresh, IISc., Mar., 2013
Object geometry, haptic robot workspace and graphics screen are synchronized.
33
Force measurement
Modified Falcon grip
Data acquisitionacquisition board
Force sensorForce sensor
G. K. Ananthasuresh, IISc., Mar., 2013
34
Position measurementBoth position and forces have to be measured simultaneously y
x
z
z
( )( ) z
xR y
T t
( )
0 0 0 1
T tz
Rotations at every time instance is measured using
Forward kinematics for position estimation Linear position measurement device
G. K. Ananthasuresh, IISc., Mar., 2013
encoders
35
Rough estimation of tissue ti f lit b dproperties for reality‐based
modelingg
exp1
erimentali i
FEM
FE E
F
S h d
experimental FEMF F
Synthetic data
G. K. Ananthasuresh, IISc., Mar., 2013
36Local elastic modulus
I iti l f EInitial guess for E
from experimentU
Nonlinear Finite Element Method
experimental FEMF F
experimentalF exp1
erimentali i
FEM
E EF
E(U) for the synthetic data
considered
G. K. Ananthasuresh, IISc., Mar., 2013
37
Grid method for collision detection Impose lookup grid inside the bounding box containing thebounding box containing the polygon.
Categorize each cell as being Ca ego i e eac ce as ei gfully inside, fully outside, or indeterminate with respect to the polygon. p yg
For the indeterminate cells use line crossing test to see if the point i i idis inside.
Once collision is detected use pre computed cell data to get the
G. K. Ananthasuresh, IISc., Mar., 2013
pre‐computed cell data to get the nearest node.
Haines, Point in Polygon Strategies, Graphics Gems IV, 1994.
38
Deformation mechanicsDeformation mechanics Problem Definition:
0b in 0 fixedon U
2tr E I E someportion ofon hU U
Haptic DOF
Problem in discretized frame work:
11 12 13
21 22 23 2 2
1 1K K KK K K U F
U F
G. K. Ananthasuresh, IISc., Mar., 2013
31 332 33 3K K K U F
Fixed DOF
39
Lagrange multiplier Method
Optimization Problem Min 1 UTKU - UTF Subj. to : hU=U
Equivalent Problem
Problem MinU 2
U KU U F j h
MinU,
12
UTKU - UTF 12 CU - Uh
U, 2 2
isLagrange multiplier vectorwhere isconstrained matrixC
g g p
T
FK C UUC 0
G. K. Ananthasuresh, IISc., Mar., 2013
T
hUC 0
Cook et al., Concepts and Applications of Finite Element Analysis.
40
Small area touch paradigm p g
1 FU K C
T
h
FU K CUC 0
I f d
11 1 1 1 11 TTK K C C K C K C K CK C
Inverse of partitioned matrix
11 1
T T TC 0 K C C K C
Advantages with this method• Solution is exact
Lagrange multipliers are reaction forces
G. K. Ananthasuresh, IISc., Mar., 2013
• Lagrange multipliers are reaction forces• Reaction forces are 11
ThC K C U
41
Large deformation modelg
Warping the stiffness along the rotation field
1( )f R K R x X 0( )elasticf R K R x X
Rotations are computed using geometric algebraVolume expansion in linear simulation
R
0x X
G. K. Ananthasuresh, IISc., Mar., 2013
0X 10R x X
1R x
42
Model reduction Suppressing interior DOF
int interior erior
11 12 13
2
1
1 22 23
1K K KK K KK K K F
UFF
UU
Zero
31 32 3 3 33K K K FU
int i-1 -1
22 21 22 231 3= - K K -K KU UU
interior 22 21 22 231 3K K K KU UU
Reduced stiffness matrix
fixed
1 3
-1 -111 12 22 21 13 2 22 2
-1 -11 1K K K K K K K K
K K K U FK KU F
K K K
G. K. Ananthasuresh, IISc., Mar., 2013
3 1 33 3 3 1 1
31 2 22 2 2 3 322 2K K K U FK K K K K
43Path replayMethod one
Force rendered to the user using force‐displacement
model
G. K. Ananthasuresh, IISc., Mar., 2013
Global deformation update computed using real‐time FEM
44
Region of attractionMethod two
A controller draws you to the points that were interacted before. Once the i e a e e o e O e epoint is reached the user can experience the forces that were experienced before
G. K. Ananthasuresh, IISc., Mar., 2013
Global deformation update computed using real‐time FEM
45Controller Disturbance from human hand
RobotPIDPath to nearest snapping point
Thorough evaluation of the controller is still to be done
G. K. Ananthasuresh, IISc., Mar., 2013
46
2D demonstration
Modified Falcon grip
Data acquisition qboard
Force sensorE l f D h d lExploration of a 2D stomach model using a falcon device with modified grip
G. K. Ananthasuresh, IISc., Mar., 2013
Both force and position information is read in the same simulation loop (Visual C++)
47
Haptic capture
G. K. Ananthasuresh, IISc., Mar., 2013
48Haptic playback
G. K. Ananthasuresh, IISc., Mar., 2013
49
Main points
• Haptic playback• Endoscopic simulator can be used for haptic playback of endoscopy.p p y py
• Challenges and issues– Synchronous measurement of position andSynchronous measurement of position and force
– Real‐time large deformation analysisReal time large deformation analysis– Control– The hardware
G. K. Ananthasuresh, IISc., Mar., 2013
The hardware
50
Thank youThank you.
Multi‐disciplinary andThe www.mecheng.iisc.ernet.in/~m2d2
Multi disciplinary and Multi‐scale Device and D i
M2D2 group
G. K. Ananthasuresh, IISc., Mar., 2013
Designgroup