Haptic playback

transcript

l b kHaptic PlaybackG.K. Ananthasuresh M h i l E i iMechanical EngineeringIndian Institute of ScienceBangaloreBangalore

Work done withShanthanu ChakravarthyP. MuddukrishnaAvinash KumarAvinash KumarSantosh D. Bhargav

Outline

• About rbccps• Haptics and CPS• The contextThe context

– Tele‐operation with force feedback– Micro to macro– Micro to macro– From here to there

• Ha ti layba k• Haptic playback– Endoscopy simulation and playback

G. K. Ananthasuresh, IISc., Mar., 2013

• The details• Main points

Robert Bosch CentrefforCyber Physical Systems• A philanthropic grant from the Robert Bosch Foundation

(Rs. 11.5 crores per year for 10 years)• Five verticals

– Agriculture; Buildings; HealthcareTransportation; Water– Transportation; Water

• 20 projects Centre staff 50 faculty Students

• Liaison with industries Internships!

rbccps.iisc.ernet.in rbccps@gmail.com

Internships!

Send e‐mail for summer and other internships with “Internship” as the subject of the e‐mail.

InvestigatorsCyber Surgery and Remote Patient CareAshitava Ghosal, ME Dept., IISc, BangaloreG K Ananthasuresh, ME Dept., IISc BangaloreV Natarajan, CSA Dept., IISc BangaloreCh d Sh k S l l EE D IIS B lChandra Shekar Seelamantula, EE Dept., IISc BangaloreB Gurumoorthy, CPDM, IISc BangaloreDibakar Sen, CPDM, IISc Bangalore

Dr. Pradeep Rebala, Asian Institute of Gastroenterology, HyderabadDr. D Nageswara Reddy, Asian Institute of Gastroenterology, Hyderabad

Project Staff & StudentsDr. Parama Pal – RBCCPS, IISc BangaloreAditya K – ME Dept., IISc Bangalorey p , gAbin Jose – EE Dept., IISc BangaloreAvinash Kumar – ME Dept., IISc BangaloreSarvesh Kolekar – CPDM, IISc Bangalore

S. Sagar – CPDM, IISc BangaloreRahul Sharma – CSA Dept., IISc BangaloreShanthanu Chakravarthy – ME Dept., IISc Bangalore

Cyber Surgery and Remote Patient Care\ Virtual exploration of objects WP3

oscope

ination Virtual exploration of objects

Haptic simulation

ble endo

c simula

nt exami

Haptic control WP5

ke flexib

doscop

Interactive visualization

Heterogeneous material

Snake‐lik

obot En

Remote Heterogeneous material

representation

S r Image processing

WP1 WP2 WP9

\oscope

ination

SMA‐activated

ble endo

c simula

nt exami

ke flexib

doscop

Snake‐lik

obot En

Remote

Silicone tube with nylon braid

WP1 WP2 WP9

\oscope

ination

ble endo

c simula

nt exami

ke flexib

doscop

Snake‐lik

obot En

Remote

WP1 WP2 WP9

Patient in the ambulance

oscope

ination

ble endo

c simula

nt exami

ke flexib

doscop

Doctor in the hospital

Snake‐lik

obot En

Remote hospital

WP1 WP2 WP9

Virtual exploration of objects WP3Virtual exploration of objects

Haptic simulation

Haptic control WP5

Heterogeneous material representation

Image processing WP8

Haptic simulation

Haptic control WP5

Virtual exploration of objects WP3Position Virtual exploration of objects

Haptic simulation

Position

Haptic control WP5User

WP6Haptic Device

WP7ForceInput Path

O t t P th

Image processing WP8Output Path

Haptic simulation

Haptic control WP5

Haptic simulation

Haptic control WP5

Haptic simulation

Haptic control WP5

The Centre also has teeny weeny ψφ projects.

Desalination bottle Mobile diagnostic tool Breath sensor Breath sensor Touch‐screen anywhere Granular flow sensorGranular flow sensor Cognitive jewellery

CognitiveCognitive Jewellery

Technology + Aesthetics + FashionN l t i t f ith th h i l ldNovel ways to interface with the physical world.One can wear biomedical sensors in style.

With …Anand P.Pragathi M.Dhruv S.Deepika M. S.

Outline

• Haptics and CPS• The context

– Tele‐operation with force feedbackp– Micro to macro– From here to thereFrom here to there

• Haptic playbackEndoscopy simulation and playback– Endoscopy simulation and playback

• The details

• Main points

Micromanipulation with haptics

Cell grasped with two contacts for

x xO OI ・ Input ports

actuated by fine‐motion stages

Compliant micro mechanism

Miniature grippers to hold and manipulate cells.Haptic interfaceI

Cell probed with a single contact

contacts for manipulation

Light source

Compliant micro mechanism

O ・ Output ports in contact with cellx ・ Observation ports for tracking and force computation

mechanism

Haptic interface.Mechanical characterization by solving inverse problems in mechanics.

Gross motion stage

Circular motion t

Fine motion stage

Tiltable armm

Controller

Microscope

Haptic interfacefor human operator

CCD camera

Image processingComputation of forces and displacements

I/O to controller and haptic interface

Micromanipulation station

20Miniature grippersthat also sense forcesthat also sense forces Reddy, A. N., Sahu, D. K., Maheswari, N., and Ananthasuresh, G. K., “Miniature Compliant Grippers with Vision‐based Force‐sensing,” IEEE Transactions on Robotics, Vol. 26, No. 5, pp. 867‐877, 2010.

Miniature gripper

Zebrafish egg in jaws

G. K. Ananthasuresh, IISc., Mar., 2013Squeezing HeLa cellsGripper small than “1” in the one rupee coin

Piercing into a zebrafish egg cell

Master‐slave motion with force‐f db k i ti ttifeedback in tissue‐cutting

23Setup

Visual

display

Slave device

Compliant cutting toolPh i

p gPhantom tissue

MP 285MP 285

camera

Scalpel

Passive control with a compliant end-effector

Output displacement AnchoredAnchored

Rigid-body translation of the

l t (X )

plate (Xa) Rigid plate of end-effector casing

X Ensuring cutting force (Fc)Phantom tissue

Phantom tissue with a hard inclusion

Gelatin-based hydrogel

27Comparing rigid end-effector with one that is compliantp

Saturation of the load-cell

Haptic PlaybackHaptic Playback

Vision can be captured by camera , which can be viewed l tlater. Audio can be captured using sound recorder and replayed later using a speaker

What about other senses?Photo courtesy Santosh

Haptic replay

Environment

Position

Haptic replay is difficult because of the duality of force and displacement

Endoscopy PlaybackPlaying back a procedure done in the past…

Snapshot from endoscope image sensorimage sensor

Case for haptic replay Palpation is one of the important modes of diagnosis and in current

endoscopy procedure, no force information is stored for later replay.

py p p y Important information like stiffness from earlier examination can be

replayed to the doctor to help in diagnosis.

Haptic playback with an endoscopy simulator

Step one Step two Step three

Endoscopic session

Step one Step two Step three

Endoscopic session carried out using a modified endoscope.

Rough estimation of tissue properties

Data acquisition from the instrumented endoscope

Creating a reality‐based haptic model

Haptic Device

instrumented endoscope haptic model

Doctors can explore the virtual model to recall what was experienced

earlier.

Virtual haptic model

Position

Collision Detection

Object DBC

HA Collision information

SensorO/PData Physical properties

Objectgeometry

Physically based simulation(FEM)

Haptic

Collision information

Actuato

User of Object

D (FEM)

DeviceHaptics Rendering

engine

r I/P Data

Force gInput Path

Output Path“The Haptics Loop”

Object geometry, haptic robot workspace and graphics screen are synchronized.

Force measurement

Modified Falcon grip

Data acquisitionacquisition board

Force sensorForce sensor

Position measurementBoth position and forces have to be measured simultaneously y

( )( ) z

0 0 0 1

Rotations at every time instance is measured using

Forward kinematics for position estimation Linear position measurement device

encoders

Rough estimation of tissue ti f lit b dproperties for reality‐based

modelingg

erimentali i

experimental FEMF F

Synthetic data

36Local elastic modulus

I iti l f EInitial guess for E

from experimentU

Nonlinear Finite Element Method

experimental FEMF F

experimentalF exp1

erimentali i

E(U) for the synthetic data

considered

Grid method for collision detection Impose lookup grid inside the bounding box containing thebounding box containing the polygon.

Categorize each cell as being Ca ego i e eac ce as ei gfully inside, fully outside, or indeterminate with respect to the polygon. p yg

For the indeterminate cells use line crossing test to see if the point i i idis inside.

Once collision is detected use pre computed cell data to get the

pre‐computed cell data to get the nearest node.

Haines, Point in Polygon Strategies, Graphics Gems IV, 1994.

Deformation mechanicsDeformation mechanics Problem Definition:

0b in 0 fixedon U

2tr E I E someportion ofon hU U

Haptic DOF

Problem in discretized frame work:

11 12 13

21 22 23 2 2

1 1K K KK K K U F

31 332 33 3K K K U F

Fixed DOF

Lagrange multiplier Method

Optimization Problem Min 1 UTKU - UTF Subj. to : hU=U

Equivalent Problem

Problem MinU 2

U KU U F j h

UTKU - UTF 12 CU - Uh

U, 2 2

isLagrange multiplier vectorwhere isconstrained matrixC

FK C UUC 0

Cook et al., Concepts and Applications of Finite Element Analysis.

Small area touch paradigm p g

1 FU K C

FU K CUC 0

11 1 1 1 11 TTK K C C K C K C K CK C

Inverse of partitioned matrix

T T TC 0 K C C K C

Advantages with this method• Solution is exact

Lagrange multipliers are reaction forces

• Lagrange multipliers are reaction forces• Reaction forces are 11

ThC K C U

Large deformation modelg

Warping the stiffness along the rotation field

1( )f R K R x X 0( )elasticf R K R x X

Rotations are computed using geometric algebraVolume expansion in linear simulation

0X 10R x X

Model reduction Suppressing interior DOF

int interior erior

11 12 13

1 22 23

1K K KK K KK K K F

31 32 3 3 33K K K FU

int i-1 -1

22 21 22 231 3= - K K -K KU UU

interior 22 21 22 231 3K K K KU UU

Reduced stiffness matrix

-1 -111 12 22 21 13 2 22 2

-1 -11 1K K K K K K K K

K K K U FK KU F

3 1 33 3 3 1 1

31 2 22 2 2 3 322 2K K K U FK K K K K

43Path replayMethod one

Force rendered to the user using force‐displacement

Global deformation update computed using real‐time FEM

Region of attractionMethod two

A controller draws you to the points that were interacted before. Once the i e a e e o e O e epoint is reached the user can experience the forces that were experienced before

Global deformation update computed using real‐time FEM

45Controller Disturbance from human hand

RobotPIDPath to nearest snapping point

Thorough evaluation of the controller is still to be done

2D demonstration

Modified Falcon grip

Data acquisition qboard

Force sensorE l f D h d lExploration of a 2D stomach model using a falcon device with modified grip

Both force and position information is read in the same simulation loop (Visual C++)

Haptic capture

48Haptic playback

Main points

• Haptic playback• Endoscopic simulator can be used for haptic playback of endoscopy.p p y py

• Challenges and issues– Synchronous measurement of position andSynchronous measurement of position and force

– Real‐time large deformation analysisReal time large deformation analysis– Control– The hardware

The hardware

Thank youThank you.

Multi‐disciplinary andThe www.mecheng.iisc.ernet.in/~m2d2

Multi disciplinary and Multi‐scale Device and D i

M2D2 group

Designgroup