Haptic playback

l b kHaptic PlaybackG.K. Ananthasuresh M h i l E i iMechanical EngineeringIndian Institute of ScienceBangaloreBangalore

Work done withShanthanu ChakravarthyP. MuddukrishnaAvinash KumarAvinash KumarSantosh D. Bhargav

2

Outline

• About rbccps• Haptics and CPS• The contextThe context

– Tele‐operation with force feedback– Micro to macro– Micro to macro– From here to there

• Ha ti layba k• Haptic playback– Endoscopy simulation and playback

G. K. Ananthasuresh, IISc., Mar., 2013

• The details• Main points

3

Robert Bosch CentrefforCyber Physical Systems• A philanthropic grant from the Robert Bosch Foundation

(Rs. 11.5 crores per year for 10 years)• Five verticals

– Agriculture; Buildings; HealthcareTransportation; Water– Transportation; Water

• 20 projects Centre staff 50 faculty Students

• Liaison with industries Internships!

rbccps.iisc.ernet.in [email protected]

Internships!


Send e‐mail for summer and other internships with “Internship” as the subject of the e‐mail.

4

InvestigatorsCyber Surgery and Remote Patient CareAshitava Ghosal, ME Dept., IISc, BangaloreG K Ananthasuresh, ME Dept., IISc BangaloreV Natarajan, CSA Dept., IISc BangaloreCh d Sh k S l l EE D IIS B lChandra Shekar Seelamantula, EE Dept., IISc BangaloreB Gurumoorthy, CPDM, IISc BangaloreDibakar Sen, CPDM, IISc Bangalore

Dr. Pradeep Rebala, Asian Institute of Gastroenterology, HyderabadDr. D Nageswara Reddy, Asian Institute of Gastroenterology, Hyderabad

Project Staff & StudentsDr. Parama Pal – RBCCPS, IISc BangaloreAditya K – ME Dept., IISc Bangalorey p , gAbin Jose – EE Dept., IISc BangaloreAvinash Kumar – ME Dept., IISc BangaloreSarvesh Kolekar – CPDM, IISc Bangalore


S. Sagar – CPDM, IISc BangaloreRahul Sharma – CSA Dept., IISc BangaloreShanthanu Chakravarthy – ME Dept., IISc Bangalore

5

Cyber Surgery and Remote Patient Care\ Virtual exploration of objects WP3

oscope

\

ator

ination Virtual exploration of objects

Haptic simulation

WP3

WP4

ble endo

c simula

nt exami

Haptic control WP5

ke flexib

doscop

ic

e pa

tien

Interactive visualization

Heterogeneous material

WP6

Snake‐lik

obot En

d

Remote Heterogeneous material

representation

I i

WP7


S r Image processing

WP1 WP2 WP9

WP8

6

\oscope

\

ator

ination

SMA‐activated

ble endo

c simula

nt exami

ke flexib

doscop

ic

e pa

tien

Snake‐lik

obot En

d

Remote

Silicone tube with nylon braid


S r

WP1 WP2 WP9

7

\oscope

\

ator

ination

ble endo

c simula

nt exami

ke flexib

doscop

ic

e pa

tien

Snake‐lik

obot En

d

Remote


S r

WP1 WP2 WP9

8

\

Patient in the ambulance

oscope

\

ator

ination

ble endo

c simula

nt exami

ke flexib

doscop

ic

e pa

tien

Doctor in the hospital

Snake‐lik

obot En

d

Remote hospital


S r

WP1 WP2 WP9

9

Virtual exploration of objects WP3Virtual exploration of objects

Haptic simulation

WP3

WP4

Haptic control WP5



WP6

Heterogeneous material representation

I i

WP7


Image processing WP8

10


Haptic simulation

WP3

WP4

Haptic control WP5



WP6


I i

WP7



11

Virtual exploration of objects WP3Position Virtual exploration of objects

Haptic simulation

WP3

WP4

Position

Haptic control WP5User



WP6Haptic Device


I i

WP7ForceInput Path

O t t P th


Image processing WP8Output Path

12


Haptic simulation

WP3

WP4

Haptic control WP5



WP6


I i

WP7



13


Haptic simulation

WP3

WP4

Haptic control WP5



WP6


I i

WP7



14


Haptic simulation

WP3

WP4

Haptic control WP5



WP6


I i

WP7



15

The Centre also has teeny weeny ψφ projects.

Desalination bottle Mobile diagnostic tool Breath sensor Breath sensor Touch‐screen anywhere Granular flow sensorGranular flow sensor Cognitive jewellery


16

CognitiveCognitive Jewellery

Technology + Aesthetics + FashionN l t i t f ith th h i l ldNovel ways to interface with the physical world.One can wear biomedical sensors in style.

With …Anand P.Pragathi M.Dhruv S.Deepika M. S.


17

Outline

• Haptics and CPS• The context

– Tele‐operation with force feedbackp– Micro to macro– From here to thereFrom here to there

• Haptic playbackEndoscopy simulation and playback– Endoscopy simulation and playback

• The details


• Main points

18

Micromanipulation with haptics

I

I

Cell grasped with two contacts for

x xO OI ・ Input ports

actuated by fine‐motion stages

Compliant micro mechanism

Miniature grippers to hold and manipulate cells.Haptic interfaceI

I I

Cell probed with a single contact

contacts for manipulation

x

x O

Light source

Compliant micro mechanism

O ・ Output ports in contact with cellx ・ Observation ports for tracking and force computation

mechanism

x y z

Haptic interface.Mechanical characterization by solving inverse problems in mechanics.

I I

I I

Gross motion stage

Circular motion t

Fine motion stage

Tiltable armm

xyz

x y z

Controller

stage

Microscope

Haptic interfacefor human operator

x

CCD camera

Image processingComputation of forces and displacements

PC


I/O to controller and haptic interface

19

Micromanipulation station


20Miniature grippersthat also sense forcesthat also sense forces Reddy, A. N., Sahu, D. K., Maheswari, N., and Ananthasuresh, G. K., “Miniature Compliant Grippers with Vision‐based Force‐sensing,” IEEE Transactions on Robotics, Vol. 26, No. 5, pp. 867‐877, 2010.

Miniature gripper

Zebrafish egg in jaws

G. K. Ananthasuresh, IISc., Mar., 2013Squeezing HeLa cellsGripper small than “1” in the one rupee coin

21

Piercing into a zebrafish egg cell


22

Master‐slave motion with force‐f db k i ti ttifeedback in tissue‐cutting


23Setup

Visual

display

Slave device

Compliant cutting toolPh i

p gPhantom tissue


24

MP 285MP 285

CMOS

camera


Scalpel

25

Passive control with a compliant end-effector

Output displacement AnchoredAnchored

Rigid-body translation of the

l t (X )

21

plate (Xa) Rigid plate of end-effector casing

Y

X Ensuring cutting force (Fc)Phantom tissue


26

Phantom tissue with a hard inclusion

Gelatin-based hydrogel

PDMS


27Comparing rigid end-effector with one that is compliantp

Saturation of the load-cell


28

Haptic PlaybackHaptic Playback


29

Vision can be captured by camera , which can be viewed l tlater. Audio can be captured using sound recorder and replayed later using a speaker

What about other senses?Photo courtesy Santosh

Haptic replay

Environment

Force

Position

Haptic replay is difficult because of the duality of force and displacement


30

Endoscopy PlaybackPlaying back a procedure done in the past…

Snapshot from endoscope image sensorimage sensor

Case for haptic replay Palpation is one of the important modes of diagnosis and in current

endoscopy procedure, no force information is stored for later replay.


py p p y Important information like stiffness from earlier examination can be

replayed to the doctor to help in diagnosis.

31

Haptic playback with an endoscopy simulator

Step one Step two Step three

Endoscopic session

Step one Step two Step three

Endoscopic session carried out using a modified endoscope.

Rough estimation of tissue properties

User

Data acquisition from the instrumented endoscope

Creating a reality‐based haptic model

Haptic Device

instrumented endoscope haptic model

Doctors can explore the virtual model to recall what was experienced


earlier.

32

Virtual haptic model

Position

Collision Detection

Object DBC

HA Collision information

SensorO/PData Physical properties

Objectgeometry

AI

3D

Physically based simulation(FEM)

Haptic

Collision information

Actuato

User of Object

D (FEM)

Force

DeviceHaptics Rendering

engine

r I/P Data

Force gInput Path

Output Path“The Haptics Loop”


Object geometry, haptic robot workspace and graphics screen are synchronized.

33

Force measurement

Modified Falcon grip

Data acquisitionacquisition board

Force sensorForce sensor


34

Position measurementBoth position and forces have to be measured simultaneously y

x

z

z

( )( ) z

xR y

T t

( )

0 0 0 1

T tz

Rotations at every time instance is measured using

Forward kinematics for position estimation Linear position measurement device


encoders

35

Rough estimation of tissue ti f lit b dproperties for reality‐based

modelingg

exp1

erimentali i

FEM

FE E

F

S h d

experimental FEMF F

Synthetic data


36Local elastic modulus

I iti l f EInitial guess for E

from experimentU

Nonlinear Finite Element Method

experimental FEMF F

experimentalF exp1

erimentali i

FEM

E EF

E(U) for the synthetic data

considered


37

Grid method for collision detection Impose lookup grid inside the bounding box containing thebounding box containing the polygon.

Categorize each cell as being Ca ego i e eac ce as ei gfully inside, fully outside, or indeterminate with respect to the polygon. p yg

For the indeterminate cells use line crossing test to see if the point i i idis inside.

Once collision is detected use pre computed cell data to get the


pre‐computed cell data to get the nearest node.

Haines, Point in Polygon Strategies, Graphics Gems IV, 1994.

38

Deformation mechanicsDeformation mechanics Problem Definition:

0b in 0 fixedon U

2tr E I E someportion ofon hU U

Haptic DOF

Problem in discretized frame work:

11 12 13

21 22 23 2 2

1 1K K KK K K U F

U F


31 332 33 3K K K U F

Fixed DOF

39

Lagrange multiplier Method

Optimization Problem Min 1 UTKU - UTF Subj. to : hU=U

Equivalent Problem

Problem MinU 2

U KU U F j h

MinU,

12

UTKU - UTF 12 CU - Uh

U, 2 2

isLagrange multiplier vectorwhere isconstrained matrixC

g g p

T

FK C UUC 0


T

hUC 0

Cook et al., Concepts and Applications of Finite Element Analysis.

40

Small area touch paradigm p g

1 FU K C

T

h

FU K CUC 0

I f d

11 1 1 1 11 TTK K C C K C K C K CK C

Inverse of partitioned matrix

11 1

T T TC 0 K C C K C

Advantages with this method• Solution is exact

Lagrange multipliers are reaction forces


• Lagrange multipliers are reaction forces• Reaction forces are 11

ThC K C U

41

Large deformation modelg

Warping the stiffness along the rotation field

1( )f R K R x X 0( )elasticf R K R x X

Rotations are computed using geometric algebraVolume expansion in linear simulation

R

0x X


0X 10R x X

1R x

42

Model reduction Suppressing interior DOF

int interior erior

11 12 13

2

1

1 22 23

1K K KK K KK K K F

UFF

UU

Zero

31 32 3 3 33K K K FU

int i-1 -1

22 21 22 231 3= - K K -K KU UU

interior 22 21 22 231 3K K K KU UU

Reduced stiffness matrix

fixed

1 3

-1 -111 12 22 21 13 2 22 2

-1 -11 1K K K K K K K K

K K K U FK KU F

K K K


3 1 33 3 3 1 1

31 2 22 2 2 3 322 2K K K U FK K K K K

43Path replayMethod one

Force rendered to the user using force‐displacement

model


Global deformation update computed using real‐time FEM

44

Region of attractionMethod two

A controller draws you to the points that were interacted before. Once the i e a e e o e O e epoint is reached the user can experience the forces that were experienced before


Global deformation update computed using real‐time FEM

45Controller Disturbance from human hand

RobotPIDPath to nearest snapping point

Thorough evaluation of the controller is still to be done


46

2D demonstration

Modified Falcon grip

Data acquisition qboard

Force sensorE l f D h d lExploration of a 2D stomach model using a falcon device with modified grip


Both force and position information is read in the same simulation loop (Visual C++)

47

Haptic capture


48Haptic playback


49

Main points

• Haptic playback• Endoscopic simulator can be used for haptic playback of endoscopy.p p y py

• Challenges and issues– Synchronous measurement of position andSynchronous measurement of position and force

– Real‐time large deformation analysisReal time large deformation analysis– Control– The hardware


The hardware

50

Thank youThank you.

Multi‐disciplinary andThe www.mecheng.iisc.ernet.in/~m2d2

Multi disciplinary and Multi‐scale Device and D i

M2D2 group


Designgroup

Date post:	25-May-2015
Category:	Technology
Upload:	cpsworkshop
View:	450 times
Download:	1 times