Feasibility of time-delayed teleoperation NASA_FISO...Different approaches Credit: ... (FLS) •...

Feasibility of time-delayed teleoperationHuman limitations in critical tasks

Tamás Haidegger

Lab. of Biomedical Engineering, Budapest University of Technology and Economics (BME)Austrian Center for Medical Innovation and Technology (ACMIT)

NASA Future In-Space Operations

07.27.2011. Telecon

A year at Johns Hopkins

2001–06 MSc in EE at Budapest Univ. of Technology and Economics2005–08 MSc in Biomedical Engineering at BME2006–09 EE doctoral school at BME2007–08 visiting scholar at CISST ERC, Johns Hopkins2009– 11 doctoral candidate 2010– assistant research fellow at BME 2010–11 visiting research fellow/postdoc at ACMIT2012– adjunct professor at BME2011– CEO/CTO Clariton Ltd.

Background

NASA Future In-Space Operations 07.27.2011.

2

Content

• Space healthcare

• Introduction to teleoperated surgical robots

• s

3

A year at Johns Hopkins

“The same technology that made the "Canadarm" a household name in

space will soon be used by brain and spinal surgeons in Calgary.”

/MD Robotics, 2002/

The origins 4

DARPA projekt 5


Requirements for safe human space exploration

Sending humans to space 6


• To meet the mission objectives � Ties to the military

• With best people� Thorough screening� Advanced technology � Ethical questions

• With the best tools to support� Best devices/equipment� Use of modern medtech� Computer-Integrated Surgery (CIS)� Simulate patients

Courtesy of Rick Satava

Motivation

“The future of surgery is not about blood and guts;

the future of surgery is about bits and bytes.”

/Rick Satava/

Clinical: healing lesions

• Reducing error margins during surgery

Imag

e c

red

it:

Eday

a

7

Engineering: applying technical innovation

• Conquering new fields

• Improving effectiveness, benefits while reducing costs

“If at first an idea is not absurd, then there is no hope for it.”

/Albert Einstein/


A year at Johns HopkinsMotivation

Effective Space Medicine

Handling latency

Handling environmental constraints

Software support Hardware support

Human capability assessment

Skill training

8

Enabling technologies for remote operation• critical tasks: e.g. health care

Source: Joel Jensen, SRI International, Menlo Park, CA

MIS/open surgery

Pre-op planning

Intra-operative navigation

Telesurgery

Simulation, practice and warm-up

Sensor fusion

Robot-assisted surgery

New paradigm

in medicine

INFORMATION Augmented reality

Crew health management

Crew Health Care System (CHeCS)/ Integrated Medical System

• on the ISS

• on the Moon

• L1, L2

• beyond[Campbell 2008]

10

[Allen 2003]

Deployment of telemedicine 11

Risk control

Various metrics and standards

• Probability of serious illness or injury: 0.9

� 2.5-year-long Mars mission with six crew members [Allen 2003]

• Health Care Standard – Level of Care 5 recommended

� For any mission longer than 210 days; Johnson Space Center Space Medicine Division [NASA SFHSS, 2007]

• Probability of death during a mission: <0.03/year

� European Space Agency standard [Homeck 2001].

12


DARPA projekt 13


Robots for space medicine

Robotics for interventional medicine

• Seed idea: NASA should provide surgical care for astronauts with remote controlled robots [Alexander 1972]

• DoD finance for a battlefield remote rescue system [Satava 1995]

• Telesurgery systems Zeus and da Vinci made it out of lab

DARPA: Trauma Pod project

Operating Room of the Future

14


Why robots?

[Taylor 1997]


15

Teleoperating a robot

• Non time-critical teleoperation

• Time-critical teleoperation


16

Exploration of unstructured,

dynamic environments

Facing the challenges

• Human-in-the-loop control

– Leave the mapping to the surgeon

• Registration (image) based

– Human oversight

Different approaches

Cre

dit

: C

UR

EX

O In

c.

Methods to improve the accuracy of treatment delivery


17

DARPA projekt 18


Current surgical robot systems and prototypes

master slave

Complete teleoperational system 19


The da Vinci surgical system

• 1900 systems, 600 000++ operations, 120 types of procedures

• Σ850 kg, 2x1x1 m

• $1.5 M upfront fee

• $2K /device (10 uses)

• $100K annual service fee

Intuitive Surgical Inc.

20


DARPA projektRobot systems 21

• Robot Assisted Micro Surgery (RAMS), 1995• Surgenius (U. Verona), 2011

• M7 – SRI International, 1998• Raven – University of Washington, 2006• Lightweight arm III – DLR, 2006• Mobile in-vivo robot, 2008

– University of Nebraska

RAMS

M7

Raven

Wheeled in-vivo

DLR

Raven

• University of Washington, 2006• 22 kg overall mass• Haptic feedback• 6 DOF small arms• 5 DOF positioning arms

Surgical robots for space

M7

• SRI International, 1998• Light weight – 15 kg• 7 DOF arms• 1:10 scale down • tremor filtering

22

RAMS

• Robot-Assisted Microsurgery• NASA – JPL, 1995• Two 6 DOF arms for • 40 ccm workspace• 10 micron accuracy• 1:100 scale down• tremor filtering

DARPA projekt 23


Challenges of teleoperation

The evil latency

Improving the comm. infrastructure

Standards and protocols

• SCPS (Space Comm. Protocol Specif.)• STRS (Space Telecomm. Radio System)• Redundant networks of satellites• Communication hubs: e.g. L1

Bandwidth is also critical

• 2–4 Mbps is the minimum for teleoperationwith visual feedback

• 40 Mbps required for HD teleoperation

24

Quantum physics is a hard-limit on speed of communication

Network delays are even worth

• LEO: 80++ ms, depending on the network infrastructure• MEO: 100+ ms• GEO: 540+ ms• Mars: 6.5–44 min• Military: 4–8 ms/hop

John B. Charles – FISO; June 29, 2011

[Rayman 2007]

Breaking down the latency

Focusing on applicationsFrom Canadian experiments [Nguan 2008]• robotic data transport delay: 70 ms• commanded robotic motion on the patient side: 85.7 ms• CCD and camera controller latency: 20 ms• video transport delay: 70 ms• video compression/decompression (CODEC): 90 ms• video synchronizer delay: 33.6 ms

25

[Rayman 2007]

[Credit: CSTAR]

Human operator model

Classical control in telesurgical tasks

Simple master–slave teleoperation task

• equalizaQon type (look and move)―developed for pilots in the ‘60s• Possible to observe adaptation to latency

Cross-over model

Fitts’ law

Human learning

[McRuer 1965]

[Chien et al. 2010]

26

Human adaptation

Short/long term learning

Outline ¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers ¤ Standardization efforts ¤ Case study ¤ Conclusion

Based on various experiments

• Unnoticeable <100 ms• Most humans: max. 500 ms• Some might be able to adapt to 1000 ms• Jitter/consistent latency • Short term adaptation• Haptic feedback• Completion time, path length, forces


27


[Rayman 2005]

[Tholey 2007]

Measuring adaptation

Reducing the degrees of freedom


Fundamentals of Laparoscopic Surgery (FLS)

• Society of American Gastrointestinal and Endoscopic Surgeons (SAGES)

• TeleRobotic FLS (UW, BioRobotic group)• Haptic feedback • Pilots’/gamers’/drivers’ behavior


28


[Lum 2008]

[Thompson 1999]

DARPA projekt 29


Experiments and concepts to support teleoperation

over large distance

Teleoperation experiments— non-critical

Internet is the ultimate platform

• Using mostly UDP or TCP/IP• Internet latency is decreasing

� 85 ms across the U.S. (2009)� 400 ms world-wide

Telemedicine on the rise

• Simulated experiences, early ‘90s

• 1st telesurgery consultation, 1996

• 1st professional telementoring, 1997

• 1st international and mobile procedures, 1999

• 1st regular telemedicine network

(Centre for Minimal Access Surgery)

30

[Rayman 2005]

[Anvari 2004]

[Fabrizio 2000]

Telesurgical experiments—critical

Long range procedures around the world

The Lindbergh operation

• 7th Sept, 2001 • New York <---> Strasbourg• Hour-long gallbladder removal• Master setup in France Telecom office• Average latency: 150 ms

CSTAR

• Canadian Surgical Technologies and Advanced Robotics London (ON, Canada)

PlugFest 2009

• Telesurgery experiment w/14 systems world wide

• 21–112 ms latency for the U.S. • 115–305 ms for intercontinental

31

Credit: IRCAD

DARPA projektNEEMO 32


32

NASA Extreme Environment Mission Operation

7th NEEMO (October, 2004)

● Reference procedures with a Zeus ● Telesurgery from 2500 km ● 0.1–2 s delay● Feasibility test for telementoring

9th NEEMO (April, 2006)

● Simulated procedures with M7 ● Test of wheeled in-vivo robots ● Satellite comm., up to 3s delay ● Telemedicine tests

12th NEEMO (May, 2007)

● Telesurgery with Raven and M7 robots ● Suturing test in zero gravity simulation● Laparoscopic object manipulation● Max. 1 s communication lag time ● Automated robotic operations

Concept of complete telemedical support


Haidegger&Benyó2008

33

Concept of complete telemedical support I

■ Earth Orbit (near Earth):

. Classical telesurgery

� App. within 380 000km

� Under 2 s signal delay• Excellent for ISS• On Earth spin-offs

.

34

Enhancing human capabilities


35

VIDEO: UTwente‘s TELEFLEX HMI

Determining control models II

Slave robot model

Tissue characteristics

[Kawashima et al. 2008]

Outline ¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers ¤ Standardization

efforts ¤ Case study ¤ Conclusion


efforts ¤ Case study ¤ Conclusion


efforts ¤ Case study ¤ ConclusionNASA Future In-Space Operations 07.27.2011.

36

Cascade control

Applicability• secondary loop dynamics must be faster than primary loop dynamics,• secondary loop must have influence on the primary loop,• secondary loop must be observable and controllable.





37

Controller design methods

Empirical methods:

Kessler’s method extended to both loopsModulus Criteria (CM) or (Extended) Symmetrical Optimum (SO)

Soft computing methods:

PID–Fuzzy controllersModern control theories:

Model predictive methodsCommunication disturbance observers (ND – CDOB)




Realistic control task

• Stability criteria• Bilateral and transparent teleoperation• Appropriate parameters• Tolerate 1 s delay in a telesurgical system


38

Concept of complete telemedical support II

Middle range: Telementoring

� App. within 10 000 000km

� Under 50–70s signal delay

� Permanent video contact with the ground

39

40

Concept of complete telemedical support III

� Long distance: Consultancy telemedicine

� Within the range of the Mars

( 200 000 000 km)

� App. 10–40 min signal delay

� Preoperative simulations and consulting

� Prerecorded and locally adapted procedures

4040

Automation on the horizon




41

We are in the age of intelligent tools • No real automation in the OR yet

• Automated manufacturing• Enhancement (e.g. tremor filtering, auto-navigation, camera handling)• Voice, eye-gaze control

• 2006 , automated teleoperated heart ablation [Pappone 2006] • Research phase

• Knot tying• Knot tightening

• Repeating pre-defined sequences

VIDEOS: enhanced da Vinci robot at JHU; collaborative surgery; endosuturing device

DARPA projekt 42


The real prospects of telesurgery in space

• Heavy weight, big size

• High R&D costs

– da Vinci was created with $0.5B investment

• Long development time

– From idea to product: 8–10 years

• Universality vs. added value

• Further uses of a robot on board

– To gain routine and to improve performance

– To conduct research, servicing

– For micromanipulation ops, repetitive functions

– Post-operative care

Cost / benefit 43

NASA Future In-Space Operations

SSSA

NTU

Environmental constraints

Surgery in weightlessness is challenging

Lack of experience

• Slower motion • Higher failure rate• Question of evaluation

• Diagnostics difficulties• Managing body fluids• Retraction

44

[Panait 2004]

[Rafiq 2005][Broderick 2005]

Surgery in space

NASA zero G experiments

• On board of a DC-9 hyperbolic aircraft• Simulated surgery (2007)• Suturing with M7 robot• Human control / automated task execution

Zero G surgeries

• Laparotomy and celiotomy on rabbits by cosmonauts (1967)• Feasibility of endoscopic surgery [Campbell 2001]• First survival surgery in weightlessness on a rat (2003)• Removal of a cyst from the arm of a human (2006)

• Through 25 parabolic sessions of microgravity• ESA Zero-G plane (modified Airbus A-300)

Non-invasive diagnostics on the ISS

• Ultra Sound experiments• BioLab module within Columbus

Cre

dit

: ES

A, N

ASA

45

DARPA projektOther requirements 46


Software infrastructure

• Complete health monitoring system• On-line clinical information database access• Strategic health care research planner

(for data analysis and support)

• Medical knowledge base (to identify the risks and hazards)

• Astronauts should receive comprehensive medical training

On Earth support• Terrestrial health support centre• Complete surgical simulator on Earth• Multimodal physiologic model of astronauts

(for reference on health status)

DARPA projektConclusion 47


Teleoperation through time-delayed channels is challenging • Various hardware and software issues• Question of human adaptation• Lack of standards (both for instruments and communication)• Ethical issues

• User confidence

Space medicine is also hard • Physical and ergonomical challenges• Environmental constraints

Fundamental research is required to develop the science behind

Thank you for your attention!

SurgRoba blog on CIS and medical robotics

http://surgrob.blogspot.com

Post script

Thesis is available: http://tinyurl.com/3v3dmwt

IEEE RAS society 50

• http://ieee-ras.org

Acknowledgment

My research projects were funded by:

Austrian COMET-K1, EU FP7-PEOPLE-2009-IIF,

US NSF EEC 9731748, Hungarian NKTH OTKA T80316,New Hungary Development Plan

(Project ID: TAMOP-4.2.1/B-09/1/KMR-2010-0002)

Stiftung Aktion Österreich-Ungarn (AÖU) grants

I am deeply grateful to all my supervisors, colleagues and students.

51

BME by night

Date post:	30-Jun-2018
Category:	Documents
Upload:	buiminh
View:	215 times
Download:	0 times

Feasibility of time-delayed teleoperation NASA_FISO...Different approaches Credit: ... (FLS) •...

Documents