Supplementary Materials for

Supervised autonomous robotic soft tissue surgery

Azad Shademan, Ryan S. Decker, Justin D. Opfermann, Simon Leonard, Axel Krieger,

Peter C. W. Kim*

*Corresponding author. Email: pkim@childrensnational.org

Published 4 May 2016, Sci. Transl. Med. 8, 337ra64 (2016)

DOI: 10.1126/scitranslmed.aad9398

This PDF file includes:

Materials and Methods

Fig. S1. Young’s modulus calculations for pig bowel and human bowel, with

geometry and applied forces kept constant.

Fig. S2. Deformation patterns for markers illustrate the uncertain nature of motion

propagation in soft tissue.

Fig. S3. Robotic construction of end-to-end anastomosis.

Table S1. Quantitative geometric quality of ex vivo linear suturing.

Table S2. Quantitative geometric quality of ex vivo end-to-end anastomosis.

Legend for movie S1

Other Supplementary Material for this manuscript includes the following:

(available at www.sciencetranslationalmedicine.org/cgi/content/full/8/337/337ra64/DC1)

Movie S1 (.mp4 format). Supervised autonomous end-to-end intestinal

anastomosis.

www.sciencetranslationalmedicine.org/cgi/content/full/8/337/337ra64/DC1

Supplementary Materials and Methods:

Vision system

The vision system provided two key pieces of information about the scene: (i) A 3D point cloud of the

visible surfaces within surgical field, and (ii) a specific near-infrared (NIR) band image of the surgical

field to highlight near-infrared fluorescent (NIRF) markers. The 3D point cloud was obtained by a

custom plenoptic camera (R12, Raytrix GmbH) with custom field-of-view of 70×65×30 mm3. The

Raytrix camera has an Application Programming Interface (API) with proprietary binary libraries and

some read-only parameters. The API provided a virtual depth image and a focused image, which are

captured on a dedicated Windows machine with an NVidia GeForce GTX Titan graphics card. Virtual

3D depth is processed at 10 frames per second (fps) at a pixel resolution of 2008×1508. This virtual

depth was then streamed over LAN to a Linux machine running a custom ROS node. This camera was

calibrated in-house to provide metric coordinates of surface point cloud in the camera coordinate system

(18). This was achieved by moving a planar checkerboard pattern at regular fixed intervals along the

view axis and correcting for both depth scale and depth distortion.

An LED ring light with peak emission at 760 nm (North Coast Technical Inc.) was used to excite the

NIRF markers which were made from mixing indocyanine green (ICG) and cyanoacrylate. The excited

fluorescent light from the NIRF markers was captured by a custom camera (acA-50gm 2000NIR GigE,

Basler) equipped with CMOSIS CMV2000 CMOS sensor with 2048×1088 pixel resolution with high

quantum efficiency in the NIR region and a band-pass 845 ± 55 nm emission filter (Chroma Technology

Corp) (19). The plenoptic camera was registered to the NIRF camera such that metric Cartesian

coordinates of a point in the field of view correspond to its reflected fluorescent intensity on the NIRF

image. Therefore, the metric coordinates of a NIRF marker can be expressed in the camera coordinate

frame.

The last step in calibration was to calculate the relative transformation between the camera

coordinate system and the robot coordinate system, also known as hand-eye calibration. This was

necessary as the automation software assumed that the metric coordinates of NIRF markers were

expressed in the robot coordinate frame, the same coordinate frame where the feedback control loop was

closed. The hand-eye calibration was performed by solving a linear system of equations, which relate

the tool position expressed in the robot coordinate frame to the tool position expressed in the camera

coordinate frame, at several points. After metric calibration of plenoptic camera and hand-eye

calibration, there were still small inaccuracies in metric depth values. This type of inaccuracy was

compensated with force measurements by automatically advancing along the z-axis until the suturing

tool touched the tissue and senses tactile force. Similarly, small planar uncertainty in the quantification

of marker motion was compensated by semi-autonomous corrections to complete the suturing tasks. In

the event that the error of a detected motion was greater than the desired system accuracy, or greater

than the suturing tool jaw size, the surgeon supervised to adjust the tool position prior to placing a

suture, ensuring that sutures were placed at exactly the desired location.

Suture automation

In the case of an end-to-end anastomosis, the procedure was broken down to two phases: (i) Suturing of

the inside (or back) wall, and (ii) suturing of the outside (or front) walls. The tissue was staged by four

stay sutures similar to a clinically accepted staging method (fig. S3A). Biocompatible NIRF markers

were placed near the stay sutures to track deformation (fig. S3B). The high signal-to-noise ratio of the

excited NIRF signal provided bright blobs in the NIR images that were robust to blood or tissue

occlusion. The markers were mapped to the 3D point cloud, where a suture plan was calculated based on

the point locations and the procedure type. During execution of the suture plan, the system

autonomously placed the suture head at the suture position and then paused prior to actuating the needle

drive for each suture. The system would only place the suture and continue with suture tension and the

next suture if the user accepted the position with the click of a mouse. If the surgeon was not satisfied

with the position based on the video images provided by the system, he was able to make positional

adjustments by typing in x, y, and z offsets. The system would then reposition the suture head

accordingly until the user accepted the placement and the system continues with the next suture. If a

mistake occurred such as a locked suture, the user was able to stop the system with a click and the

needle was removed manually before the suturing routine was restarted. After suturing inside wall (fig.

S3C), the bowel was re-staged manually to complete the anastomosis (fig. S3D). The NIR image (fig.

S3E) was similar to the previous segment. The final anastomosis is shown in fig. S3F.

Supplementary Figures

Figure S1. Young’s modulus calculations for pig bowel and human bowel, with geometry and

applied forces kept constant. (A) The typical displacement was modeled by a cylindrical geometry 50

mm long, 1.3 mm thick, and 15 mm in diameter. The tissue was fixed at one end, as if the rest of

intestine was restricting movement. At the other end a virtual 3.0 sutures (diameter, 0.3 mm) were

placed at the calculated bite size and tensioned to 2 N, which induced a longitudinal deformation

exemplary of those experienced during surgery. Deformations were calculated by Solidworks

Simulation Package 2014. (B) Resulting deformations of human and porcine bowel based on Young’s

Modulus (34).

Figure S2. Deformation patterns for markers illustrate the uncertain nature of motion

propagation in soft tissue. In (A to D), three induced deformations (1–3) were quantified at five marker

locations on the tissue. The norm motion of the deformed marker, in green, resulted in further motion

propagated throughout the tissue as seen by deformations, in blue. First, a random motion in the range of

[2.6, 6.6] mm was applied to marker 2 (green) only, but motion propagated to the other markers (blue)

throughout the entire length of the tissue randomly. Second, a random motion of [2.6, 6.6] mm was

applied to marker 3 was moved and the entire bowel was rotated randomly between [-7.5, 17.5] degrees.

Finally, a random motion of [2.6, 6.6] mm was applied to marker 4 and similar to first deformation,

motion propagated randomly through the tissue. The proportionately induced motions on markers 2, 3,

and 4, respectively, were: (A) 2, 3, 1; (B) 3, 2, 1; (C) 1, 2, 3; (D) 2, 2, 2. Data are averages over 5 trials

for each of 5 experiments. (F) A representative deformation of markers (m1, …, m5) over time. The

initial markers at time t0 (blue), the first deformation on m2 at t1, the second deformation on m3 and

rotation of all markers at t2 (yellow), and the third rotation on m4 at t3 (purple) are shown.

Figure S3. Robotic construction of end-to-end anastomosis. (A) The porcine bowel was first staged

manually using four stay sutures, where the two inside walls of the two sections are paired closely. (B)

Five NIRF markers were distinct from the background and could be tracked. (C) The completed back

wall with sutures intact. (D) At the second phase, the tissue was re-staged such that the outside walls

were adjacent. (E) The NIR image for the outside wall. The markers were again distinctly visible. (F)

At the completion of the anastomosis, an instrument knot was tied to the unused section of the first

thread from the inside suture.

Supplementary Tables Table S1. Quantitative geometric quality of ex vivo linear suturing. Distances between consecutive

entry locations and consecutive exit locations were measured per stitch in each of the ex-vivo linear

tasks. Error from the mean refers to the value each stitch is away from the mean suture spacing.

Minimum suture spacing refers to the value for the shortest distance between consecutive stitches in all

tissue samples. A value of zero indicates that each in that modality had at least one set of consecutive

stitches that were 0 mm apart. Maximum suture spacing refers to the value for the furthest distance

between consecutive stitches in all tissue samples. Data are averages ± SD. P values compare the

respective surgical technique to STAR, using one-way ANOVA with post hoc Games-Howell tests.

Metric OPEN LAP RAS STAR

Suture spacing (mm) 2.99 ± 1.34

(P = 0.0100)

(n = 174)

3.55 ± 2.15

(P < 0.0001)

(n = 128)

2.90 ± 1.48

(P = 0.1079)

(n = 176)

2.60 ± 1.04

(n = 206)

Average error from

mean (Delta) (mm)

1.01 ± 0.88

(n = 174)

1.69 ± 1.32

(n = 128)

1.17 ± 0.90

(n = 176)

0.81 ± 0.65

(n = 206)

Median absolute deviation

(MAD) (mm)

1.00 1.00 1.00 0.00

Minimum suture spacing

(mm)

0.60 ± 0.80

(n = 5)

0.40 ± 0.80

(n = 5)

0.80 ± 1.17

(n = 5)

0

(n = 5)

Maximum suture spacing

(mm)

5.40 ± 1.36

(n = 5)

6.80 ± 2.40

(n = 5)

5.80 ± 0.75

(n = 5)

4.20 ± 0.40

(n = 5)

Table S2. Quantitative geometric quality of ex vivo end-to-end anastomosis. Distances between

consecutive entry locations and consecutive exit locations were measured per stitch in each of the ex-vivo

linear tasks. Error from the mean refers to the value each stitch is away from the mean suture. Minimum

suture spacing refers to the value for the shortest distance between consecutive stitches in all tissue

samples. A value of zero indicates that each in that modality had at least one set of consecutive stitches

that were 0 mm apart. Maximum suture spacing refers to the value for the furthest distance between

consecutive stitches in all tissue samples. Data are averages ± SD. P values compare the respective

surgical technique to STAR, using one-way ANOVA with post hoc Games-Howell tests.

Metric OPEN LAP RAS STAR

Suture spacing (mm) 2.69 ± 1.36

(P = 0.3693)

(n = 138)

4.28 ± 2.10

(P < 0.0001)

(n = 98)

3.04 ± 1.36

(P = 0.0004)

(n = 132)

2.46 ± 1.05

(n = 180)

Average error from mean

(Delta) (mm)

1.11 ± 0.79

(n = 138)

1.50 ± 1.48

(n = 98)

1.12 ± 0.77

(n = 132)

0.88 ± 0.57

(n = 180)

Median absolute deviation

(MAD) (mm)

1.00 1.00 1.00 1.00

Minimum suture spacing

(mm)

1.00 ± 0.63

(n = 5)

1.0 ± 1.55

2.0 (n = 5)

0.60 ± 0.49

(n = 5)

0

(n = 5)

Maximum suture spacing

(mm)

4.80 ± 0.98

(n = 5)

9.00 ± 1.79

(n = 5)

5.60 ± 0.49

(n = 5)

4.00 ± 0.00

(n = 5)

Movie S1. Supervised autonomous end-to-end intestinal anastomosis.

An end-to-end porcine intestinal anastomosis is demonstrated. Two open ends of porcine

intestine were staged for anastomosis in a similar manner to a common clinical set-up. The

intestinal edges were suspended using four anchor sutures, positioning two tubular ends into

relatively linear front and back edges. Fluorescent dye markers were placed at the front and back

edges to facilitate tracking using near-infrared fluorescence (NIRF) imaging in combination with

a pleoptic 3-D camera. The anastomosis began in the middle of the back edge with a knot

followed by running sutures to both corners transitioning into the front edge. Once suturing of

the back edge was completed, the front edge was positioned for completion of the anastomosis.

The role of human assistant here was to manage the loose suture ends while the robot completed

the task in supervised or in fully autonomous mode.