22University of University of Iowa Iowa

Iowa City, IA Iowa City, IA 5224652246

22University of University of Iowa Iowa

Iowa City, IA Iowa City, IA 5224652246

William E. HigginsWilliam E. Higgins1,21,2 Anthony J. SherbondyAnthony J. Sherbondy11

James P. HelfertyJames P. Helferty11 Atilla P. KiralyAtilla P. Kiraly11

Geoffrey McLennanGeoffrey McLennan22 Eric A. HoffmanEric A. Hoffman22

Janice Z TurlingtonJanice Z Turlington11

William E. HigginsWilliam E. Higgins1,21,2 Anthony J. SherbondyAnthony J. Sherbondy11

James P. HelfertyJames P. Helferty11 Atilla P. KiralyAtilla P. Kiraly11

Geoffrey McLennanGeoffrey McLennan22 Eric A. HoffmanEric A. Hoffman22

Janice Z TurlingtonJanice Z Turlington11

RSNA 2000RSNA 2000Chicago, ILChicago, IL

26 November 200026 November 2000

RSNA 2000RSNA 2000Chicago, ILChicago, IL

26 November 200026 November 2000

11Penn State Penn State University University University University

Park, PA Park, PA 1680216802

11Penn State Penn State University University University University

Park, PA Park, PA 1680216802

1.1. Introduce the feasibility of PC-based Introduce the feasibility of PC-based virtual endoscopy for both 3D CT virtual endoscopy for both 3D CT assessment and live bronchoscopy.assessment and live bronchoscopy.

2.2. Demonstrate the concept of a multimedia Demonstrate the concept of a multimedia case study for CT-based report case study for CT-based report generation and bronchoscopic guidance.generation and bronchoscopic guidance.

3.3. Describe a method for linking 3D CT Describe a method for linking 3D CT data to live bronchoscopic video.data to live bronchoscopic video.

1.1. Introduce the feasibility of PC-based Introduce the feasibility of PC-based virtual endoscopy for both 3D CT virtual endoscopy for both 3D CT assessment and live bronchoscopy.assessment and live bronchoscopy.

2.2. Demonstrate the concept of a multimedia Demonstrate the concept of a multimedia case study for CT-based report case study for CT-based report generation and bronchoscopic guidance.generation and bronchoscopic guidance.

3.3. Describe a method for linking 3D CT Describe a method for linking 3D CT data to live bronchoscopic video.data to live bronchoscopic video.

(1 (1 of of 32)32)(1 (1 of of 32)32)

1.1. Overview of virtual bronchoscopy and our Overview of virtual bronchoscopy and our system (Virtual Navigator)system (Virtual Navigator)

2.2. Stage-1 CT-only Analysis: Stage-1 CT-only Analysis: Human caseHuman case

3.3. Stage-2 Bronchoscopy examples:Stage-2 Bronchoscopy examples:a.a. Human caseHuman caseb.b. Phantom and animal studiesPhantom and animal studies

1.1. Overview of virtual bronchoscopy and our Overview of virtual bronchoscopy and our system (Virtual Navigator)system (Virtual Navigator)

2.2. Stage-1 CT-only Analysis: Stage-1 CT-only Analysis: Human caseHuman case

3.3. Stage-2 Bronchoscopy examples:Stage-2 Bronchoscopy examples:a.a. Human caseHuman caseb.b. Phantom and animal studiesPhantom and animal studies

(2 (2 of of 32)32)(2 (2 of of 32)32)

virtual copy of chest anatomyvirtual copy of chest anatomy virtual copy of chest anatomyvirtual copy of chest anatomy

Input:Input:Input:Input:high-resolution 3D radiologic chest imagehigh-resolution 3D radiologic chest imagehigh-resolution 3D radiologic chest imagehigh-resolution 3D radiologic chest image

permits unlimited permits unlimited

“exploration”“exploration” no risk to patientno risk to patient

permits unlimited permits unlimited

“exploration”“exploration” no risk to patientno risk to patient

Explore:Explore:Explore:Explore:the virtual anatomy using computerthe virtual anatomy using computerthe virtual anatomy using computerthe virtual anatomy using computer

(3 (3 of of 32)32)(3 (3 of of 32)32)

Permit CT-only analysisPermit CT-only analysis

• No link to No link to follow-onfollow-on live bronchoscopy live bronchoscopy

Limited quantitative path planning to interesting sites Limited quantitative path planning to interesting sites

Do not provide complete examination packageDo not provide complete examination package

Often require expensive computersOften require expensive computers

Permit CT-only analysisPermit CT-only analysis

• No link to No link to follow-onfollow-on live bronchoscopy live bronchoscopy

Limited quantitative path planning to interesting sites Limited quantitative path planning to interesting sites

Do not provide complete examination packageDo not provide complete examination package

Often require expensive computersOften require expensive computers

(4 (4 of of 32)32)(4 (4 of of 32)32)

(5 (5 of of 32)32)(5 (5 of of 32)32)

Complete CT Complete CT examination examination

Guide live Guide live bronchoscopybronchoscopy

Automate Automate steps in CT steps in CT assessment assessment

Inexpensive, Inexpensive, PC-basedPC-based

Complete CT Complete CT examination examination

Guide live Guide live bronchoscopybronchoscopy

Automate Automate steps in CT steps in CT assessment assessment

Inexpensive, Inexpensive, PC-basedPC-based

Build with Graphics/Processing ToolsBuild with Graphics/Processing Tools Build with Graphics/Processing ToolsBuild with Graphics/Processing Tools

3D CT assessment3D CT assessment3D CT assessment3D CT assessment

Guide bronchoscopyGuide bronchoscopyGuide bronchoscopyGuide bronchoscopy Supplemental plan Supplemental plan Supplemental plan Supplemental plan

Multimedia report Multimedia report Multimedia report Multimedia report

(6 (6 of of 32)32)(6 (6 of of 32)32)

3.3. Reporting AbstractionsReporting Abstractions SnapshotsSnapshots PlotsPlots MoviesMovies Case NotesCase Notes MeasurementsMeasurements

3.3. Reporting AbstractionsReporting Abstractions SnapshotsSnapshots PlotsPlots MoviesMovies Case NotesCase Notes MeasurementsMeasurements

1.1. Data SourcesData Sources 3D CT Image3D CT Image BronchoscopicBronchoscopic VideoVideo

1.1. Data SourcesData Sources 3D CT Image3D CT Image BronchoscopicBronchoscopic VideoVideo

2.2. Data AbstractionsData Abstractions Root SiteRoot Site Key SitesKey Sites PathsPaths TreeTree

2.2. Data AbstractionsData Abstractions Root SiteRoot Site Key SitesKey Sites PathsPaths TreeTree

(7 (7 of of 32)32)(7 (7 of of 32)32)

VirtualscopeVirtualscopeVirtualscopeVirtualscope

3D Surface Tool3D Surface Tool3D Surface Tool3D Surface ToolStudy ManagerStudy ManagerStudy ManagerStudy Manager

Projection ToolsProjection ToolsProjection ToolsProjection Tools

(8 (8 of of 32)32)(8 (8 of of 32)32)

Sliding Slab Depth ToolsSliding Slab Depth ToolsSliding Slab Depth ToolsSliding Slab Depth Tools

Plot ToolPlot ToolPlot ToolPlot ToolOblique Cross SectionsOblique Cross SectionsOblique Cross SectionsOblique Cross Sections

Slicer Tools (MPR Views)Slicer Tools (MPR Views)Slicer Tools (MPR Views)Slicer Tools (MPR Views)

(9 (9 of of 32)32)(9 (9 of of 32)32)

CT-Video Live Match ToolCT-Video Live Match ToolCT-Video Live Match ToolCT-Video Live Match ToolCube ToolCube ToolCube ToolCube Tool

Tube Viewer Tube Viewer Tube Viewer Tube Viewer Notes ToolNotes ToolNotes ToolNotes Tool

(10 (10 of of 32)32)(10 (10 of of 32)32)

Examine case and plan Examine case and plan bronchoscopybronchoscopyExamine case and plan Examine case and plan bronchoscopybronchoscopy

Virtual CT guidance of live Virtual CT guidance of live bronchoscopybronchoscopy

Virtual CT guidance of live Virtual CT guidance of live bronchoscopybronchoscopy

Stage 1:Stage 1:Stage 1:Stage 1:CT AssessmentCT AssessmentCT AssessmentCT Assessment

Stage 2:Stage 2:Stage 2:Stage 2:BronchoscopyBronchoscopyBronchoscopyBronchoscopy

(11 (11 of of 32)32)(11 (11 of of 32)32)

1.1.Create new Case Study.Create new Case Study.2.2.Compute guidance data.Compute guidance data.3.3.Build complete Case Study. Build complete Case Study.

1.1.Create new Case Study.Create new Case Study.2.2.Compute guidance data.Compute guidance data.3.3.Build complete Case Study. Build complete Case Study.

Stage 1:Stage 1:Stage 1:Stage 1:CT AssessmentCT AssessmentCT AssessmentCT Assessment

1.1.Load Case Study.Load Case Study.2.2.Set up graphical tools.Set up graphical tools.3.3.Perform virtual-guided bronchoscopy.Perform virtual-guided bronchoscopy.

1.1.Load Case Study.Load Case Study.2.2.Set up graphical tools.Set up graphical tools.3.3.Perform virtual-guided bronchoscopy.Perform virtual-guided bronchoscopy.

Stage 2:Stage 2:Stage 2:Stage 2:BronchoscopyBronchoscopyBronchoscopyBronchoscopy

(12 (12 of of 32)32)(12 (12 of of 32)32)

a.a. Build Case Study registry Build Case Study registry a.a. Build Case Study registry Build Case Study registry BB BB

Create CaseCreate Case Create CaseCreate Case

b.b. Set up for computing guidance data Set up for computing guidance data b.b. Set up for computing guidance data Set up for computing guidance data SS SS

Set upSet up Set upSet up

1.1. Create new Case Study.Create new Case Study.1.1. Create new Case Study.Create new Case Study.

2.2. Automatically Automatically compute compute guidance data.guidance data.

2.2. Automatically Automatically compute compute guidance data.guidance data.

3.3. Build complete Build complete Case Study. Case Study.

3.3. Build complete Build complete Case Study. Case Study.

AA AA

BB BB

(13 (13 of of 32)32)(13 (13 of of 32)32)

DetermDetermine ine

global global 3D 3D

interior interior detail detail

DetermDetermine ine

global global 3D 3D

interior interior detail detail

Travel Travel through through virtual virtual airway airway paths.paths.

Travel Travel through through virtual virtual airway airway paths.paths.

View View structurstructur

e e relations relations

within within their their

environenvironment. ment.

View View structurstructur

e e relations relations

within within their their

environenvironment. ment.

Study Study interior interior

path path dynamidynami

cs.cs.

Study Study interior interior

path path dynamidynami

cs.cs.

Extract Extract paths paths

through through airways.airways.

Extract Extract paths paths

through through airways.airways.

Render Render airway airway

tree with tree with paths, paths,

traverse traverse pathwaypathway

s.s.

Render Render airway airway

tree with tree with paths, paths,

traverse traverse pathwaypathway

s.s.

(14 (14 of of 32)32)(14 (14 of of 32)32)

Patient underwentPatient underwent EBCT scan (Electron Beam):EBCT scan (Electron Beam):

•single 20-sec breath-holdsingle 20-sec breath-hold•133 contiguous slices 133 contiguous slices

Reconstructed 3D CT image:Reconstructed 3D CT image:•Slice = 512X512 voxelsSlice = 512X512 voxels•Slice thickness =1.5mmSlice thickness =1.5mm•axial-plane [x-y] resolution = 0.586mm.axial-plane [x-y] resolution = 0.586mm.

Patient underwentPatient underwent EBCT scan (Electron Beam):EBCT scan (Electron Beam):

•single 20-sec breath-holdsingle 20-sec breath-hold•133 contiguous slices 133 contiguous slices

Reconstructed 3D CT image:Reconstructed 3D CT image:•Slice = 512X512 voxelsSlice = 512X512 voxels•Slice thickness =1.5mmSlice thickness =1.5mm•axial-plane [x-y] resolution = 0.586mm.axial-plane [x-y] resolution = 0.586mm.

Virtual Navigator showsVirtual Navigator shows•Characteristics of collapseCharacteristics of collapse•Condition of extended bronchial pathways Condition of extended bronchial pathways

Virtual Navigator showsVirtual Navigator shows•Characteristics of collapseCharacteristics of collapse•Condition of extended bronchial pathways Condition of extended bronchial pathways

(15 (15 of of 32)32)(15 (15 of of 32)32)

(16 (16 of of 32)32)(16 (16 of of 32)32)

MPR MPR ViewsViewsIndicaIndica

teteGlobal Global

3D3Dpositiopositio

nn

MPR MPR ViewsViewsIndicaIndica

teteGlobal Global

3D3Dpositiopositio

nn

DepthDepth--

weighweightedtedSlab Slab

showsshowsgeomgeometryetry

DepthDepth--

weighweightedtedSlab Slab

showsshowsgeomgeometryetry

Same 3D site focused on by all Same 3D site focused on by all tools. tools. Same 3D site focused on by all Same 3D site focused on by all tools. tools.

(17 (17 of of 32)32)(17 (17 of of 32)32)

Selected 3D Selected 3D site (BLUE site (BLUE

DOT)DOT)Highlighted Highlighted

ononFive Five

different different tools at tools at once. once.

Selected 3D Selected 3D site (BLUE site (BLUE

DOT)DOT)Highlighted Highlighted

ononFive Five

different different tools at tools at once. once.

Bottom to top view of collapse

Site #99Site #99

Near carina, leaving collapse

Approaching collapse

Site #20

Site #47

Within tracheal collapse

Site #86Leaving trachea

(18 (18 of of 32)32)(18 (18 of of 32)32)

drop in areadrop in areadrop in areadrop in area

Approaching collapse

Site #20Site #20

Within tracheal collapse

Site #47Site #47

Plot clearly shows section Plot clearly shows section where blockage occurs where blockage occurs

Plot clearly shows section Plot clearly shows section where blockage occurs where blockage occurs

(19 (19 of of 32)32)(19 (19 of of 32)32)

Pathology documented by Pathology documented by captured Depth-Slab captured Depth-Slab

snapshots.snapshots.

Pathology documented by Pathology documented by captured Depth-Slab captured Depth-Slab

snapshots.snapshots.

Extent of collapse shown in Extent of collapse shown in rendered Airway tree.rendered Airway tree.

Extent of collapse shown in Extent of collapse shown in rendered Airway tree.rendered Airway tree.

(20 (20 of of 32)32)(20 (20 of of 32)32)

Virtual Navigator showsVirtual Navigator shows•Details of existing stentDetails of existing stent•Basis for intervention analysis Basis for intervention analysis (laser therapy was performed)(laser therapy was performed)

Virtual Navigator showsVirtual Navigator shows•Details of existing stentDetails of existing stent•Basis for intervention analysis Basis for intervention analysis (laser therapy was performed)(laser therapy was performed)

Patient underwentPatient underwent EBCT scan (Electron Beam):EBCT scan (Electron Beam):

•single 20-sec breath-holdsingle 20-sec breath-hold•133 contiguous slices 133 contiguous slices

Reconstructed 3D CT image:Reconstructed 3D CT image:•Slice = 512X512 voxelsSlice = 512X512 voxels•Slice thickness =1.5mmSlice thickness =1.5mm•axial-plane [x-y] resolution = axial-plane [x-y] resolution =

0.586mm.0.586mm.

Patient underwentPatient underwent EBCT scan (Electron Beam):EBCT scan (Electron Beam):

•single 20-sec breath-holdsingle 20-sec breath-hold•133 contiguous slices 133 contiguous slices

Reconstructed 3D CT image:Reconstructed 3D CT image:•Slice = 512X512 voxelsSlice = 512X512 voxels•Slice thickness =1.5mmSlice thickness =1.5mm•axial-plane [x-y] resolution = axial-plane [x-y] resolution =

0.586mm.0.586mm.

(21 (21 of of 32)32)(21 (21 of of 32)32)

(22 (22 of of 32)32)(22 (22 of of 32)32)

Stent Stent visiblevisibleIn this In this andandother other viewsviews

Stent Stent visiblevisibleIn this In this andandother other viewsviews

Same 3D site focused on by all Same 3D site focused on by all tools. tools. Same 3D site focused on by all Same 3D site focused on by all tools. tools.

(23 (23 of of 32)32)(23 (23 of of 32)32)

Stent Stent encroachencroach

ing on ing on main main

carinacarina

Stent Stent encroachencroach

ing on ing on main main

carinacarina

3.3. Perform Perform virtual-guided virtual-guided bronchoscopy.bronchoscopy.

3.3. Perform Perform virtual-guided virtual-guided bronchoscopy.bronchoscopy.

2.2. Set up Set up graphical tools.graphical tools.

2.2. Set up Set up graphical tools.graphical tools.

1.1. Load Case Load Case Study. Study.

1.1. Load Case Load Case Study. Study.

LL LL

Load CaseLoad Case Load CaseLoad Case SS SS

Graphical ToolsGraphical Tools Graphical ToolsGraphical Tools PP PP

Virtual GuidanceVirtual Guidance Virtual GuidanceVirtual Guidance

(24 (24 of of 32)32)(24 (24 of of 32)32)

2.2.LIVE LIVE bronchoscope bronchoscope videovideo

2.2.LIVE LIVE bronchoscope bronchoscope videovideo

Video Match Tool Video Match Tool shows a matched shows a matched point between point between

Video Match Tool Video Match Tool shows a matched shows a matched point between point between

3.3.Corresponding Corresponding videobronchosvideobronchoscopy copy ((ROI ROI superimposed)superimposed)

3.3.Corresponding Corresponding videobronchosvideobronchoscopy copy ((ROI ROI superimposed)superimposed)

Virtual data Virtual data guidesguides

airway traversal.airway traversal.

Virtual data Virtual data guidesguides

airway traversal.airway traversal.

Coronal Projection Coronal Projection shows extracted shows extracted

airway treeairway tree

Coronal Projection Coronal Projection shows extracted shows extracted

airway treeairway tree

1.1. CT rendering of CT rendering of airway region airway region((ROI rendered)ROI rendered)

1.1. CT rendering of CT rendering of airway region airway region((ROI rendered)ROI rendered)

(25 (25 of of 32)32)(25 (25 of of 32)32)

1.1.PhantoPhantomm

2.2.Animal Animal

3.3.HumaHumann

1.1.PhantoPhantomm

2.2.Animal Animal

3.3.HumaHumann

Rubber Rubber phantomphantomRubber Rubber phantomphantom

Experimental set-Experimental set-up:up: physician was blind to physician was blind to phantomphantom

Experimental set-Experimental set-up:up: physician was blind to physician was blind to phantomphantom

(26 (26 of of 32)32)(26 (26 of of 32)32)

ExtracteExtracted tree d tree and and

pathspaths

ExtracteExtracted tree d tree and and

pathspaths

MatcheMatched video d video frame frame with with ROIROI

MatcheMatched video d video frame frame with with ROIROI

RegistereRegisteredd

virtualvirtualshotshot

RegistereRegisteredd

virtualvirtualshotshot

(27 (27 of of 32)32)(27 (27 of of 32)32)

Physician #1 (trial 1)Physician #1 (trial 1) Physician #1 (trial 2)Physician #1 (trial 2) Physician #2Physician #2

Distance (mm)Distance (mm) Time sec.Time sec. Distance (mm)Distance (mm) Time sec.Time sec. Distance (mm)Distance (mm) Time sec.Time sec.

2.182.18 12.61312.613 1.731.73 9.6729.672 2.012.01 10.9110.91

1.091.09 8.8658.865 0.970.97 8.7898.789 0.890.89 5.3255.325

Average biopsy error: 1.98 mm Average biopsy error: 1.98 mm

Average match time: 11.065 sec.Average match time: 11.065 sec.

Average biopsy error: 1.98 mm Average biopsy error: 1.98 mm

Average match time: 11.065 sec.Average match time: 11.065 sec.

AverageAverage

Std DevStd Dev

Distance and time measured to match each ROI target.Distance and time measured to match each ROI target.Distance measured from line extrapolated from the needle direction to metal bead edge.Distance measured from line extrapolated from the needle direction to metal bead edge.Note:Note:

(28 (28 of of 32)32)(28 (28 of of 32)32)

RegistereRegisteredd

virtual virtual shotshot

RegistereRegisteredd

virtual virtual shotshot

LiveLiveEndoscoEndosco

picpicvideovideo

LiveLiveEndoscoEndosco

picpicvideovideo

MatcheMatched video d video frame frame with with ROIROI

MatcheMatched video d video frame frame with with ROIROI

(29 (29 of of 32)32)(29 (29 of of 32)32)

Snapshots are misaligned to compensate for Snapshots are misaligned to compensate for differing placement during CT scanning.differing placement during CT scanning.

Note:Note:

Actual siteActual site after guided after guided dye marker placement.dye marker placement.Actual siteActual site after guided after guided dye marker placement.dye marker placement.

Planned sitePlanned site from CT analysis. from CT analysis.Planned sitePlanned site from CT analysis. from CT analysis.

(30 (30 of of 32)32)(30 (30 of of 32)32)

(31 (31 of of 32)32)(31 (31 of of 32)32)

(32 (32 of of 32)32)(32 (32 of of 32)32)

Bronchoscope Bronchoscope video video

matched to matched to rendered CT rendered CT during live during live procedure.procedure.

Bronchoscope Bronchoscope video video

matched to matched to rendered CT rendered CT during live during live procedure.procedure.

National Center National Center for for Research Research Resources Resources of theof the NIH NIH

(grant RR11800)(grant RR11800)

National Center National Center for for Research Research Resources Resources of theof the NIH NIH

(grant RR11800)(grant RR11800)

Whitaker FoundationWhitaker FoundationWhitaker FoundationWhitaker Foundation

National Cancer Institute National Cancer Institute of theof the NIH NIH (grant CA74325)(grant CA74325)National Cancer Institute National Cancer Institute of theof the NIH NIH (grant CA74325)(grant CA74325)

(End)(End)(End)(End)