TBA

#23

GE Corporate R&D

Niskayuna, NY

lorensen@crd.ge.com

Unification of Vision, Geometry and Graphics

Through Toolkits

Bill Lorensen

GE Corporate R&D

Niskayuna, NY

lorensen@crd.ge.com

What is a Toolkit?

Mathematics+

Algorithms+

Software

Edelsbrunner, 2001

Dual Interests

Marching Cubes 1984

Baseball Visualization 1989

Stream Polygons - 1991

Triangle Decimation - 1992

IEEE CG&A 1992

Swept Surfaces 1993

Removal Removal PathPath

Swept Swept SurfaceSurface

Virtual Endoscopy 1994

Creating Models from Segmented Medical Data

Surface and Volume Rendering

Hypothesis

Many real world problems cannot be solved by a single discipline

Core Technologies for 3D Medical Image Analysis

• Registration– Intra-modality (MRI to MRI, CT to CT)– Inter-modality (MRI to PET)– Model to Modality (Atlas to MRI)– Metadata to Modality (Clinical data,

biochip to MRI/CT)

• Filters– Edge preserving– Noise reduction– Non uniform intensity correction

• Segmentation– Edge detection– Region growing– Multi-channel

• Pattern Recognition– Tissue classification

• Visualization– Surface / volume rendering

– Fusion

• Quantification– Area, volume, shape

• Change detection– Longitudinal tracking

– Signal variation

• Information Analysis/Visualization

Discipline-specific Toolkits

• Use “best of breed” algorithms implemented by domain experts– Point matching– Voronoi diagram computation– Registration– Pose estimation– Isosurface extraction– Mathematical morphology– Skeletonization– Subdivision surfaces– Similarity measures– Surface simplification– Geometric compression

Discipline-specific Toolkits

• Examples– vtk, The Visualization Toolkit– Open Inventor, Graphics– Insight, Segmentation and Registration– CGAL, Computational Geometry– vxl, Image Understanding– Khoros, Image Processing

vtk, The Visualization Toolkit

• Open source toolkit for scientific visualization, computer graphics, and image processing

• C++ Class Library• 250,000 Lines of Code

– (~120,000 executable)• 20+ developers• 8 years of development• 1000 user mailing list

public.kitware.com/VTK

Insight Segmentation and Registration Toolkit

What is it?

• A common Application Programmers Interface (API).– A framework for software development– A toolkit for registration and segmentation– An Open Source resource for future research

• A validation model for segmentation and registration.– A framework for validation development– Assistance for algorithm designers– A seed repository for validated segmentations

Who’s sponsoring it?

The NationalScienceFoundation

The NationalInstitute for Dental and Craniofacial Research

The National Institute of Neurological Disorders and Stroke

$7.5 million, 3 year contract

Who’s creating it?

Contractor Roles

• GE CRD/Brigham and Womens– Architecture, algorithms, testing, validation

• Kitware– Architecture, user community support

• Insightful (formerly MathSoft)/UPenn– Statistical segmentation, mutual information registration, deformable

registration, level sets– Beta test management

• Utah– Level sets, low level image processing

• UNC/Pitt– Image processing, registration, high-dimensional segmentation

• UPenn/Columbia– Deformable surfaces, fuzzy connectedness, hybrid methods

Toolkit Requirements

• Shall handle large datasets– Visible Human data on a 512MB PC

• Shall run on multiple platforms– Sun, SGI, Linux, Windows

• Shall provide multiple language api’s• Shall support parallel processing• Shall have no visualization system

dependencies• Shall support multi-dimensional images• Shall support n-component data

Insight - Schedule

• Alpha Release, April 4, 2001.– Source code snapshot– Some non-consortium participation

• Limited Public Alpha Version, Aug 8, 2001.• Public Beta Release, December 15, 2001.• Software Developer’s Consortium Meeting

– Nov. 8-9, 2001, NLM, Bethesda.

www.itk.org

Testing Design

• Distributed testing– Developers and users must be able to easily

contribute testing results– Pulled together in a central dashboard

• Separate data from presentation• Cross-platform solution• Strive to have the same code tested in all

locations

Using vtk and Insight

Registration of Volumetric Medical Data

Mutual Information

• Computes “mutual information” between two datasets, a reference and target– MI(X,Y) = H(X) + H(Y) – H(X,Y)

• Small parameter set• Developed by Sandy Wells (BWH) and Paul Viola

(MIT) in 1995• Defacto standard for automatic, intensity based

registration

Insight Mutual Information Registration

• There is no MI open source implementation• The Insight Registration and Segmentation

Toolkit has an implementation• GE and Brigham as Insight contractors have

early access to the code• Code was developed at MathSoft (now called

Insightful)• GE was able to “guide” development with

input from Sandy Wells

Longitudinal MRI Study

• Register multiple volumetric MRI datasets of a patient taken over an extended time

• Create a batch processing facility to process dozens of datasets

• Resample the datasets

Approach

• Validate the algorithm• Pick a set of parameters that can be used

across all the studies• For each pair of datasets

– Perform registration– Output a transform

• View the resampled source dataset in context with the target dataset

Division of Labor

vtk

itk

vtk

Read data

Normalize data

Export data

Import Data

Register

Report transform

Read data

Reslice

Display

MRIRegistration.cxx

MultiCompare.tcl

The Pipeline

ImageReader ImageCast ImageShiftScale

ImageStatistics

ImageShrink3D

ImageExportImportImage

ImageToImageRigidMutualInformationGradientDescentRegistration

vnl_quaternion Matrix4x4

Oregon Data

• 25 Registrations• 13 Subjects• Qualitative comparison• One set of parameters for all studies

Longitudinal MRI No Registration

Checkerboard

SourceOriginalimage

Difference

TargetOriginal image

Longitudinal MRI Registration

Checkerboard

SourceOriginalimage

Difference

TargetOriginal image

Multi Field MRI Data

• Register 1.5T and 3T to 4T data• Resampled 1.5T and 3T to correspond to the

4T sampling• Volume rendering of the 3 datasets from the

same view

1.5T vs 4T MRI No Registration

Checkerboard

SourceOriginalImage

Difference

TargetOriginal Image

1.5T vs 4T MRI Registration

Checkerboard

SourceOriginalImage

Difference

TargetOriginal Image

3D Visualization of the same subjectScanned with different MR field Strengths

4T

3T 1.5T

All Registered To 4T

CT Lung Longitudinal Study

• Register two CT exams of the same patient taken at two different times

• Side-by-side synchronized view for visual comparison

Lung CT No Registration

Checkerboard

SourceOriginalImage

Difference

TargetOriginal Image

Lung CT Registration

Checkerboard

SourceOriginalImage

Difference

TargetOriginal Image

microPet/Volume CT

Back to the Software

Why Now?

• Internet enables distributed software development

• There are some successful Open Source projects

• A basic set of algorithms (and sometimes mathematics) exist

• Light weight software engineering processes exist– Low investment to support software development– Minimally invasive

Software Trends

Lightweight Software Engineering Processes

IEEE Computer October, 1999

Extreme Programming

Extreme Testing

Continuous Testing

Insight Project Management

• Robust code repository (cvs)• Active mailing list (mailman)• Automated documentation (doxygen)• Stable, cross platform build environment (cmake)• Weekly t-cons• Stable nightly build and test (300 builds)• Continuous build• Stable nightly dashboard (dart)• Quarterly face-to-face developer meetings• Semi-annual project meetings

Recipe for Success

• Vision

• Openness

• Community

• Strong core team

• Core Architecture

• Funding

Unification of Vision, Geometry and Graphics

Through Toolkits

Bill Lorensen

GE Corporate R&D

Niskayuna, NY

lorensen@crd.ge.com