3D Reconstruction of Anatomical Structures from Serial EM images

Biological Motivation• Reconstruct 3D structures

from serial stack of EM images to understand– Distribution of cell type and

organelles.– Connectivity and

vasculature.• Requires

– Tracing of cell membranes, organelles, blood vessels, etc.

– Identification of structures for serial correspondence.

• Currently, reconstruct only a small fraction of volume (very few objects).– Time consuming (~20hours

per specimen).– Wealth of information in

surround structures not utilized.

Synapse Web, Kristen M. Harris, PI

http://synapses.clm.utexas.edu/

Serial TEM Dataset• typical volume:

– 20-50 slices – 8 x 5 x 0.05 m per slice– 40 to 100 m3 volume– fly brain volume: 0.1mm3

• resolution: – xy: 2.6 nm/pixel

(350~400 pixels/ m) – z: 0.05 m ~20 pixels

apart• storage size:

– small volume: ~100MB– fly brain: 3.2x1014 pixels– compression of 4 would

result in 73 terabytes. – (source: Fiala, BU)

serial direction

Data from Synapse Web, Kristen M. Harris, PI

http://synapses.clm.utexas.edu/

Challenges for Computer Vision• Segmenting objects

– EM images are inherently noisy.– Gaps in membrane.– Adjacent structures share weak

membrane boundary.– Organelles too small to use

common descriptors such as texture.

• Identification and correspondence– Structures can merge, split,

appear, or disappear (yellow arrow).

– z-axis structures (red) are easier to maintain correspondences than lateral structures (green).

– z resolution much lower than xy resolution (large changes serially).

– Automatic registration difficult (no ground truth)

• 3D reconstruction– good software available, but

getting to this step is the challenge.

Preliminary 2D Segmentation

• Parametric snakes• Red-initial contour• Green-final contour• Highly sensitive to

initialization (bottom)

• Automatic initialization is a big challenge.

Preliminary 2D Segmentation

• Geometric active contours.

• Provides topological flexibility.

• Less sensitive to initialization.

• Adjacent objects often merge (bottom).

Preliminary 2D Segmentation

• Level set with elastic edge interaction*

• Zero level contour of v provides “good” initialization.

• Still many problems.

*Xiang et al. J. Comp. Phys. 2006

@Á@t

=µπr ¢

r Ájr Áj

+ v¶jr Áj

v = ¡Z

≠

r ¢r (G ? I )r3

dxdy

@Á@t

=µπr ¢

r Ájr Áj

+ v¶jr Áj

v = ¡Z

≠

r ¢r (G ? I )r3

dxdy

Preliminary 2D Segmentation

• Previous method produces binary masks of cross sections.

• Correspondences can be made based on distance and area of overlap.

• Inconsistencies occur often (green)

Preliminary 3D Reconstruction

• Reconstructing “everything” at the same time produces confusing volume.

• Inconsistencies in segmentation and correspondence produce artifacts.

Open Issues

• 2D Segmentation challenges– Automatic initialization.– Segmenting adjacent objects sharing weak

edges.– Noise.

• Cross section correspondence– Identifying objects (synapse, mitochondria,

etc.)– Tracking contours serially and detecting

merging/splitting events.– Automatic registration.

• Current Work: simultaneous segmentation and correspondence.