62
Lifesaving visualizations Interactive visualization in the design of Decision Support Systems Robert G. Belleman, PhD Computational Science University of Amsterdam Email: [email protected]
Lifesaving visualizations
Interactive visualization in the design of Decision Support Systems
Robert G. Belleman, PhD
Computational Science
University of Amsterdam
Email : [email protected]
About us
• Scientific Visualization and
Virtual Reality team
– part of Computational Science at UvA/IvI
– close collaboration with SARA
• Research theme: interactive visual
exploration
– Software solutions and architectures,
Problem Solving Environments,
Interactive graphics devices
• Application areas: computational science
– (astro)physics, medicine, biology, finance,
architecture, computer science, …
Software solutions and architectures
Software solutions and architectures
Visualization as a Service: remote visualization on high performance computing resources
High performance visualization on low performance devices
Interactive Problem Solving Environments
Visual exploration of large parameter/data spaces
Integrated services for visualization and on-demand simulation
Augmented Reality visualization of DTI fiber tracks on a TabletPC
Interactive parameter exploration of the ACGT Oncosimulator in the OncoRecipesheet
Interactive graphics devices
SARA Tiled Panel Display
UvA DRIVE UvA PSS UvA WiVR UvA MTT
Microsoft Surface
Overview
Outline:
The role of interactive visualization in Decision Support Systems
Lessons learned
Applications:
patient specific disease treatment: simulated vascular reconstruction, cancer treatment
prevention and/or containment of flood disasters
social-genetic analysis of infectious diseases
Decision support
Characteristics:
Non-trivial problems
Presentation of current state
Information integration and presentation
Human-in-the-loop
Domain expert
What-if … ?
Challenges:
Data deluge
Heterogeneous, complex, dynamic
Searching large parameter spaces
Urgency
Simulated vascular reconstruction
Vascular disease
Stenosis: Treatment: thrombolysis, balloon angioplasty, stent placement, endarterectomy, bypass
Aneurysm: Treatment: shunt, bypass
The problem
Best treatment often not obvious
Human body is a complex structure
A treatment is not always best under all situations
Angio w/ Fem-Fem &
Fem-Pop
AFB w/ E-S Prox .
Anast .
Angio w/ Fem-Fem
AFB w/ E-E Prox .
Anast .
Preop
Traditional treatment of vascular disease
Interactive simulated vascular surgery
Simulated Vascular Reconstruction
Simulated vascular reconstruction
Patient specific angiography data
Fluid flow simulation software
Simulation of reconstructive surgical procedure in VR
Interactive visualization of simulation results in VR
Pre-operative planning
Explore multiple reconstruction procedures
Parallel fluid flow simulation
Lattice Boltzmann Method (LBM)
Lattice based particle method
Regular lattice, similar to CT or MRI datasets
Allows irregular 3D geometry
Allows changes at run-time
Velocity, pressure and shear stress calculated from particle densities
Non-compressible homogeneous fluid, laminar flow
Spatial and temporal locality
Ideal for parallel implementation
CAVE, Personal Space Station (PSS)
Visualize simulation results
Flow field, pressure, shear stress, at run-time
Interactive grid editing
Simulate vascular reconstruction procedure
Interactive exploration
VR interaction to locate regions of interest
Interactive exploration in VR
CAVE, Personal Space Station (PSS)
Visualize simulation results
Flow field, pressure, shear stress, at run-time
Interactive grid editing
Simulate vascular reconstruction procedure
Interactive exploration
VR interaction to locate regions of interest
Interactive exploration in VR
Results
Major achievements:
Architecture for distributed interactive simulation and visualization
Complicated framework using the High Level Architecture (HLA)
Interactive visualization in VR environments
Lessons learned:
High visual impact
Good PR...
Domain experts are hard to please...
Not considered “intuitive”, or even easy to use
“VR is too difficult to use”
“CAVE is too immobile”
ACGT: Advancing Clinico Genomic Trials on Cancer
In Silico Oncology
Interactive 3D/4D visualization of In Silico Oncology simulations Collaboration ICCS/NTUA and UvA
3D volumes
96 x 96 x 96 voxels per timestep
Tens to hundreds of timesteps
Tumour “layers”:
Proliferating cell layer
Dormant cell layer
Dead cell layer (necrotic core)
Treatment depends on: Patient specific pathology
Type of treatment (25+ parameters)
Cell behaviour (50+ parameters)
PSNC
Oncosimulator
Visualization services
Grid Resource Management System
(GRMS)
Data Management System (DMS)
Recipesheet/ web portal
Data request
s
Simulation parameters
Job identifier
Job submission
Job monitoring
Visualization parameters
Images
• job identifier • vis parameters • …
References to: • DICOM images (before/after CT) by patient id • segmentation data by patient id • (intermediate) oncosimulator data by job id
(intermediate) oncosimulator
data
Data
1 3
2 4
8
9
10
11
Segmentation data
Data request
5
6
7
user
• simulation parameters
Reference to: • segmentation data by patient id
Oncosimulator demonstration: component collaboration diagram
Interactive parameter exploration
Automatic job execution on grid architecture
Interactive visualization as a service used by Web based applications (portal)
Hokaido RecipeSheet: “Parameter space exploration”
Personal Space Station: “Reach in and touch your data”
Web based visualization Personal Space Station (PSS) Hokaido RecipeSheet
OncoRecipesheet
Interactive parameter exploration of the ACGT Oncosimulator in the OncoRecipesheet (with Aran Lunzer, Hokaido Univ, Japan).
Results
Major achievements:
Parameter exploration on a Grid-based architecture Automatic job execution
Web-based visualization services “Visualization as a Service”
Serves multiple visualization front-ends from the same set of services
Collaborative visualization
High-end visualization on low-end devices
Lessons learned:
High visual impact
Good PR...
Domain experts are hard to please...
Not considered “intuitive”, or even easy to use
The gap is widening…
Urbanflood
Sensor systems, numerical models, computational simulation and scientific visualization for the
development of Early Warning Systems
Context
Context
Total length primary flood defenses in Netherlands: 2875km spread over 90 dike “rings”
Context
Source: “Assessment of primary flood defences in The Netherlands”, Inspectie Verkeer en Waterstaat, National Report on 2006.
“Worst Possible Flood” West coast scenario (source: Rijkswaterstaat)
Maximum waterdepth
1 week
48 h
12 h
4 h
UrbanFlood Online Early Warning System
S S S
Control Centre S S S
Control Centre
Public
UrbanFlood cascade
8 7 6 5 4 3 2 1 0 - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9
- 11 - 10 - 9 - 8 - 7 - 6 - 5 - 4 - 3 - 2 - 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
8 7 6 5 4 3 2 1 0 - 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9
- 11 - 10 - 9 - 8 - 7 - 6 - 5 - 4 - 3 - 2 - 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
.
.
.
Fibre optics Water in/outlets
Fibre
optics
Inverted
pendulum
Seismic (acoustic)
Macro stability dike Channel dike
Cross section of dike with sensors
Basin
Sensors + AI Structural analysis
Breach modeling
Inundation simulation Population Dynamics
Monitoring through sensors
GeoBeadsTM, Alert Solutions Pore pressure
Temperature
Inclination (2 DOF)
Currently installed at 5 locations in NL
St. Petersburg, RU
Boston, UK
…
AnySense
IJkdijk, Bellingwedde, Groningen
Since Oct 2009 4 cross-sections, 230m apart, fibre optic cable
Livedijk Eemshaven
Structural analysis
Sensor monitoring
AI Anomaly Detection
Reliability Analysis
Breach simulator
Virtual Dike
Yes High risk?
Inundation simulator
Population Dynamics
Risk Assessment
Decision Support System
Structural analysis: the Virtual Dike FSI model: water flow through porous media + structural dynamics
Partially saturated soils with water retention
Finite element method
Dike in Groningen, NL tidal water load and flood conditions
sensor input
Velocity vectors
Pressure dynamics
X 500
520
540
560
580
600
620
640
660
680
27.08.2010 22:12 28.08.2010 10:12 28.08.2010 22:12
pre
ss
ure
he
ad
, m
m1E4 virtualE4(K=1E-9m2) virtualE4(K=1E-10m2)
Virtual Dike simulations
Virtual and real sensor dynamics
Breach prediction
time
Dis
ch
arg
e (
m3/s
)
Hydrograph
Inundation simulation
Model: simplified shallow water equations Developed by HR Wallingford, UK
DTM: AHN-1 Rijkswaterstaat (25m2/pixel)
-7m
20m
My office
Inundation simulation Amsterdam Science Park
Inundation simulation
St. Petersburg, Russia Boston, UK
Decision Support System
Integration of:
Sensor data + AI
Simulation results
Maps, weather, ships, roadwork, traffic, location of emergency services, …
Population Dynamics
Life Safety Model (HR Wallingford)
City evacuation
Multi-touch displays
Visualization as a Service
Visualization services • data aggregation • data transformation • feature detection • collaboration coordination
Sensor data, simulation data, live streams
Simulations
Amsterdam
London St. Petersburg
Groningen
Visualization server
Web access High performance visualization on low performance devices
Results
Major achievements:
Cloud-based simulation
Information integration through Common Information Space
Multitouch interfaces Web-based visualization
services
Lessons learned:
Multitouch displays are easy to use, even by domain experts!
The gap is closing again...
Must remember to keep interaction simple
DynaNets
Research Question
Sequence info. Demographic info. - Maximum likelihood analysis - Bayesian analysis
Social, demographical and clinical data analysis
Annotated HIV sequence data
Phylogenetic tree Social tree
From theoretical point of view, is it possible to come up with a mathematical formulation to infer phylogenetic tree from a social tree (network), and how redundant is that?
Building the Network: Reductionism approach
Filtering
Fully-connected Network Filtered Network
Start with an undirected fully-connected network of HIV sequences (each node is connected to all other nodes). Then apply social and demographical filters to drop the socially impossible connections between every two nodes.
Filters
Filter 1: Age difference Age difference for having sexual relationship (for MSMs and
Heterosexuals) or social contact (for IDUs to share needle) Free parameter
Filter 2: Mode of infection (m) & gender (g) For patients 1 and 2:
If(g1 = g2 = Male && m1 = Homo, m2=Hetero) connection = 0
If(g1 = g2 = Male && m1 = m2 = Homo) connection = 1
If(g1=Male, g2 = Female && m1 = m2 = Hetero) connection = 1
….
Filter 3: therapy date (t) & seroconversion date (s) Transmission probability decreases 80-98% after treatment [2][3].
For patients 1 and 2: If(t1 is older than s2) connection = 0
If(t2 is older than s1) connection = 0
HIV Sequence data - Annotated with social and
genetic information
- Unique patients
Fully-connected network - All patients are connected
Filtered-network - Social filters (age difference,
gender and mode of infection,
seroconversion & therapy date)
Directed-filtered network - Direction from a patient with
older sero date to a patient with
more recent sero date (no loops)
Workflow
Filtering
Patient with older seroconversion date
Patient with newer seroconversion date
Seroconversion function
Building Fully-connected network
Twilight Interactive network visualization
Overview of features: Cross platform
Developed on Linux, Windows, known to work on Mac, Microsoft Surface
Focus on interaction: the application invites exploratory analysis Navigation (zoom, pan)
Selection (single node, rubber-band)
Appearance (layout, size/colour/position of nodes, edges, labels)
Supports visualization of large graphs and multiple graphs simultaneously Comparative visualization; let layout of one graph
follow another
Supports several layout algorithms random, circular, tree, Kamada-Kawai, Fruchterman-
Reingold, Reingold-Tilford, graphopt
Transitions between layouts are smooth to provide a visual cue of the chances
Analysis methods Path finding, connectivity distribution, degree
distribution, connected components
Scripting interface to Python
Results
Major achievements:
Interactive graph visualization for large graphs
Support for dynamic (time variant) graphs
Interactive exploration through multitouch interface
Lessons learned:
Multitouch displays are easy to use by domain experts
Thanks to iPad/iPhone
“Move over” effect
Main challenge: mapping functionality to affordance
Conclusions
Designing Decision Support Systems is challenging, but the potential is huge
From distributed HPC, Grids to Clouds: nobody noticed
Domain experts prefer familiar interfaces
VR still has a long way to go
Robert G. Belleman, PhD
Computational Science
Universiteit van Amsterdam
Science Park 904
1098 XH Amsterdam
Email : [email protected]
Web : http://uva.computationalscience.nl/
