Post on 15-Apr-2022
transcript
Ultrascale Visualization WorkshopNovember 13, 2011
Seattle, WA
Ralf Kähler (KIPAC/SLAC)
Tom Abel (KIPAC/Stanford)Marcelo Alvarez (CITA)Oliver Hahn (Stanford)
Hans-Christian Hege (ZIB)Ji-hoon Kim (KIPAC)
Stuart Marshall (SLAC) Matthew Turk (Columbia) Risa Wechsler (Stanford)John Wise (Georgia Tech)
Visualization Challenges for Large Scale Astrophysical Simulation Data
COMPUTATIONAL ASTROPHYSICS
COMPUTATIONAL PHYSICS DEPARTMENT
Evolution of First Galaxies
Re-ionization EpochLarge-Scale Structure Formation
Evolution of First StarsSimulation: Wise & Abel
Simulation: Wu, Hahn & Wechsler
Simulation: Kim & Abel
Simulation: Alvarez & Abel
Large-Scale Structure FormationSimulation: Wu, Hahn & Wechsler
Evolution of First Stars
Evolution of First GalaxiesSimulation: Kim & Abel
Simulation: Turk & Abel
Reionization EpochSimulation: Alvarez & Abel
Numerical Astrophysics
High dynamical range
Numerical Astrophysics
High dynamical range
Formation of First Stars
- Protogalaxy > 10,000 light years
Simulation: T. Abel
Numerical Astrophysics
High dynamical range
Formation of First Stars
- Protogalaxy > 10,000 light years
- Protostar ~ 10 solar radii
Simulation: T. Abel
Numerical Astrophysics
> 10 spatial orders of magnitude
Simulation: T. Abel
STRUCTURED ADAPTIVE MESH REFINEMENT
Simulation Data: Tom Abel
TEMPORAL REFINEMENT
AMR schemes perform refinement in time
Stability condition for solvers:
- Δt proportional to Δx
- global Δt determined by smallest Δx
➔ Computational overhead on coarse levels
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
TEMPORAL REFINEMENT
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
enzo - Adaptive Mesh Refinement
enzo
- Astrophysical Adaptive Mesh Refinement code
- Cosmological structure formation simulations
http://code.google.com/p/enzo/
DATA SIZES
10-40 levels of refinement
10,000-100,000 time steps
up to 100 million patches
- less than 32^3 cells per patch
up to billions of cells per time step
- different fields ( gas density, gas temperature, etc. )
10s of TByte of data
VISUALIZATION SOFTWARE
C++, JavaScript
Graphical User Interface (Qt)
OpenGL
Available as Open Source soon
Simulation data: Kim & Abel
VISUALIZATION SOFTWARE
Simulation data: Kim & Abel
Modular Design: add vis-routines & readers
Readers for Enzo, Gadget, RAMSES
VISUALIZATION SOFTWARE
Modular Design: add vis-routines & readers
Readers for Enzo, Gadget, RAMSES
Data nodes:Grid-based (AMR)Point-based (unstructured)Image data Simulation data: Kim & Abel
VISUALIZATION SOFTWARE
Visualization methods:
Simulation data: Kim & Abel
VISUALIZATION SOFTWARE
Real-Time Rendering:OpenGL & OpenGL Shading Language
Customize rendering at run-time by modifying OpenGL Shaders
Simulation data: Kim & Abel
VISUALIZATION SOFTWARE
Visualization methods:SlicingDirect Volume RenderingStreamlines...
VISUALIZATION SOFTWARE
Visualization methods:SlicingDirect Volume RenderingStreamlines...
Simulation Data: Mike Norman
VISUALIZATION SOFTWARE
Visualization methods:SlicingDirect Volume RenderingStreamlines...
Simulation Data: John Wise & Tom Abel
VISUALIZATION SOFTWARE
Visualization methods:SlicingDirect Volume RenderingStreamlines...
Simulation Data: Ji-hoon Kim & Tom Abel
VISUALIZATION SOFTWARE
Visualization methods:SlicingDirect Volume RenderingStreamlines...
Simulation Data: Wu, Hahn & Wechsler
Pre-Rendering for American Museum of Natural History Show “The Big Bang”, Narrated by Liam NeesonSimulation data: Ji-hoon Kim (Stanford), Tom Abel (Stanford/SLAC)
DATA ACCESS PATTERNS
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
DATA ACCESS PATTERNS
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
DATA ACCESS PATTERNS
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
DATA ACCESS PATTERNS
0 1 2 3 4 TIME
Level 0
Level 1
Level 2
Generation of Proxy-Grid Structure
Time A Time B
Generation of Proxy-Grid Structure
Time A Time B
Generation of Proxy-Grid Structure
Time A Time B
Generation of Proxy-Grid Structure
Time A Time B
Generation of Proxy-Grid Structure
DATA STORAGE
HDF5 for data I/O
- http://www.hdfgroup.org/HDF5/
Enzo HDF5 output
- each processor writes stream of separate HDF5 groups
- no spatial or temporal relations
GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}
GROUP "grid-1" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}
GROUP "grid-n" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}...
HDF5 INDEX FILE
Post-processing:
- create HDF5 index file
- match the AMR grid structure
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
available time steps
HDF5 INDEX FILE
time groups
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
level groups
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
grids on this level
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
grid meta data
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
previous & next time steps
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
links to datasets
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {
...
}
HDF5 INDEX FILE
HDF5 "./rho.a5" {
GROUP "/" {
GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }
... }
GROUP "time-0" { GROUP "level-0" {
ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }
ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... }
GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}
GROUP "grid-1" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}
GROUP "grid-n" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}...
links to datasets
POSTGRES DATABASE
Alternative:
- Index-structure via databases
Meta data in database, data (fields) in original HDF5 files
QtSQL Module: http://doc.qt.nokia.com/4.7/qtsql.html
PostgreSQL plugin
COMPARISON
File sizes
- Postgres database ~ 20% of HDF5 index file
Metadata access
- Postgres twice as fast as HDF5
GPU-Assisted Ray Casting
[Stegmaier, et al. 2005]
Input data ➔ 3D texture
Render front faces
Fragment Shader for each covered pixel:
Compute ray-direction
Sampling and color mapping
Resulting intensities --> frame buffer
GPU-Assisted Ray Casting for AMR Data
Problem: Overlapping regions
➤ Decomposition of data domain
Adaptive kD-tree
- Nodes represent of non-overlapping blocks of cells
- View-dependent sorting (front-to-back or back-to-front)
GPU-Assisted Ray Casting for AMR Data
View-dependent selection of nodes:
- Resolution level based on distance to viewpoint
~200,000 nodes ~20,000 nodes
Simulation data: John Wise (Georgia Tech), Tom Abel (Stanford/KIPAC)
Pre-Rendering for California Academy of Sciences, Dome Show “Life: A Cosmic Journey”, Narrated by Jodie Foster, November 2010Simulation data: John Wise (Princeton), Tom Abel (Stanford/SLAC)
POINT-BASED DATASETS
Unstructured point datasets
- Dark matter density
- Single stars and star clusters
- ...
Point attributes
- Position
- Mass
- Age
- Accretion rate
- ...
Simulation: Wu, Hahn & Wechsler
Solution for “opaque” point representation
1. Pass
Enable depth-buffer updates
Render points
2. Pass
Perform ray-tracing up to pixels depth value
3. Blending step
GPU-Raycasting of Combined Grid- and Point-based Data
GPU-Raycasting of Combined Grid- and Point-based Data
Usually semi-transparent point representation
gaussian opacity profile for galaxies, stars, DM, etc.
Potential solution:
Resampling of point data to grid structure
Simultaneous rendering of both (grid) data sets
GPU-Raycasting of Combined Grid- and Point-based Data
Requires highly resolved grids
High (texture-)memory consumption
and/or sacrifices data resolution
Low-pass filtering
Rendering artifacts
GPU-Raycasting ofCombined Grid- and Point-based Data
• Point data ➤ Octree structure- Efficient GPU-representation- Implicit bounding-box information
• Recursive refinement of octree nodes
• Stopping criteria- Number of inserted points < threshold
Combining Grid and Point Data
Node texture
3D-RGBA texture
One texel per node
Alpha-channel stores node type
Internal node Leaf node
RGB-channel
Index of first child node Index into data texture
Simulation: Alvarez & Abel
Combining Grid and Point Data
Data texture
3D-RGBA floating point
RGB-channels: center
Alpha-channel: radius
Radius=0 indicates end of list
Simulation: Alvarez & Abel
GPU-Raycasting of Combined Grid- and Point-based Data
1. Grid data ➤ 3D-texture
2. Point set ➤ two 3D-textures (GPU-octree representation)
3. Render front faces of bounding box
4. In Fragment shader
• Compute ray-direction
• Sampling of grid-based data
• Sampling of point-based data
• Combination of partial intensities
• Combination with total intensity
5. Resulting intensity ➤ frame-buffer
Combined Point & Grid Raycasting
Summation of Intensities
• Relation between opacity and extinction coefficient
• Combination of opacities for segment si
• Combination with accumulated intensity/opacity of ray
Combined Point & Grid Raycasting
Performance Optimizations
• Example: Gaussian profiles
Combined Point & Grid Raycasting
Performance Optimizations
• ROI around camera location
Combined Point & Grid Raycasting
Performance Optimizations
• Point and grid data inside ROI rendered with GPU approach
Combined Point & Grid Raycasting
Performance Optimizations
• Outside ROI: - Process grid data until depth of first pass- blending with point splats
Combined Point & Grid Raycasting
Performance Optimizations
Combined Point & Grid Raycasting
Performance Optimizations
• Correct result if only one splat hit by ray outside ROI
Combined Point & Grid Raycasting
Performance Optimizations
• Partially incorrect depth sorting for more than one splat• Artifacts usually not visible in far-field
Combined Point & Grid Raycasting
Comparison of Rendering Quality
ROI = 0% of volume ROI = 25% of volumeROI = 100% of volume
Simulation: Alvarez & Abel
Combined Point & Grid Raycasting
Numerical Simulation: Marcelo Alvarez (CITA), Tom Abel (KIPAC/Stanford)
Combined Point & Grid Raycasting
Numerical Simulation: Marcelo Alvarez (CITA), Tom Abel (KIPAC/Stanford)
Numerical Simulation: Marcelo Alvarez (CITA), Tom Abel (KIPAC/Stanford)
Numerical Simulation: Marcelo Alvarez (CITA), Tom Abel (KIPAC/Stanford)
Visit us as the SLAC booth (303) and watch some visualizations in 3D !