Supercomputing • Communications • Data

NCAR Scientific Computing Division

Experiences with multiresolution methods for Experiences with multiresolution methods for qualitative and quantitative analysis of qualitative and quantitative analysis of

terascale turbulence data setsterascale turbulence data sets

John ClyneJohn Clyne11, Alan Norton, Alan Norton11 and Mark Rast and Mark Rast22

National Center for Atmospheric ResearchNational Center for Atmospheric Research

SCD Users ForumSCD Users Forum

1 Scientific Computing Division

2 High Altitude Observatory

This work is funded in part through a U.S. National Science Foundation, This work is funded in part through a U.S. National Science Foundation, Information Technology Research program grantInformation Technology Research program grant

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

Goal: Facilitate analysis of terascale size earth sciences data Goal: Facilitate analysis of terascale size earth sciences data sets sets

Our ability to numerically model earth sciences phenomena Our ability to numerically model earth sciences phenomena has surpassed our ability to analyze and gain insight from the has surpassed our ability to analyze and gain insight from the resultant simulation outputsresultant simulation outputs

– Historical focus of computing centers on batch processingHistorical focus of computing centers on batch processing See tomorrow’s 10:40am talk: See tomorrow’s 10:40am talk: Towards a Comprehensive Environment Towards a Comprehensive Environment

for Data Analysis and Visualizationfor Data Analysis and Visualization

– Dichotomy of batch and interactive processing needsDichotomy of batch and interactive processing needs Analysis is an inherently interactive taskAnalysis is an inherently interactive task Numerical simulation is well suited to batch processingNumerical simulation is well suited to batch processing

– Lack of scalable toolsLack of scalable tools 32bit32bit single threadedsingle threaded in-core algorithmsin-core algorithms

– Moore’s law doesn’t apply to all technology curvesMoore’s law doesn’t apply to all technology curves E.g. File access speedsE.g. File access speeds

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

Visualization and Analysis Platform for ocean, Visualization and Analysis Platform for ocean, atmosphere, and sun Researchers (VAPoR)atmosphere, and sun Researchers (VAPoR)

Key componentsKey components1.1. Domain specific application focus: numerically simulated Domain specific application focus: numerically simulated

turbulence turbulence

2.2. Integrate visualization into analysis process, and employ as a Integrate visualization into analysis process, and employ as a first order data reduction techniquefirst order data reduction technique

3.3. Employ multiresolution data representation as a second order Employ multiresolution data representation as a second order data reduction techniquedata reduction technique

AssumptionsAssumptions– High resolution 3D gridsHigh resolution 3D grids– Desktop PC with commodity, hardware accelerated graphics Desktop PC with commodity, hardware accelerated graphics – Familiarity with existing analysis tools (e.g. IDL, Matlab)Familiarity with existing analysis tools (e.g. IDL, Matlab)– For *some* analysis operations, results from coarsened data can For *some* analysis operations, results from coarsened data can

yield same interpretation as results from native datayield same interpretation as results from native data

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

Enabling speed/quality tradeoffs with multiresolution data Enabling speed/quality tradeoffs with multiresolution data representationrepresentation

1

•Multiple copies of data at varying power of two resolutions

•Storage costs:

1/2

1/41/8

dddL

l

dl //// 32

0

212121121

•2D Example: Texture MIP MappingExample: Texture MIP Mapping

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

Wavelets Transforms for 3D Multiresolution data Wavelets Transforms for 3D Multiresolution data representationrepresentation

Permit hierarchical data Permit hierarchical data representationrepresentation

Invertible and lossless Invertible and lossless (subject to floating point (subject to floating point round off errors)round off errors)

Numerically efficient – Numerically efficient – forward and inverse forward and inverse transformtransform

No additional storage No additional storage costs!!!costs!!!

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

VAPoR Project StatusVAPoR Project Status

Alpha release of software currently under evaluation Alpha release of software currently under evaluation by steering committeeby steering committee– Scalar dataScalar data– Cartesian gridsCartesian grids

Expect first production release later this yearExpect first production release later this year– Vector dataVector data– More general computation grids (e.g. AMR, spherical More general computation grids (e.g. AMR, spherical

grids)grids)

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

PeoplePeople

Steering CommitteeSteering Committee– Nic Brummell - CU, JILANic Brummell - CU, JILA– Aimé Fournier – NCAR, IMAGeAimé Fournier – NCAR, IMAGe– Helene Politano - Helene Politano -

Observatoire de la Cote Observatoire de la Cote d'Azurd'Azur

– Pablo Mininni, NCAR, IMAGePablo Mininni, NCAR, IMAGe– Yannick Ponty - Observatoire Yannick Ponty - Observatoire

de la Cote d'Azurde la Cote d'Azur– Annick Pouquet - NCAR, ESSLAnnick Pouquet - NCAR, ESSL– Mark Rast - NCAR, HAOMark Rast - NCAR, HAO– Duane Rosenberg - NCAR, Duane Rosenberg - NCAR,

IMAGeIMAGe– Matthias Rempel - NCAR, Matthias Rempel - NCAR,

HAOHAO– Yuhong Fan - NCAR, HAOYuhong Fan - NCAR, HAO

DevelopersDevelopers– Alan Norton – NCAR, SCDAlan Norton – NCAR, SCD– John Clyne – NCAR, SCDJohn Clyne – NCAR, SCD

Research CollaboratorsResearch Collaborators– Kwan-Liu Ma, U.C. DavisKwan-Liu Ma, U.C. Davis– Hiroshi Akiba, U.C. DavisHiroshi Akiba, U.C. Davis– Han-Wei Shen, Ohio StateHan-Wei Shen, Ohio State

Systems SupportSystems Support– Joey Mendoza, NCAR, SCDJoey Mendoza, NCAR, SCD

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

Preliminary experiencesPreliminary experiences

Supercomputing • Communications • Data

NCAR Scientific Computing Division John Clyne & Mark Rast{clyne,mprast}@ncar.ucar.edu

Example: Compressible plume dynamicsExample: Compressible plume dynamics 504x504x2048504x504x2048 5 variables 5 variables

(u,v,w,rho,temp)(u,v,w,rho,temp) ~500 time steps saved~500 time steps saved 9TBs storage 9TBs storage

(4GBs/variable/timestep)(4GBs/variable/timestep) Six months compute time Six months compute time

required on 112 IBM SP required on 112 IBM SP RS/6000 processors RS/6000 processors

Three months for post-Three months for post-processingprocessing

Data may be analyzed for Data may be analyzed for several yearsseveral years

M. Rast, 2004. Image courtesy of Joseph Mendoza, NCAR/SCD

Integrated visualization and analysis on interactively Integrated visualization and analysis on interactively selected subdomains:selected subdomains:

u

2ur

pg

z

1 pr

1 pr

2ur

z

Vertical vorticity of the flow

Mach number of the vertical velocityFull domain seen from above Subdomain from side

Full domain seen from above Subdomain from side

Efficient analysis requires rapid calculation and visualization of unanticipated derived quantities. This can be facilitated by a combination of subdomain selection and resolution reduction.

A test of multiresolution analysis: Force A test of multiresolution analysis: Force balance in supersonic downflowsbalance in supersonic downflows

Sites of supersonic downflow are also those of very high vertical vorticity. The core of the vortex tubes are evacuated, with centripetal acceleration balancing that due to the inward directed pressure gradient. Buoyancy forces are maximum on the tube periphery due to mass flux convergence.

The same interpretation results from analysis at half resolution.

1 pr

u

2ur

pg

z

1 pr

2ur

z

u

2ur

pg

z

1 pr

1 pr

2ur

z

Full

Half

Resolution

Subdomain selection and reduced resolution together yield data reduction by a factor of 128