26
HPC in Environmental Modeling: Status and Challenges Bert Jagers 4th PRACE International Seminar “HPC driving innovation in Europe” Bologna, 16-17 april 2012
HPC in Environmental Modeling:Status and Challenges
Bert Jagers
4th PRACE International Seminar“HPC driving innovation in Europe”Bologna, 16-17 april 2012
Overview
• Environmental modeling: a brief overview• Current HPC status of various modules• Ongoing developments & challenges
Environmental modeling:a brief overview
Process based environmental modeling
CONTROL
WAQ/ECOWAVES MORPHOFLOW
RAINFALLRUNOFF
GROUNDWATER
For every core discipline a separate module has been developed. Each module has its own characteristics with respect to resolution, data demand and uncertainty, solution space, numerical algorithms, etc.
This presentation focuses mainly on the detailed hydrodynamics solver represented by FLOW.
FLOW is not just one module …
Delft3D-FLOW (2D/3D)
WAQUA/SIMONA (2D/3D)
DUFLOW (1D)
SOBEK-RE (1D)
SOBEK-Rural/Urban (1D/2D)
Flexible Mesh (../2D/..) 1D/2D/3DD-FLOW FM
Clear desire to converge on one hydrodynamic module. Advantage: reduced development costs in long term.Challenge: satify user needs w.r.t. speed, functionality, stability.
since 80’s
since 80’s
since 90’s
since 80’s
since 80’s
since recently
Typical WAQUA/SIMONA applications
High resolution hydrodynamic models of the Dutch rivers, estuaries, lakes & North Sea.
These detailed river and estuary models are only used for design and impact studies.
Derived 1D SOBEK models of rivers and estuaries are used for real-time operational systems of the storm surge barriers.
20-40 m resolution
Typical Delft3D applications include …
Somewhat lower resolution (hydrodynamic and) sediment transport and morphology models of the Dutch rivers, estuaries, and coastal regions.
For instance to simulate the effect of dredging strategies to maintain navigability of the main rivers.
Compared to WAQUA different choices were made for Delft3D when choosing between features and speed.
… but also …
Delft3D has been developed for a wide range of engineering applications as illustrated by some applications shown here.
Although the typical domain size of Delft3D models is 10-100 km with resolutions going down to 10-100 m, Delft3D models have been set up for laboratory scale (centimetres, seconds) to global models and simulation periods of 1000s of years.
Coastal & estuarinemorphology
Coastal &riverine flooding
Biogeomorphological interaction& self-organization
1D-3D modelintegrated flow and water quality model
Storm surge & waveforecasting
… as well as …
Long-term (ideally 10k years and more) development of river and delta development a.o. to increase the understanding of geological stratigraphy improving oil/gas exploration.
Geleynse et al. (2011)
… and also water quality & ecology
It’s an integrated modeling environment in itself since it contains approximately 500 dynamic quantities which can individually be switched on and off with over 700 associated processes, and up to 5000 process input parameters …
… and you can add
your own at run time.
Next step: increase resolution and flexibility
We are currently developing new high-resolution flooding & flexible integrated 1D/2D/3D solvers.
<1m resolution
Current HPC status of various modules
Status of parallelization
WAQUA/SIMONA (2D/3D)
DUFLOW (1D)
SOBEK-RE (1D)
SOBEK-Rural/Urban (1D/2D)
No
No
YES, worked on it since late 90’s
No
Component Parallelized?
Preprocessor: automatic or user assisted partitioning of input filesSimulation: ADI row/column sweeps & comm. along partition boundariesPostprocessor: combine multiple result-files
Initial: PVM, now: MPI (MPICH2 / OpenMPI)Initially only 1:1 coupling, now also 1:n grid refinement at boundaries
DUFLOW (1D)
SOBEK-RE (1D)
SOBEK-Rural/Urban (1D/2D)
No
No
No
Except for Monte Carlo ensembles for uncertainty and parameter estimation.
Except for Monte Carlo ensembles for uncertainty and parameter estimation.
Scaling performance WAQUA/SIMONA
• Good scalability up to 100 cores depending on model and hardware• Pre/post partitioning step is expensive for short simulations (mostly file IO)• Flexible partitioning and core assignment methods
Partitioning can be tuned manually
Tools have been developed to assist the user in optimizing partitioning beyond standard strip-wise or orthogonal recursive bisection methods.
Load balancing tool
Status of parallelization
Delft3D-FLOW (2D/3D)
WAQUA/SIMONA (2D/3D)
DUFLOW (1D)
SOBEK-RE (1D)
SOBEK-Rural/Urban (1D/2D)
No
No
YES!
Partly, we started work on this in 2006.
No
Component Parallelized?
Parallelization Delft3D-FLOW
• Only strip-wise partitioning (MPI)• Reads one input file, and writes one output file (no pre/post step)• Supported: flow, turbulence, transport, morphology
• no multi grid / refinement• only sigma layer in vertical• no coupling with wave module• no water quality/ecology coupling• no structures or floating particles• etc.
Parallel performance of Delft3D-FLOW
• Happy we got this far, but clear room for improvement compared to WAQUA/SIMONA.
Status of parallelization
Delft3D-FLOW (2D/3D)
WAQUA/SIMONA (2D/3D)
DUFLOW (1D)
SOBEK-RE (1D)
SOBEK-Rural/Urban (1D/2D)
Water Quality & Ecology (1D/2D/3D)
No
No
YES!
Partly
The alternative way…
No
Component Parallelized?
MPI partitioning of the explicit advection/diffusion solvers.OpenMP statements to schedule independent processes and transports of individual substances on different cores.
Status of parallelization
Delft3D-FLOW (2D/3D)
WAQUA/SIMONA (2D/3D)
DUFLOW (1D)
SOBEK-RE (1D)
SOBEK-Rural/Urban (1D/2D)
Water Quality & Ecology (1D/2D/3D)
No
No
YES!
Partly
The alternative way
D-FLOW FM (1D/2D/3D) Just starting OpenMP & MPI …
No
Component Parallelized?
SWAN waves OpenMP & MPI (scaling to 3k nodes)
MODFLOW & MT3D ground water MPI
Ongoing developments & challenges
Building a world-wide community
http://oss.delft3d.nl
Over 1600 new users registered in just over one year
Distr. Research Infrastr. for Hydro-Meteorology
• EU FP7 project of CIMA Foundation, LMU, DLR, CNR-IMATI, CNRS-Meteo France, CERFACS, Deltares, HR Wallingford, hydrometeo service Serbia, technical university Madrid, CUAHSI.
• Enable the uptake of dedicated Grid and HPC services to facilitate a multi-disciplinary and global collaboration between meteorologists, hydrologists and other Earth Science experts.
• Vision: run HPC ensembles of coupled meteorology, hydrology, flooding models for forecasting purposes
• No longer use 1D hydrodynamic models,but 2D (or even 3D) models
http://www.drihm.eu/
See also NSF-funded Community SurfaceDynamics Modeling Systemhttp://csdms.colorado.edu
OBSERV. DATA
Integrated Modeling
CONTROL
WAQ/ECO
WAVES
MORPHO
FLOW
RAINFALLRUNOFF
WINDTRANSPORT
GROUNDWATER
The main challenge for the future will not so much be running the individual modules efficiently in a parallel environment, but to run the modules efficiently together for heterogeneous real-life applications.
A balance must be found between optimized parallelization and flexible, plug & play connectivity.
