SOFTWARE PRODUCTIVITY CHALLENGES IN ENVIRONMENTAL APPLICATIONS
SIAM CSE 15, Salt Lake City, UT.
David Moulton, Ethan Coon Los Alamos Na:onal Laboratory Carl Steefel, Sergi Molins Lawrence Berkeley Na:onal Laboratory Sco8 Painter Oak Ridge Na:onal Laboratory March 15, 2015.
LA-‐UR-‐15-‐21857
Collaborators (ASCEM, LDRD, IDEAS, …)
2
Ben Andre (NCAR) Erin Barker (PNNL) Markus Berndt (LANL) David Bernholdt (ORNL) Ethan Coon (LANL) Marc Day (LBNL) Vicky Freedman (PNNL) Carl Gable (LANL) Rao Garimella (LANL) Dylan Harp (LANL) Glenn Hammond (SNL) Mike Heroux (SNL)
Hans Johansen (LBNL) Jeff Johnson (LBNL) Konstan:n Lipnikov (LANL) Lois Curfman McInnis (ANL) Terry Miller (LANL) Sergi Molins (LBNL) Sco[ Painter (ORNL) Tim Scheibe (PNNL) Carl Steefel (LBNL) Daniil Svyatskiy (LANL) Haruko Wainwright (LBNL) Cathy Wilson (LANL) …
Outline
¨ Background/MoKvaKon ¤ Science Ques:ons ¤ Science and So_ware Challenges
¨ Environmental ApplicaKons ¤ Use a wide range model complexity ¤ Process-‐rich models go beyond tradi:onal mul:physics
¨ ProducKvity Challenges ¤ Community-‐based interdisciplinary opportuni:es (IDEAS) ¤ Driving collabora:on and development through Use Cases
¨ Conclusions
Environmental ApplicaKons
4
¨ Climate impacts and feedbacks (carbon and nitrogen cycling)
¨ Contaminant transport and reac:ons. ¨ Complex interac:on between
subsurface and land surface processes.
Science and SoQware Challenges
¨ Terrestrial systems span science challenges: ¤ Inherently multsicale with coupling that must be
explored through simula:on ¤ Development and leveraging of mechanis:c
models at all scales is required ¤ Mul:scale data must integrate seamlessly with
itera:ve development of predic:ve models
¨ Established codes have significant capabili:es but are not ready/suitable for: ¤ refactoring for emerging architectures ¤ interdisciplinary development teams ¤ are not sufficiently flexible or interoperable
So_ware produc:vity is a cri:cal factor in realizing exascale simula:ons of terrestrial systems.
5
Amanzi: High-‐Level ObjecKves
6
Wide Range of Complexity
Wide Range of Plaiorms
¨ Flexible and Extensible: Modular simula:on capability for waste form degrada:on, variably saturated flow, and reac:ve transport.
¨ Performance Portable: ¨ Efficient, robust
simula:ons from laptops to supercomputers.
¨ Design and build for refactoring required by emerging mul:-‐core and accelerator-‐based systems.
¨ Community Par@cipa@on: Open-‐source project with strong mee:ng the needs of a strong interdisciplinary community.
Leverage advances from across DOE (e.g., ASCR, ASC) and Academia.
Amanzi: Simulator CapabiliKes
¨ Process Kernels ¤ Transient unsaturated flow with Richards
equa:on, including op:ons to steady-‐state ini:aliza:on.
¤ Transient single-‐phase flow with specific storage/yield
¤ Volume based sinks/sources ¤ Reac:ve-‐transport, with operator splikng
for reac:ons. ¤ Support for a wide range of chemical
reac:ons.
¨ Framework and Infrastructure: ¤ Unstructured meshes with polyhedral cells,
block-‐structured AMR, and internal genera:on of hexahedral meshes in rectangular domains.
¤ Designed to integrate with Akuna/Agni model setup and toolsets.
¤ Flexible and extensible MPC/PK APIs. ¤ Parallel I/O: visualiza:on & restart
Amanzi/ATS Design
MPC
(Base)
PK:: Flow
RichardsPK:: Transport
Advective
PK:: Reactions
Geochemistry
HPC Toolsets
Data management Mesh Infrastructure Discretizations Solvers
HPC Core Framework (services) and Third Party Libraries
8
ASCE
M-HPC
-F-Are
a-Sava
nnah R
iver S
ite
ASCEM-HPC. F-Area. Geochemical System. ASCEM-F-Area. Meshes for Amanzi.
! 4 meshes at different horizontal resolution: ! 5 m. ! 2.5 m. ! 1.25 m.
! 0.625 m.
! ExodusII format. ! Available in the
HPC wiki.
Alquimia: A geochemistry interface library
¨ Alquimia currently assumes reac:ve transport uses operator-‐splikng.
¨ Fully-‐implicit reac:ve transport support is planned.
¨ Assists in enforcing geochemical condi:ons (specia:on) for transport boundary condi:ons.
¨ Alquimia can facilitate benchmarking of geochemical capabili:es in exis:ng codes.
¨ Geochemistry libraries, such as PFLOTRAN and CrunchFlow, are implemen:ng interfaces to Alquimia.
Several geochemistry libraries are established in the community making geochemistry ideal to explore componen:za:on and interface design. Alquimia is an interface, and does not perform any reacKon calculaKons.
Alquimia is open source, h[ps://bitbucket.org/berkeleylab/alquimia
SRS F-‐ Area: Phase II Background and Modeling
Ø F-‐Area at the Savannah River Site (SRS): § Variably saturated flow, with uncertain vadose zone source term § Unstructured grids capturing hydrostra:graphy/topography. § Mineral precipita:on and dissolu:on plus sorp:on in groundwater. § UQ studies in 2D, and representa:ve 3D simula:on of F-‐Area.
ASC
EM-H
PC-F
-Are
a-Sa
vann
ah R
iver
Site
ASCEM-HPC. F-Area. Geochemical System. ASCEM-F-Area. Domain.
ASCE
M-HPC
-F-Are
a-Sava
nnah R
iver S
ite
ASCEM-HPC. F-Area. Geochemical System. ASCEM-F-Area. Meshes for Amanzi.
! 4 meshes at different horizontal resolution: ! 5 m. ! 2.5 m. ! 1.25 m.
! 0.625 m.
! ExodusII format. ! Available in the
HPC wiki.
SRS F-‐Area: SimulaKon of Uranium ReacKve Transport
Simula'on shows the retarda'on of the Uranium plume due to sorp'on, rela've to pH (non-‐reac've tracer), highligh'ng the importance of biogeochemistry.
Thaw-‐induced topographic reorganizaKon
Progressive Degrada:on Jorgenson
Evolu'on of polygonal paAerned ground is expected to control hydrology and thus the carbon cycle
Hand-‐coded model couplings become unmanageable with more than a few ecohydrochemical processes
Managing complexity in process-‐rich simulaKons
The Arc:c Terrestrial Simulator (ATS) leverages Amanzi’s infrastructure, adds an innova:ve process-‐kernel management system, new process kernels, and new meshing workflow.
A New MulK-‐Physics Framework
¨ Use object oriented concepts for flexibility and extensibility, ¤ Dynamic data management allows Process Kernels (PKs) to register their
data needs at run'me ¤ This registra'on provides the necessary informa'on to represent the
rela'onships between variables (i.e., dependent vs. independent) and to enable a dynamically configured model and simula:on.
¨ Enables integrated hierarchical approach to tes:ng and QA ¤ build confidence in models and implementa'ons ¤ study structural model uncertainty, and understand process coupling ¤ support hypothesis tes'ng (“What if?” scenarios)
¨ Significantly enhances interdisciplinary collabora:ons by allowing scien'sts to engage effec'vely in their area of exper'se, and immediately seeing the impact of their contribu:ons.
Benefits of the new MulK-‐Physics Framework Developing a Multi-Physics Model
MPC
SubsurfaceWater
PK::Flow
Permafrost
PK::Energy
?????
Flow:@
@t
[�(⇢l
s + ⇢g
(1 � s)] = �r · (⇢V)
V = �k(s)K(x)
µrp
Energy: ??
Constitutive Relations: ??
4
• Flow:
• Energy: ?? • Cons@tu@ve Rela@ons: ??
Developing a Multi-Physics Model
MPC
SubsurfaceWater
PK::Flow
Permafrost
PK::Energy
?????
Flow:@
@t
[�(⇢l
s + ⇢g
(1 � s)] = �r · (⇢V)
V = �k(s)K(x)
µrp
Energy: ??
Constitutive Relations: ??
4
Automated Dynamic ConstrucKon of the Dependency Graph (DAGs)
Consider three alterna:ve models for the thermal energy
Dynamic Construction of Dependency Graph
Consider three alternative models for the thermal energy
p
s
µ ⇢
k
r
V
res
flow
energy not useds(p), µ, ⇢(p)
p
T
s
µ ⇢
k
r
V
res
flow
prescribed temperatures(p), µ(T), ⇢(p, T)
p
T
s
µ ⇢u
h
k
r
V
res
flow
res
energy
conservation of energys(p), µ(T), ⇢(p, T)u(p, T), h(p, T)
5
Dynamic Construction of Dependency Graph
Consider three alternative models for the thermal energy
p
s
µ ⇢
k
r
V
res
flow
energy not useds(p), µ, ⇢(p)
p
T
s
µ ⇢
k
r
V
res
flow
prescribed temperatures(p), µ(T), ⇢(p, T)
p
T
s
µ ⇢u
h
k
r
V
res
flow
res
energy
conservation of energys(p), µ(T), ⇢(p, T)u(p, T), h(p, T)
5
Dynamic Construction of Dependency Graph
Consider three alternative models for the thermal energy
p
s
µ ⇢
k
r
V
res
flow
energy not useds(p), µ, ⇢(p)
p
T
s
µ ⇢
k
r
V
res
flow
prescribed temperatures(p), µ(T), ⇢(p, T)
p
T
s
µ ⇢u
h
k
r
V
res
flow
res
energy
conservation of energys(p), µ(T), ⇢(p, T)u(p, T), h(p, T)
5
Energy not used s(p), μ, ρ(p)
Prescribed temperature s(p), μ(T), ρ(p,T)
Conserva:on of Energy s(p), μ(T), ρ(p,T), u(p,T), h(p,T)
Notz, Pawlowski, Sutherland, ACM TOMS (2012); Coon, Moulton, Painter submi[ed to Environmental Modelling & So_ware (2014).
ATS Process Modeling CapabiliKes
16
¨ Surface flow: ¤ Diffusive wave approxima:on ¤ Non-‐isothermal
¨ Subsurface flow: ¤ 3-‐phase, Richards-‐like equa:on(nonlinear parabolic)
¤ Non-‐isothermal
Coupling Surface/Subsurface Flow
17
Visualiza'on enhances understanding and analysis of strongly coupled surface and subsurface flow models. Here flow to and from the lower troughs depends on external forcing and current satura'on of the subsurface.
ProducKvity Crisis (and Opportunity)
¨ Further strengthen the role modeling and simula:on play in science.
¨ Enable researchers to study more complex mechanis:c models of single and coupled processes.
¨ Add more processes to exis:ng systems models.
¨ Enable simula:ons that bridge the vast range of temporal and spa:al scales found in natural systems.
Con:nuous growth in computa:onal power should …
Companion BER workshop report: h8p://www.doesbr.org/VirtualEcosystems/
a
DRAFT
Interoperable Design of Extreme-‐scale ApplicaKon SoQware (IDEAS)
19
Motivation Enable increased scien@fic produc@vity, realizing the poten:al of extreme-‐ scale compu:ng, through a new interdisciplinary and agile approach to the scien@fic soNware ecosystem.
Objectives Address confluence of trends in hardware and
increasing demands for predic:ve mul:scale, mul:physics simula:ons.
Respond to trend of con:nuous refactoring with efficient agile so_ware engineering methodologies and improved so_ware design.
Approach ASCR/BER partnership ensures delivery of both crosscukng methodologies and metrics with impact on real applica:on and programs.
Interdisciplinary mulK-‐lab team (ANL, LANL, LBNL, LLNL, ORNL, PNNL, SNL) ASCR Co-‐Leads: Mike Heroux (SNL) and Lois Curfman McInnes (ANL) BER Lead: David Moulton (LANL) Topic Leads: David Bernholdt (ORNL) and Hans Johansen (LBNL)
Integra@on and synergis@c advances in three communi@es deliver scien:fic produc:vity; outreach establishes a new holis:c perspec:ve for the broader scien:fic community.
Impact on Applications & Programs Terrestrial ecosystem use cases @e IDEAS to modeling and simula@on goals in two Science Focus Area (SFA) programs and both Next Genera:on Ecosystem Experiment (NGEE) programs in DOE Biologic and Environmental Research (BER).
SoQware ProducKvity for Extreme-‐Scale
Science Methodologies for SoQware ProducKvity
Use Cases: Terrestrial Modeling
Extreme-‐Scale ScienKfic SoQware Development Kit
(xSDK)
Use Cases: SelecKon Criteria & Goals
20
¨ Use exisKng modeling capabiliKes and align with development plans in currently funded projects: ¤ Science Focus Area Projects (SFAs) at LBNL and PNNL, ¤ Next Genera:on Ecosystem Experiment (NGEE) projects in Arc:c and
Tropics (mul:-‐ins:tu:on led by ORNL and LBNL respec:vely). ¨ Leverage soQware producKvity tools and methodologies to enable
efficient refactoring, or new development, of components to address important scien:fic ques:ons,
¨ Demonstrate higher fidelity soluKons to scien:fically interes:ng problems by taking advantage of advanced architectures,
¨ Use/couple mechanis:c models across a range of scales, ¨ Use exisKng observaKons to demonstrate higher produc:vity workflows
for model/data integra:on, ¨ Support a longer-‐term vision of a "virtual ecosystem" with mechanis:c
representa:ons of vegeta:on processes embedded in large-‐scale watershed models.
Use Case 1: Hydrological and Biogeochemical Cycling in the Colorado River System.
Photos courtesy of R. Kaltschmidt h[p://photos.lbl.gov/albums.php?albumId=428202 (LBNL)
Hyporheic zone biogeochemistry involves a collec:on of coupled mul:scale processes that play a cri:cal role in carbon cycling and watershed performance.
Use Case 1: Highlights
22
Lower East River Catchment ¨ High resolu:on models of individual stream
meander flow and biogeochemistry using Amanzi/ATS and ParFlow
¨ Upscale flow and biogeochemistry within lower East River and in hyporheic zone while maintaining high spa:al resolu:on locally (< 0.1 m).
Science Outcomes: Use Case 1
Science Objec@ve: A be[er understanding of aquifer redox status and climate impacts on watershed carbon and nitrogen cycling through higher fidelity, mul:-‐scale models simulated at high spa:al resolu:on. 23 ¨ Year 1: Perform a high-‐resolu:on subsurface/
surface flow simula:on for a single meander of the East River with new EcoTrait biogeochemical reac:on network, with horizontal spa:al resolu:on of ~0.1m.
¨ Year 2: Extend simula:ons to a small meandering sec:on of the East River with lateral inputs, and upscale this flow and the EcoTrait subsurface reac:ve transport model over this domain.
¨ Year 3: Incorporate the effects of vegeta:on on hydrology and biogeochemical fluxes within integrated surface and subsurface water flow system within the larger East River watershed.
Use Case 2: Thermal hydrology and carbon cycling in tundra at the Barrow Environmental Observatory.
5-‐polygon cluster
Lobster catchment
Models of warming tundra are very process-‐rich and mul:scale (microtopography to large basins), including thermal-‐hydrology, biogeochemistry, and deforma:on.
Use Case 2: Highlights
¨ Mul:scale approach combining 1-‐D thermal hydrology models with 2-‐D overland flow using Amanzi-‐ATS
0 5 10 154.04.55.05.56.0
¨ Large-‐scale simula:ons using Amanzi/ATS for thermal hydrology, PFLOTRAN for biogeochemistry, and CLM for land surface processes
Soil organic ma[er decomposi:on rates in a undegraded polygon
Science Outcomes: Use Case 2
Science Objec@ve: Determine how thawing permafrost caused by warming Arc:c temperatures alters the hydrologic and carbon cycles of Arc:c lowland tundra. 26 ¨ Year 1: Demonstrate that a mul:scale model comprising
independent 1D ver:cal TH models coupled to a 2D rou:ng of surface water is feasible in the new Amanzi/ATS framework.
¨ Year 2: Verify that this intermediate-‐scale representa:on, with ~10m resolu:on, is an adequate upscaling of the fine-‐scale representa:on, with ~0.1m resolu:on, including subsurface biogeochemistry and land surface processes (through PFLOTRAN, and CLM respec:vely).
¨ Year 3: Perform simula:ons on domains of 1-‐10km using the intermediate-‐scale model, and use those simula:ons to evaluate exis:ng parameteriza:ons in global land surface models.
Conclusions
27
¨ Environmental applica:ons offer significant challenges and exci@ng opportuni@es for interdisciplinary collabora@on
¨ Significant progess has been made in recent years: ¤ Modular open-‐source applica:on codes provide a bridge between
disciplines ¤ Flexible and extensible frameworks can manage the complexity of
process-‐rich models enabling run:me experimenta:on ¨ Significant challenges remain:
¤ Significant exper:se resides in monolithic established codes ¤ Current tes:ng prac:ces do not support refactoring required for new
architectures with more abstract programming models ¤ Inherently mul:scale: understanding and predic:ons gained at finer
scales are needed to impact watershed, and even Earth System scales. ¨ A new interdisciplinary and agile approach to the scien@fic soNware
ecosystem is cri@cal and will be explored in the IDEAS project, driven by use cases from terrestrial ecosystems.