OverviewNext generation earth system models that scale to emergent HPChardware systems will need in situ analysis coupled to climatesimulation components. In situ analysis can enable quicker and higherfidelity results than postprocessing.

In this work, we enable in situ analysis for MPAS-Ocean simulationusing Decaf, a data flow system for managing dataflow among tasks ina workflow. We also develop methods to improve load balancing indecoupled analysis tasks.

Coupling MPAS-O with stand alone particle tracer

We move the inbuildMPAS-Ocean particletracer LIGHT to a standalone tracer usingDecaf. This allows forbetter control oversharing of resourcesbetween simulationand analysis (tracer).

Decaf: Decoupled dataflow for in-situ workflows

Dynamic load balancing for unstructured data using constrained graph partitioningState-of-the art methods to loadbalance based on constrained k-dtree do not directly extend tounstructured data. We introduce aconstrained graph partitioning basedload balancing method that naturallyhandles data on both structured andunstructured grids.

Dynamic load balancing for unstructured data using graph distance based embedding The use of graph partitioningto balance workload ischallenged by computationalcost of obtaining balancedpartitions. We propose amethod to address this issueby adapting the constrained k-d tree method for graphdistance based embeddings.

In Situ Flow Analysis for MPAS-Ocean SimulationsMukund Raj (ANL), Hanqi Guo (ANL), Orcun Yildiz(ANL), Phillip Wolfram (LANL), Tom Peterka (ANL)

Dynamic load balancing for parallel particle tracing using workload prediction

Existing methods to dynamically loadbalance parallel particle tracing assumesworkload distribution at upcoming epoch tobe same as the current workload whenpartitioning data for upcoming epoch. Wepropose a method to achieve better loadbalance by partitioning the data based onpredicted workload for upcoming epoch,rather than current workload distribution.

Decaf Dataflow Definition

Transport Layer

Workflow Definition

Swift

Flowcontrol

Nessie Mercury MPI

Python

Datamodel

Datadistribution

Resilience

Decaf Runtime

XML Pegasus

Decaf software organization

MPAS-O

MPAS-O

MPAS-O

MPAS-O

Particle Tracer

Particle Tracer

Particle Tracer

Particle Tracer

Decaf

Coupling MPAS-O with particle tracer using Decaf (left) andpreliminary timing results for MPAS-Ocean built-in particletracer (LIGHT), and data movement using Decaf

Imbalance in particle distribution (left) and advectionworkload (right) for synthetic data in prediction andbaseline cases

Constrained graph partitioning for dynamic load balancingparticle tracing with unstructured data (left) and preliminarytiming results for constrained graph partitioning using BFSexpansion (right)

A workflow is a directed graph of tasks and communicationbetween them. Graph nodes are tasks and graph links arecommunication; both of which can be parallel processes.A dataflow is the communication over links in a workflow.While workflows consists of high level tasks, dataflowsconsists of communication between processes.The Decaf software stack consists of a workflow definitionlayer, a dataflow definition layer, and a transport layer.

Vertex layout in the MPAS-Ocean grid usingintrinsic spatial coordinates (left) and graphdistance based embedding via Pivot MDS (right)