WWOSC 2014, Montreal, August 16-21 2014 A review of technologies and algorithms used to couple Earth...

WWOSC 2014, Montreal, August 16-21 2014

A review of technologies and algorithms

used to couple Earth System model components

S. Valcke - CERFACS


Outline

Introduction

1.Two main technical approaches to coupling in climate

2.Current coupling technologies used in climate modelling• ESMF• CESM/cpl7• FMS• OASIS

3. Few ocean-atmosphere (+ ice, land, wave) coupling algorithms:

European ESMs, Env Canada, CESM

Conclusions


Why couple ocean and atmosphere (and sea-ice and land and …) models? Of course, to treat the Earth System globally

What does “coupling of codes” imply? Exchange and transform information at the code interface Manage the execution and synchronization of the codes

What are the constraints? Physical constraints: e.g. energy conservation at the interfaces Numerical stability of the algorithm Start from existing and independently developed codes Global performance and load balancing issues are crucial Computing platform and OS characteristics

Introduction


Outline

Introduction

1.Two main technical approaches to coupling

2.Current coupling technologies used in climate modelling• ESMF• CESM/cpl7• FMS• OASIS



Conclusions


1. Coupling framework integrated approach

Split code into elemental units at least init/run/finalize

Write or use coupling units

Adapt data structure and calling interface Use the framework to build a

new hierarchical merged code

prog1_u1 prog2_u1

coupling

prog1_u2 prog1_u3

couplingprog2_u2

program prog1…end prog1

prog1_u1

prog1_u2

prog1_u3

program prog2…end prog2

prog2_u1

prog2_u2

efficient sequential and concurrent components

existing codes

(easy)

1. Two main approaches to coupling

Ex: with built-in driving layer: CESM, FMS without pre-defined driving layer: ESMF


2. Coupler or coupling library approach

program prog2…call cpl_recv (data2, …)end

program prog1…call cpl_send (data1, …)end

cou

ple

r

couplingconfiguration

cou

ple

r

existing codes

concurrent coupling (parallelism)

possible waste of resources if sequential execution of the components is enforced by coupling algorithm

multi-executable: may be harder to manage for the OS

efficient

Ex: OASIS, TDT, Open-Palm, GOSSIP

1. Two main approaches to coupling


Outline

Introduction


2.Current coupling technologies used in climate modelling

• ESMF• Cpl7/CESM• FMS• OASIS



Conclusions


Earth System Modeling Framework

2. Current coupling technologies - ESMF

• NUOPC layer: US NWP centers conventions and templates for better interoperability

Open source software for building climate and weather applications based on components developed in different modeling centers

• Multi-agency governance (NSF, NASA, DoD, NOAA) with many partners

• 12 different modelling systems, about 85 different components

component = well-defined interface + coherent function

• Gridded Components: scientific code • Coupler Components: data transformation

user builds a new single code as hierarchy of nested components support for multiple executables and web services

• Component-based design:

Goddard Space Flight Center GEOS-5 model


1. Define Gridded Components : slip code into init, run and finalize methods

2. Wrap native data structures into ESMF data structure

3. Write Coupler Components

4. Schedule components and exchange data

5. Execute the application

subroutine myOceanRun (.. , impState, expState, clock, …) type(ESMF_State) :: impState

subroutine oceanToAtmCpl (.. ,) call ESMF_FieldRedist(oceanField, atmField, …)

…call ESMF_GridCompRun(oceanComp, …)call ESMF_CplCompRun (oceanToAtmCpl, …)call ESMF_GridCompRun(atmComp, …)

ESMF “Superstructure”: coupling tools and component wrappers with standard interfaces

2. Current coupling technologies - ESMF

ESMF “Infrastructure” :• calendar management; message logging, regridding weight generation (can be used offline) data regridding & communication

Hill et al., Comput. Sci. Eng., 2004


Cpl7 for CCSM4 and CESM1

2. Current coupling technologies – Cpl7

Software with top-level driver and coupler component for flexible assembling of atmosphere, ocean, land and sea ice into one executable

Developed by the NCAR Earth System Laboratory, uses Argonne Nat Lab MCT for data regridding and exchange

From multiple concurrent executables (cpl6) to one executable: time flow easier to understand, easier to debug

interface compatibility for ESMF-compliant components

Ported to IBM p6, Cray XT4/XT5, BGP, Linux Clusters, SGI


2. Current coupling technologies – Cpl7

Varying layouts on computing resources via external configuration (metadata) :

processors

tim

e

CPL (regridding, merging) 0.5 sec

CLM (land) 1.4 sec

CICE (ice) 5.0 sec

CAM (atm) 6.2 sec

Driver (controls time evolution)

POP (ocean) 8.2 sec

Sequential layout Hybrid layout

CPL 0.7 sec

CICE 8.4 sec POP 14.9 sec

Driver

CLM 2.3 sec

CAM 11.2 sec

7.7 sec

processors

tim

e

Hybrid layout

CPL 0.7 sec

CICE 8.5 sec POP 19.1 sec

Driver

CLM 2.3 sec

CAM 11.2 sec

5.2 sec

processors

tim

e

1.0 sec

Scaling evaluated on up to 10 000 processors: Craig et al., Int. J. High Perform. C,

2012


The Flexible Modeling System (FMS)

• Active for over a decade at GFDL; developed in F90

• FMS shown to be scalable with up to O(10000) pes

Software to assemble a climate model with domain-specific “slots” for atmosphere, ocean, ocean surface including sea ice and land surface

FMS Superstructure

FMS Infrastructure

User code

• FMS “Superstructure”: • Domain-specific coupling layer (“stubs” (no component), or “data” also possible)• Components “wrapped” in FMS-specific data structures and procedure calls• Single executable with serial or concurrent execution of components• Regridding, redistribution, or direct (hard-coded) exchanges between components• Includes data assimilation

2. Current coupling technologies – FMS


• Interface fluxes must be globally conserved atmosphere water-land fractions adjusted to fit ocean sea-land mask quantities are transferred from the parent grids to the exchange grid,

where fluxes are computed; they are then averaged on the receiving grid

2. Current coupling technologies – FMS

• Exchanges consistent with physical processes occurring near the surface

nn TAT 1

atmosphere

land

Implicit calculation of vertical diffusive fluxes over the whole column

Up-down sweep for tridiagonal matrix resolution through the exchange grid

Atm

Land

Exchange

FMS “Superstructure” obeys specific geophysical constraints


2. Current coupling technologies – OASIS

Communication and regridding library to exchange data between independent models with minimal level of interference in the codes (external configuration through namelist-like file)

• Initialization: call oasis_init(...)• Grid definition: call oasis_write_grid (...)• Local partition definition: call oasis_def_partition (...)

Ap

pli

cati

on P

rog

Inte

rfac

e

• Coupling field exchange: in model time stepping loop

call oasis_put (…, time, var_array. …) call oasis_get (…, time, var_array, …)• user external configuration: => source or target (end-point communication)

=> effective coupling frequency=> transformations and regridding

• Developed by CERFACS since 1991 with CNRS since 2005 and many others• Written in F90 and C; open source license (LGPL)• Last OASIS3-MCT version based on MCT • Large community of users: ~35 climate modelling groups world-wide


• Toy coupled model: coupling exchanges between NEMO ORCA025 grid (1021x1442) and Gaussian Reduced T799 grid (843 000)

• Bullx Curie thin nodes (Intel® procs Sandy Bridge EP; IFort 12.1.7.256, Bullx MPI 1.1.16.5)

• IBM MareNostrum3 (Intel Sandy Bridge processors, Intel MPI 4.1.0.024)

Coupling overhead for one-year long simulation with one 1 coupling exchange every hour in each direction between codes with O(1 M) grid points running on 4000 cores/component:

~60 seconds for initialisation, ~30 seconds for data exchange

2. coupling technologies – OASIS


Outline

Introduction


2.Current coupling technologies used in climate modelling• ESMF• Cpl7/CESM• FMS• OASIS



Conclusions


OCN +

ICE

ATMn-1 n

TO

TI

AlbO

AlbI

U/V/WO+I

Few coupling algorithms – EU ESMs

Taux/y/z O+I

LH+SH+LWO

LH+SH+LWI

DQ/DTSWO

SWI

EV+SB-R-SNWnd10mWndmod

RunoffCalving

n+1

All fluxes calculated in the atmosphere-surface model For most: Double ocean-ice flux calculation in the atmosphere For some: sea-land fraction in atm defined by interp of ocean mask For most: addition of a runoff model ECMWF: wave-model in addition

CNRM-CM5, IPSL-CM5, EC-Earth, CMCC-ESM, MPI-ESM, MetOffice GC2.0


NEMO+

LIM

GEM

1 2 3

TO/I

SHAO/AI

LHAO/AI

TauxAO/AI

TauyAO/AI

Ice FrachI

hS

SWA

LWA

PR TA

UA

VA

qA

PA

P0

SHAO/AI (TA, TO/I, UA, ZM, ZH)LHAO (qA, qO, TA, TO, UA, ZM, ZH)LHAI (qA, qI, TA, TI, UA, ZM, ZH)Taux/yAO/AI (U/VA, TA, TO/I, U/VO/I, ZM, ZH)

SWA (ALO/I –not from O/I)LWA ( TO

4, TI4 )

Environment Canada

1

4

2 3 4


Ocn

Atmn-1

Ice

n

Lnd

Cpl accatm/ocn

flx

merge

atm/ocn flx

merge

acc

Atm : SWd, LWd, rain, snow -> Lnd, Ice, Ocn (via acc)

Lnd, Ice, Cpl : LWu, LH, SH, EV, taux/y albedo -> Atm, Ocn (via atm/ocn flx)Ice : heat & salt fluxes, SWpen -> Ocn (via acc) + Runoff, Glacier, …

Few coupling algorithms – CESM


Different technical coupling approaches used in climate modellingValcke et al, 2012: Coupling technologies for Earth System Modelling, GeoscMD.

• External coupler and/or coupling library (e.g. OASIS): easiest solution to couple independent codes but some drawbacks

• Integrated approach: split original codes and rebuild a new code (e.g. ESMF)

more efficient in many cases but puts more constraints on the components

The “best” coupling tool is not uniquely defined; it depends on: • ready to change/adapt your original codes• efficiency you want/need to achieve, etc.

Conclusions

Diverse approaches for flux calculation at the component interfaces:• Dictated by science? By numerical constraints?• Ad-hoc developments given code constraints?• We need to better understand the impact into ESMs

Series of Coupling Technology Workshops: • Toulouse Dec 2010• Boulder Feb 2013: “Int Working Committee on Coupling Technologies”

launched • Next: Manchester 20-22 April 2015


The end


NEMO+

LIM2

IFS

1

TO/I

UO/I

VO/I

Ice frac

UA

VA

SWA

LH+SH+LWPREV

WAM

TauxTauyStokes driftTurb energy

Stokes driftRoughness

2

TO/I

UO/I

VO/I

Ice frac

UA

VA

SWA

LH+SH+LWPREV

TauxTauyStokes driftTurb energy

Stokes driftRoughness

Few coupling algorithms – ECMWF

WAM: wave-model One-executable with sequential execution of components


TA

hA

θ

φT

hI

Tf

ψ

Ψ = hA (θ – TA) ; Ψ variable d’état; θ CI

T =Tf –φ/hI ; T variable d’état; φ CI

T(n) =Tf –(hA/hI) θ(n) + (hA/hI) TA

converge si hA/hI <1 ; diverge sinon

Classiquement, hI = 4 W/m2/K ; hA/hI ~5couplage atmosphère-glace diverge

Pour l’océan ho~∞, donc hA/ho ~0couplage atmosphère-ocean stable


1*- Sensible heat flux

6*- Evaporation + int. energy [+ Qlat]

3x- Fresh water flux4x- Salt flux7x- Mass of snow and ice

1- Temperature at sea-ice base3- Highest level temperature (SST)4- Ocean radiative temp.8- Absorbed solar radiation (in 1st layer)

5- Surface ocean current7- Surface height 7

1- Continental runoff + internal Energy 8

1*- Surf. Temp2*- Surf. Roughness3*- Displacement height4x- Surface velocity

5

1- Rainfall + int. energy2- Snowfall + int. energy

3- Incoming solar radiat.4- Solar zenith angle5- Fraction of diffuse solar radiation6- Downward infrared radiation7- Sensitivity of atmos temp. & humidity to surf. fluxes

1 1

2*- Surf. emissivity 3*- Albedo, direct4*- Albedo, diffuse5*- Surf. radiative temp.

2

2

Land surface model

Ocean model

Atmosphere model

Ocean surface module

Sea ice model (wave model)

Surface layer turbulence

6

1x- Non solar heat flux2x- Solar radiation

7*- Wind stress

5x- Wind stress6x- U^3

8- Subgrid fractions

Note on subgrid fractiondependance:<>x- Sea Ice categories (incl. open ocean)<>*- Sea Ice or Land Surf. categories

8- Subgrid fractions2- Salinity at sea-ice base6- Sea surface salinity

1*- Cd 42*- Ce 43*- Ch

1- Surface pressure2-4 Air temperature, humidity and wind5- Mean scalar wind speed6- Height of these variables 3

3

Few coupling algorithms – PRISM


Typed Data Transfert (Potsdam Institute for Climate – PIK)

•library to transfer data between programs in a platform and language independent way with different protocols (Unix Socket, I/O from/to files, MPI) -> heterogeneous coupling

Current coupling technologies

Open-PALM (CERFACS - ONERA)•Tool originally designed for creating data assimilation suites•Dynamic driver and CWIPI coupling library (ONERA) performing parallel data exchanges, parallel calculation of weights, parallel remapping•Multiphysic coupling for ~40 different applications mainly in France

Bespoke Framework Generator (U. Manchester):

•Generator of wrapper code around components to build a coupled model with a chosen coupling technology (curr. OASIS, MPI, argument passing) based on external metadata for flexible model composition and deployment

•Used for 20 configurations in CIAS (Comm Integrated Assessment System)www.cs.manchester.ac.uk/cnc/ projects/bfg

Model Coupling Toolkit (Argonne National Lab):

•library and related datatypes for parallel field exchanges, parallel data transformation for creating parallel couplers / coupled models

•used in CESM/Cpl7 and OASIS3-MCT www.mcs.anl.gov/research/projects/mct/


Ocn

Atm1

Ice

2

Lnd

Cpl accatm/ocn

flx

merge

Atm : SWd, LWd, rain, snow -> Lnd, Ice, Ocn (via acc)

Lnd, Ice, Cpl : LWu, LH, SH, EV, taux/y albedo -> Atm, Ocn (via atm/ocn flx)Ice : heat & salt fluxes, SWpen -> Ocn+ Runoff, Glacier, …

accatm/ocn

flx

merge

Few coupling algorithms – CESM


I.3 Time sequence of exchanges

Atm

SLT

OSM

Oce

7

5

6sea ice t

Frequency of coupling exchanges:

F7 = F6 < F5 = F3 = F1 = F4 = F2

slow fast

3

1

4+3

Comp.Fluxes

2

atm t (implicit)

3

1

4+3

25

3

1

4+3

2

3

1

4+3

2

7

deep ocean t

Part I - On a revised ocean-atmosphere physical coupling interface

5 5

6


Separation of fast ocean + sea ice surface processes involving heat, water and momentum exchanges with the atmosphere from slower deeper ocean processes.

Calculation of fluxes at the resolution of the surface (would be

non-physical to regrid the turbulent exchange coefficients Cd, Ce,

Ch).

Implicit calculation of energy fluxes from the base of the sea-ice to the top of the atmosphere.

Part I - On a revised ocean-atmosphere physical coupling interface

Comments and conclusions

• Increased modularity with SLT and OS modules.

• SLT runs on finer grid and computes surface turbulent coefficient.

• OS computes radiation and turbulent fluxes.

Date post:	21-Dec-2015
Category:	Documents
Upload:	frederick-small
View:	214 times
Download:	1 times

WWOSC 2014, Montreal, August 16-21 2014 A review of technologies and algorithms used to couple Earth...

Documents