- How does the coupled modeling system work?

and

- Setting up a coupled application

Model Coupling ToolkitMathematics and Computer Science Division Argonne National Laboratoryhttp://www-unix.mcs.anl.gov/mct/

MCT is an open-source package that provides MPI based communications between all nodes of a distributed memory modeling component system.

Download and compile as libraries that are linked to.

Warner, J.C., Perlin, N., and Skyllingstad, E. (2008). Using the Model Coupling Toolkit to couple earth system models. Environmental Modeling and Software

Coupled Modeling System

Model A running on M nodes.

Model B running on N nodes.

Model C ………

(it also works here)

MCT providescommunications

between all models.………

Libraries

MCT - v2.60 or higher (distributed)

1) cd to the MCT dir

2) ./configureThis makes Makefile.conf. you can edit this file.

3) make

4) make install

5) set environment varssetenv MCT_INCDIR COAWST/Lib/MCT/includesetenv MCT_LIBDIR COAWST/Lib/MCT/lib(or where ever you installed them, see last slide)

Compilers dir (side note)

Model organization

master.F

mpi_init

initrun

finalize

initrun

finalize

{{

SWAN

ROMS

init_file (# procs/model)

init, run, and finalizeROMS SWAN

init(grid decomp)

run(sync. point)

finalize

init_paraminit_parallelinit_scaclarsinit_couplingro

ms_

init

rom

s_ru

n

main3d .....

waves_coupling...

mpi_finalizeclose_io

rom

s_fi

nali

ze

MPI_INIT

SW

INIT

MP

IS

WM

AIN

SW

EX

ITM

PI

SWINITSWREAD (grid)

init_coupling

swanmain .....

ocean_coupling...

mpi_finalizeclose_io

Grid decomposition (during initialization)

ROMS SWAN

- Each tile is on a separate processor.

- Each tile registerswith MCT.

init_coupling

ROMS- init_coupling SWAN- init_coupling

1

1

22

3

3

processed by each ROMS tileprocessed by each SWAN tile

Synchronization (run phase)

ROMS- ocean_coupling SWAN- waves_coupling

MCT

MCT

processed by each ROMS tile processed by each SWAN tile

ATM to OCN data fields

ATM

Uwind, V

wind, P

atm

, RH, T

air,

cloud

, rain

, eva

p, S

Wra

d, L

wrad

LH, H

FX, Ustr

ess,

Vstres

s

OCN

Integration and Application Network (ian.umces.edu/symbols),University of Maryland Center for Environmental Science.

Uwind, VwindSwrad, Lwrad,RH, Tair, cloud

Ustress, Vstress,Swrad, LwradLH, HFX

Use momentum + heat fluxes computed

in WRF for both ROMS+WRF

Use wrf vars in COARE algorithm

rain, evap

Salt flux

or#define BULK_FLUXES#define ATM2OCN_FLUXES

#define ATM_PRESS - Patm

#define EMINUSP

stflx_temp = Swrad+Lwrad +LH+HFX

stflx_salt = evap - rain

LH + HFX computed in bulk_fluxes

OCN to ATM data fields

OCN

ATM

WAV

SST

Hwave , L

pwave ,

Tpsurf ,

Sur

face

flu

xes

Momentum

Heat

Moisture

= f ( Hwave, Lpwave, Tpsurf )

SST OCN

WAV

How to create coupled application 1) Create all input, BC, init, forcing, etc files

for each model as if running separately. I recommend that you run each model separately first.

2) modify cppdefs in your header file. 3) SCRIP (if different grids or grid refinement) 4) coupling.in 5) coawst.bash 6) run it as coawstM

- Handout ends here- More in the online ppt

- Classroom tutorial will now follow:Projects/Sandy/create_sandy_application.m

1) Use each model separatelyWRF

27 vertical levels dt 36 s Physics

Lin microphysics RRTM longwave, Dudhia

shortwave Mellor-Yamada-Janjic (MYJ) PBL Kain-Fritsch (KF) cumulus scheme

6 km grid

ROMS 16 vertical levels dt 240, 48 Physics

GLS turbulence closure COARE bulk fluxes BC's from HYCOM Timestep = 240s

5km and 1 km grid(s)

These models are on different grids.

2) sandy.h

3) SCRIP - grid interpolation

Atmosphere model provides heat flux to cover entire ocean grid. SCRIP interpolations weights needed to remap data fields.

Ocean model provides higher resolution and coupled response of SST to atmosphere. But the ocean grid is limited in spatial coverage so atmosphere model must combine data from different sources, which can create a discontinuity in the forcing.

http://climate.lanl.gov/Software/SCRIP/

Flux conservativeremapping scheme

Ocean grid5 km

Atm Grid6 km

HFLX10 GFS data

SST

Libraries SCRIP - v 1.6 (distributed)Used when 2 or more models are not on the same grid.

1) cd to COAWST/Lib/SCRIP/source dir2) edit makefile3) make

3) SCRIP Need to create SCRIP weights using

COAWST\Tools\mfiles\mtools\ create_scrip_weights_master.m

3) SCRIP There will be created a weight file between each grid.So if you have 1 WRF grid, and 2 ROMS grids, it will produceatm1_to_ocn1_weights.ncatm1_to_ocn2_weights.ncocn1_to_atm1_weights.ncocn2_to_atm1_weights.nc

WRF grid 1

1000 m

5000 m

ROMS grid 1

ROMS grid 2

4) coupling.in (this is a ROMS+WRF app)

set # procs for each model

set coupling interval. Can be different for each direction.

input file names. only 1 for WRF, 1 for ROMS, multiple for SWAN

SCRIP weights are listed here

4) namelist.input

set # procs for atm model

need dt of 30 to divide evenly into coupling interval

of 1200 sec.

6) run it as coawstM

use total number of procs from coupling.in only 1 executable

Processor allocation

stdout reports processor allocation

This looks like from a different run,but you get the idea

Processor allocation

"Timing for …." = WRF

"1 179 52974 02:59:00 " = ROMS

Here is where the model coupling synchronization occurs.

so probably could re-allocate more nodes to WRF

JOE_TC - test case examples

JOE_TC test cases are distributed applications for testing ROMS+WRF coupling JOE_TCw = wrf only JOE_TCs = same grid, roms + wrf coupled JOE_TCd = different grids for roms and wrf,

needs scrip weights