CCSM4

Tony Craig

NCAR

Dec 5, 2008

Topics

• CCSM4 Overview• CCSM4 Usage• Adding new components• Status of CCSM4 Development

Subversion

• setenv SVNREPO

https://svn-ccsm-models.cgd.ucar.edu• setenv SVNDATA

https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/• You can browse these on the web• svn co $SVNREPO/ccsm4/tags/ccsm4_0_alpha38• http://bb.cgd.ucar.edu/

– Look for “Subversion Issues”, “Introduction to Subversion”• Google subversion

CCSM4 Directory Structure

ccsm4_0_alpha38

ChangeLog models scripts

atm lnd ocn icedrv csm_shareutils

pop domxocn socn docn7seq_mct shreshr dshr

piomct esmf_tmtimingpilgrimdriver

seq_ccsm_drv.F90

Inputdata

• for common shared input datasets that are likely to be reused; grids, external forcing, initial files, etc

• local area is /wrkdir/maslowsk/RACM/inputdatasubject to scrubbing

• permanent area is/archive/u2/wes/maslowsk/RACM

• Using these areas– directories should be rwxrwxr-x– files should be r--r--r--– use unique filenames (date stamped)– never modify or remove ANY inputdata files

• some automated scripts for backup and recovery from Gabriele

Model Machine Layout(processor layout set in env_mach_pes)

Driver

lnd ice

cpl

atm

ocntime

processors

Coupling Issues

• Grids– global vs regional– consistent masks/fractions– staggering/averaging

• Frequency– tight coupling of atm/lnd/ice (hourly+)– loose coupling of ocn (daily)

• Consistency of coupling frequency, model timestep• Conservation, flux accumulation and averaging• Fields• Flux calculations• Albedo/shortwave consistency• Runoff

Topics

• CCSM4 Overview• CCSM4 Usage• Adding new components• Status of CCSM4 Development

Compsets, Grids, Machines

• Note: Compsets and Grids have a short name and a long name• Compsets are the combination of components used

– X = all dead models– A = all data models– B = all active models– others are C, D, F, G, H, I

• Grids define the resolution– standard resolutions have lnd grid = atm grid, ice grid = ocean grid

(f19_g15)– recently added “trigrid” support, atm and land grids different

(f45T42_g35)• Machine is, for example, midnight

> cd scripts> ./create_newcase -list

create_newcase (1)

> cd scripts > cat README > ./create_newcase -help

• must specify case name, compset, resolution, and machine. these cannot be changed inside the case directory.

• always use -skip_rundb option

> ./create_newcase -case Btest -res f45_g35 -compset B -mach midnight -skip_rundb

• a case called Btest will be generated in the scripts directory• the -case argument can be dir/name

create_newcase (2) > cd Btest

• edit env files– env files are mixture of xml and csh– env_mach_pes sets the pe layout. this must be done before

configure is run.– env_run sets run length, restart frequency, etc– set LOGDIR to $CASEROOT/logs in env_mach_generic– set DOUT_S to FALSE in env_run.xml

> configure -case • configure generates build and run scripts plus the Buildnml_Prestage

and Buildexe directories• some env files are locked after configure. to reset, type “configure -

cleanall”

create_newcase (3)

> ./Btest.midnight.build > qsub Btest.midnight.run

• The case will build under $EXEROOT, $EXEROOT is set in env_mach_generic.xml

• The case will run under $EXEROOT/ccsm_se

• Things I haven’t described: – setting up a full production run– setting the pe layout– archiving– how the scripts work– Buildnml_Prestage and Buildexe directories

Output files

• LID is a date string, yymmdd-hhmmss created by CCSM4 everytime it runs to generate unique filenames

• log files, *.log.$LID– each component has own log file. stdout/stderr are redirected to

ccsm.log.$LID.– look in $EXEROOT/ccsm_se– look in $LOGDIR if set– look in short term archive area if DOUT_S is TRUE

• timing files– directory in $CASEROOT/timing– ccsm_timing_summary.$LID has performance summary

• history and restart files– look in $EXEROOT/ccsm_se– look in short term archive area if DOUT_S is TRUE

create_test (1) > cd scripts > ./create_test -help

• must specify testname; $test.$resolution.$compset.$machine

• recommend using -testid

> ./create_test -testname ERS.f45_g35.B.midnight -testid t001

• create_test automatically configure the test, designed to require no user intervention.

• will create a test named ERS.f45_g35.B.midnight.t001• -testroot option allows the testcase to be place in another directory• generate and compare are supported for regression testing

create_test (2)

> cd ERS.f45_g35.B.midnight.t001

> ./ERS.f45_g35.B.midnight.t001.build

> qsub ERS.f45_g35.B.midnight.t001.test

• Test results are in the file TestStatus• The file TestStatus.out provide more test result details• log files should be copied into the test directory under a

directory called logs

Topics

• CCSM4 Overview• CCSM4 Usage• Adding new components• Status of CCSM4 Development

*_comp_mct.F90 (1)

• The *_comp_mct.F90 provides the coupling between the driver and the component. This is the top level of any component

• Driver looks like

call lnd_init_mct()

while (.not. stop_alarm)

call seq_timemgr_clockAdvance()

call lnd_run_mct()

end do

call lnd_final_mct()• Must have an init, run, and finalize method

*_comp_mct.F90 (2)

• Interface generally contains– ESMF_Clock– seq_cdata datatype that holds

• mct_gGrid (domain information)• mct_gsMap (decomposition information)• seq_infodata_type (scalar data)• MPI Comm, Component ID, etc

– Attribute vectors (2d fields), one for data to the component, one for data from the component.

• Use an existing version and modify it for your component

init_mct method

• enter• extract mpicom using seq_cdata_setptrs. this is THE mpi

communicator for your component.• get ccsm4 settings using seq_infodata_GetData()• initialize and synchronize your component clock using

seq_timemgr_EClockGetData()• set the global seg map which describes the grid decomposition of your

component.• set the global grid which prescribes grid field data of your component.• allocate the attribute vectors to be used to pass data back and forth.• pack the export attribute vector with initial data• set some ccsm4 settings using seq_infodata_PutData()• return

• NOTE: there are literally dozens of examples as each component has implemented this. Review and Reuse that code.

run_mct method

• enter

• check ccsm4 settings using seq_infodata_GetData()

• get clock information using seq_timemgr_EClockGetData()

• unpack the import attribute vector

• “run your component”

• pack the export attribute vector

• set some ccsm4 settings using seq_infodata_PutData()

• verify the internal clock is in sync with the CCSM4 clock

• return

Coupling Interface Final Notes

• The driver will hand you an mpi communicator. You must use this inside your component.

• The components must check that they are staying in sync with the driver clock.

• The component must respond to a restart flag sent from the driver.

• The albedo/shortwave interaction between components is complicated and critical. This will probably cause you problems at some point.

A few other issues

• Unit numbers• History files• Namelist input• Mapping files for coupler

Component Template Scripts

• Under each component is a bld directory and a script called *.cpl7.template.

• This script is what the CCSM4 scripts execute to generate component specific scripts under your $CASEROOT Buildnml_Prestage and Buildexe directories.

• New components will have to implement a version of this for their component.

Topics

• CCSM4 Overview• CCSM4 Usage• Adding new components• Status of CCSM4 Development

Status of CCSM4 development

• beta series• New science• GLC model• scripts continue to evolve• www.cgd.ucar.edu/cseg