COLLABORATIVE DESIGN AND DEVELOPMENT OF THE

COMMUNITY CLIMATE SYSTEM MODEL FOR TERASCALE

COMPUTING (CDDCCSMTC)Hereinafter referred to as the CCSM Consortium

Phil Jones (LANL)

On behalf of all the consorts

The SciDAC CCSM Consortium consists of PI: R. Malone4, J. Drake5 , Site-Contacts: C. Ding2, S. Ghan6, D. Rotman3, J. Taylor1, J. Kiehl7, W. Washington7, S.-J. Lin8, Co-Is: J. Baumgardner4, T. Bettge7, L. Buja7, S. Chu4, T. Craig7, P. Duffy3, J.

Dukowicz4, S. Elliot4, D. Erickson5, M. Ham5, Y. He2, F. Hoffman5, E. Hunke4, R. Jacob1, P. Jones4, J. Larson1, J. Lamarque7, W. Lipscomb4, M. Maltrud4, D. McKenna7, A. Mirin3, W. Putman8, W. Sawyer8, J. Schramm7, T. Shippert6, R. Smith4, P.

Worley5, W. Yang2

 1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Los Alamos National Lab,

5Oak Ridge National Lab, 6Pacific Northwest National Lab, 7National Center for Atmospheric Research, 8NASA-Goddard Space Flight Center

Climate System

Climate Change

Science Goals• Assessment and prediction

– IPCC, national assessments (alarmist fearmongering)

– Energy policy (Dick Cheney’s private sessions)• Regional climate prediction

– High resolution, downscaling, water!• Atmospheric chemistry/ocean

biogeochemistry– Carbon cycle– Aerosols

Project Goals• Software

– Performance portability– Software engineering (repositories,

standardized testing – No Code Left Behind initiative)

• Model Development– Better algorithms– New physical processes (esp. chemistry,

biogeochemistry)

Community Climate System Model

OceanPOP

IceCICE/CSIM

AtmosphereCAM

LandLSM/CLM

Flux Coupler

7 States10 Fluxes

6 States6 Fluxes

4 States3 Fluxes

7 States9 Fluxes

6 Fluxes 11 States10 Fluxes

6 States13 Fluxes

6 States6 Fluxes

Once

OnceOnce

Once

perper

perper

day

hour

hour

hour

NSF/DOE270 Participants

Coupler ArchitectureIssues:Issues: •sequencingsequencing•frequencyfrequency•distributiondistribution•parallelism parallelism •single or multiple single or multiple executablesexecutables•stand alone executionstand alone execution

Version 1.0 ReleasedNovember 2002

• MPH3 (multi-processor handshaking) library for coupling component models

• CPL6 -- Implemented, Tested, Deployed

• ESMF/CCA

Prediction and AssessmentMany century-scale

simulations (>2500yrs) @~5yrs/day

Cycle vampires:Many dedicated cycles

at computer centers

Performance Portability• Vectorization

– POP easy (forefront of retro fashion)– CAM, CICE, CLM

• Blocked/chunked decomposition– Sized for vector/cache– Load balanced distribution of blocks/chunks– Hybrid MPI/OpenMP– Land elimination

• Performance modeling w/PERC

Performance

Rapid Climate Change

THC Crisis Center

Polar and THC

HYPOP• Arbitrary Lagrangian-Eulerian vertical

coordinate– Keep Lagrangian in deep ocean– Remap to z-coordinate in mixed layer– CSU SciDAC

• New time stepping/mode splitting• Progress

– Model currently working in z-coord mode– Examining vertical grid generators– Testing

CICE• Incremental Remapping for Sea Ice and

Ocean Transport – Incremental remapping scheme that proved to

be three times faster than MPDATA, total model speedup of about 30% --added to CCSM/CSIM

– CICE3.0 restructered for vector Community Sea Ice Model

• Sensitivity analysis and parameter tuning test of the CICE code – Automatic Differentiation (AD)-generated

derivative code

Regional Prediction

Mississipi State

Stanford

Kentucky

Oklahoma State

Atmosphere/Land

Resolution and Precipitation

CCM3 extreme precipitation events depend on model resolution. Here we are using as a measure of extreme precipitation events the 99th percentile daily precipitation amount. Increasing resolution helps the CCM3 reproduce this measure of extreme daily precipitation events.

(DJF) precipitation in the California region in 5 simulations, plus observations. The 5 simulations are: CCM3 at T42 (300 km), CCM3 at T85 (150 km) , CCM3 at T170 (75 km), CCM3 at T239 (50 km), and CAM2 with FV dycore at 0.4 x 0.5 deg.

Subgrid Orography Scheme

• Reproduces orographic signature without increasing dynamic resolution

• Realisitic precipitation, snowcover, runoff

• Month of March simulated with CCSM

Eddy-Resolving Ocean

0.1 deg0.28 deg

Obs 2 deg

Greenhouse Gases• Energy production• Bovine flatulence• Presidential campaigning

•Source-based scenarios

Aerosol Uncertainty

Atmospheric Chemistry• Gas-phase chemistry with emissions, deposition, transport and photo-

chemical reactions for 89 species. • Experiments performed with 4x5 degree Fvcore – ozone concentration at

800hPa for selected stations (ppmv)• Mechanism development with IMPACT

– A)    Small mechanism (TS4), using the ozone field it generates for photolysis rates.

– B)     Small mechanism (TS4), using an ozone climatology for photolysis rates.

– C)    Full mechanism (TS2), using the ozone field it generates for photolysis rates.

Zonal mean Ozone, Ratio A/C

Zonal mean Ozone, Ratio B/C

Ocean Biogeochemistry• LANL Ecosystem Model

– nutrients (nitrate, ammonium, iron, silicate)– phytoplankton (small, diatom, coccolithophores)– zooplankton– bacteria, dissolved organic material, detritus– dissolved inorganic carbon (DIC), alkalinity– trace gases (dimethyl sulfide, carbonyl sulfide,

methyl halides and nonmethane hydrocarbons)– elemental cyclings (C,N,Fe,Si,S)

Ocean Biogeochemistry

•Iron Enrichment in the Parallel Ocean Program•Surface chlorophyll distributions in POPfor 1996 La Niña and 1997 El Niño

Global DMS Flux from the Ocean using POP

The global flux of DMS from the ocean to the atmosphere is shown as an annual mean. The globally integrated flux of DMS from the ocean to the atmosphere is 23.8 Tg S yr-1 .

Things not mentioned…• Software engineering

• Other model improvements

• fvcore work

• land model (river transport, biogeochem, etc.)

• Ocean grid/topography generator

• Parallel I/O work

• ESG

• Now how much would you pay? You also get…