The SciDAC2 CCSM The SciDAC2 CCSM Consortium ProjectConsortium Project

2007 - 20122007 - 2012

Presented by John Drake Presented by John Drake

Atmospheric Model Working Group Meeting January 31, 2007, NCAR, Boulder

The SciDAC2 CCSM Consortium Project:The SciDAC2 CCSM Consortium Project:A Scalable and Extensible Earth System ModelA Scalable and Extensible Earth System Model

Lead PI: John B. Drake, Oak Ridge National Laboratory, Co-Lead PI: Phil Jones, Los Alamos National LaboratoryArgonne National Laboratory (ANL) Robert Jacob, Ray Loy, Brookhaven National Laboratory (BNL) Robert McGraw; Lawrence Berkeley National Laboratory (LBNL) Michael Wehner,Lawrence Livermore National Laboratory (LLNL) Phillip Cameron-Smith, Arthur Mirin; Los Alamos National Laboratory (LANL) Scott Elliot, Philip Jones, William Lipscomb, Mat Maltrud; National Center for Atmospheric Research (NCAR) Peter Gent, William Collins, Tony Craig, Jean-Francois Lamarque, Mariana Vertenstein, Warren Washington; Oak Ridge National Laboratory (ORNL) Marcia Branstetter, John Drake, David Erickson, Forrest Hoffman,W. M. Post, Trey White, Patrick Worley; Pacific Northwest National Laboratory (PNNL) Steven Ghan, Xiaohong Liu, Richard Easter, Rahul Zaveri, Sandia National Laboratories (SNL) Mark Taylor

Current Activities (cont.)

Contact Info: drakejb@ornl.gov, pwjones@lanl.govProject Web Page: www.scidac.gov/climate/earth.html

Scientific ChallengeTo determine the range of possible climate changes over the 21st century and beyond through simulations using a more accurate climate system model that includes the full range of human and natural climate feedbacks with increased realism and spatial resolution

ObjectiveThe primary objective of this proposal is to develop, test, and exploit a first generation of Earth system models based upon the CCSM. We will bring to the community a new well-validated version of CCSM that will run efficiently on thousands of processors and include significant model enhancements.

Improvements to the representation of carbon and chemical processes for treatment of greenhouse gas emissions and aerosol feedbacks will be performed in collaboration with the DOE Atmospheric Science Program, DOE Atmospheric Radiation Measurement Program, and DOE Terrestrial Carbon Program. This project will enable climate change simulations required for scheduled national and international climate change assessments to which the CCSM is committed as part of the National Climate Change Science Program (CCSP) strategy.

Goals and DeliverablesOver the next five years, we will pursue this objective through four integrated goals:1. Extend the capabilities of the Community Climate System Model

(CCSM) to include representations of biological, ecological, chemical, and aerosol processes that will allow scientists and policy-makers to simulate climate and climate change using a comprehensive Earth system model,

2. Provide the necessary software and modeling expertise to rapidly integrate new methods and model improvements,

3. Pursue the development and evaluation of innovative methods in the coupled context of the CCSM, and

4. Improve the performance, portability and scalability of the CCSM on available and future computing architectures for use in national and international assessments of climate change.

Biogeochemical Coupling:Biogeochemical Coupling:

Scalability and Performance:Scalability and Performance:

Model Integration and Evaluation:Model Integration and Evaluation:

Collaborations

Scientific Applications (SAs) and Partnerships(SAPs)

Brookhaven National Laboratory Robert McGraw

Oak Ridge National Laboratory Patrick WorleyArgonne National Laboratory Kotamarthi RaoCenters for Enabling Technology

CollaborationsEarth System Grid (ESG) - Dean WilliamsPerformance Engineering (PER) – Bob LucasVACET – Wes Bethel ; SDM - Ari ShoshaniTOPS – David Keyes; ITAPS - Lori DiachinAPDEC - Phil Colella

Aerosols and Atmospheric ChemistryAerosols and Atmospheric Chemistry

Tasks and OrganizationEarth System ModelEarth System Model

Terrestrial carbon cycle and dynamic vegetationTerrestrial carbon cycle and dynamic vegetationAtmospheric chemistry and aerosol dynamicsAtmospheric chemistry and aerosol dynamicsOcean ecosystems and biogeochemical couplingOcean ecosystems and biogeochemical couplingFeedbacks between atmospheric composition and biogenic emissions

Model Model Integration and Evaluation and EvaluationIntegration and unit testingIntegration and unit testingNew icesheet and ocean modelsNew icesheet and ocean modelsNew atmospheric dynamics: finite volume (cubed sphere), New atmospheric dynamics: finite volume (cubed sphere), discontinuous Galerkin, others(icosahedral)discontinuous Galerkin, others(icosahedral)Frameworks for model evaluationFrameworks for model evaluation

Computational Computational PerformanceScalablity to thousands of processors, load balance, (fault recovery)Scalablity to thousands of processors, load balance, (fault recovery)

Computational Climate End Station for Climate Change Science - Warren Washington (PI)

Software enabling integration:• Model Coupling Toolkit (MCT 2.3.0) released (1/10/07)• Optimized coupling strategies• New methods for model evaluation (w/ Rao)

Software enabling scalability:• Collecting performance data to quantify and identify scalability limiters. Modifying code to improve scalability:• Enabling different degrees of parallelism in physics and dynamics of atmosphere model.• Identifying and eliminating nonscalable memory allocations.• Working with Pnetcdf developers and CCSM software engineers to implement a single parallel I/O interface across all CCSM components.• Validating software solutions on large Cray XT3 and IBM BlueGene systems.

Carbon Land Model Intercomparison (C-LAMP):•Determine the best set of terrestrial biogeochemistry process representations to include a future release of CCSM•Three models coupled to CCSM3: CLM3-CASA', CLM3-CN, and LSX-IBIS•Experiments are being performed with changing climate, CO2, nitrogen, and land use•Comparison with high frequency, high quality flux tower measurements

Atmospheric CO2 resulting from ecosystem respiration from the CLM3-CASA’

Ice sheet model:•Largest missing piece of physical climate model and necessary for sea level rise prediction•Variable resolution for melting at margins of ice sheetOcean vertical grids:•Arbitrary Lagrangian-Eulerian vertical grid for deep ocean circulation with good resolution in the surface mixed layerOcean horizontal grids:•Eddy-resolving simulations required for adequate simulation of ocean circulation•Unstructured grids provide ability to focus grids in eddy-active regions at reduced cost

High Resolution Ocean and Cryosphere:High Resolution Ocean and Cryosphere:

•Aerosols alter cloud properties and the distribution of radiation in the atmosphere•Development of a faster chemistry package for the troposphere and lower stratosphere•Introduction of modal and moment representations of aerosol dynamics modeling aerosol size distribution•Modeling the surface emissions of pollutants changes atmospheric radiation

Current Activities

Sea surface temperatures simulated on 0.1o displaced pole grid using the POP2 code

CVT Delaunay tessellation of the North Atlantic with 85K triangles - Gunzburger and Ju

AMWG Related ActivitiesAMWG Related Activities

Atm chemistry and aerosol dynamics (J.-F. Lamarque)Atm chemistry and aerosol dynamics (J.-F. Lamarque)– Aerosol Indirect effect - Steve Ghan, Bill CollinsAerosol Indirect effect - Steve Ghan, Bill Collins– Moment methods - Bob McGrawMoment methods - Bob McGraw– Atm chem fast mechanisms - Philip Cameron-SmithAtm chem fast mechanisms - Philip Cameron-Smith– Sulfur cycle - Dave EricksonSulfur cycle - Dave Erickson

Model Integration and Evaluation (M. Vertenstein)Model Integration and Evaluation (M. Vertenstein)– Integration and unit testing (Rob Jacobs, Tony Craig)Integration and unit testing (Rob Jacobs, Tony Craig)– FV (cubed sphere), DG, others (Mark Taylor, Art Mirin, Pat Worley, FV (cubed sphere), DG, others (Mark Taylor, Art Mirin, Pat Worley,

J. Drake)J. Drake)– Evaluation Frameworks (K. Rao et al.)Evaluation Frameworks (K. Rao et al.)

Computational Performance (Pat Worley, Art Mirin, Ray Loy)Computational Performance (Pat Worley, Art Mirin, Ray Loy)– Scalability, load balance, (fault recovery)Scalability, load balance, (fault recovery)

2004 2005 2006 2007 2008 2009

100 TF

1000 TF

250 TF

18.5 TF

Cray

TBD

50 TF

Supercomputers at NCCSSupercomputers at NCCS

Leadership Computing FacilityLeadership Computing Facility(also BG/L at Argonne)(also BG/L at Argonne)

Leadership Computing FacilityLeadership Computing Facility(also BG/L at Argonne)(also BG/L at Argonne)

18 TF Cray X1 / X1E

• Will not expand• 1024 processors• Vector processing for

sustained performance

50 TF Cray XT3

• 5212 dual core Opterons• 21 Terabytes memory• 100TB scratch

25 TF

60 TF Cray XT4

• 6296 dual core Opterons• New interconnect• Acceptance testing

Computer Allocation:Computer Allocation:Computational End Station for Climate Computational End Station for Climate

Change ScienceChange Science

– Extensible Extensible community modelscommunity models available for computational available for computational sciencescience

– Coordination of effort among agencies and institutions (talk to Coordination of effort among agencies and institutions (talk to Lawrence Buja, J. Drake or one of the partner representatives) Lawrence Buja, J. Drake or one of the partner representatives)

PI: Warren Washington (NCAR), partners: CCSM, COSIM, PCMDI, SciDAC,NASA-GSFC, PNNL, CCRI(Universities)

What does DOE want?What does DOE want? Relation to Aerosol Science Program and Terrestrial Carbon Relation to Aerosol Science Program and Terrestrial Carbon

ProgramProgram Coordinated enterprise, relation to CETs and SAPsCoordinated enterprise, relation to CETs and SAPs ReportingReporting

– Impacts and revised scope statementImpacts and revised scope statement– 30 days: 4-6 slides for Dr. Orbach30 days: 4-6 slides for Dr. Orbach– 60 days: Management Plan, website, performance baseline60 days: Management Plan, website, performance baseline– 6months: progress reports, highlights6months: progress reports, highlights

DOE OBER Performance Targets•2006 Deliver new measurements of clouds where observations are missing•2007 Include realistic cloud simulations in a climate model•2008 Measure ecosystem responses to climate change•2010 Develop/validate new models predicting effect of aerosols on climate forcing•2010 Provide climate model that links the Earth climate system with Earth’s biological systems•2013 Reduce differences between observed temperature & model simulations at sub continental scales using several decades of recent data•2015 Deliver improved climate data & models for policy makers to determine safe levels of greenhouse gasses.