Advanced Cyberinfrastructure: An Engine for Competitiveness

Steve Meacham

National Science Foundation

CASC Workshop

September 6, 2006

Outline

• NSF CI Vision• High-End Computing Portfolio• Data, COVO and LWD• The TeraGrid

What is cyberinfrastructure?

• Cyberinfrastructure for Science and Engineering Research and Education– Is the integration of the components of information technology

necessary to advance the frontiers of scientific and engineering knowledge

– Is the use of information technology to integrate research and education

– Makes possible new modes of experimentation, observation, modeling, analysis, and collaboration

– Is built with contributions from experts in many fields:- e.g. computer science, engineering and social science

• Examples of CI components:– Optical, wired electrical, and wireless networking; simulation tools; high-

performance computing; data analysis tools; data curation; tele-operation and tele-presence; visualization hardware and software; semantic mediation and query tools; digital workflows; middleware and high-performance system software; portal technology; virtual organizations and gateways; …

Strategic Plan(FY 2006 – 2010)

Ch. 1: Call to Action

Strategic Plans for:

Ch. 2: High Performance Computing

Ch. 3: Data, Data Analysis & Visualization

Ch. 4: Collaboratories, Observatories and Virtual Organizations

Ch. 5: Learning & Workforce Development

http://www.nsf.gov/dir/index.jsp?org=OCI

Principal components

HPC

LWDCOVO

High-Performance Computing

Learning and Workforce Development

Collaboratories, Observatories and Virtual Organizations

Data, Data Analysis, and Visualization

DATA

Cyberinfrastructure Components

Inside NSF

• Cyberinfrastructure Council• Office of Cyberinfrastructure• Directorate Cyberinfrastructure Working Groups• Directorate Cyberinfrastructure Programs

Office of CyberInfrastructure

Dan AtkinsOffice Director

José MuñozDep. Office Dir.

LDWPOC: Miriam Heller

COVOPOC: Kevin Thompson

HPCPOC: Steve Meacham

Judy HaydenJoann Alquisa

Priscilla BezdekMary Daley

Irene Lombardo

Program staff: Chris Greer, Miriam Heller, Fillia Makedon, Steve Meacham, Vittal Rao, Frank Scioli, Kevin Thompson

Data POC: Chris Greer

HPC hardware acquisitions, O&M, and user support as a fraction of NSF’s overall CI budget

CI Budgets

NSF 2006 CI Budget

75%

25%Researchdirectoratesand offices

OCI

NSF 2006 CI Budget

84%

7%

9%Other CI

HPC Hardware

HPCOperationsand UserSupport

NSF CI FY07 Budget Request

FY 2006

FY 2005 Current FY 2007Actuals Plan Request Amount Percent

Biological Sciences $77.00 $84.00 $90.50 $6.50 7.7%

Computer and Information Science and Engineering 45.32 63.00 68.00 5.00 7.9%

Engineering 52.00 52.00 54.00 2.00 3.8%

Geosciences 71.35 71.35 75.00 3.65 5.1%

Mathematical and Physical Sciences 56.52 59.30 63.56 4.26 7.2%

Social, Behavioral and Economic Sciences 20.39 20.54 20.54 - -

Office of Cyberinfrastructure 123.28 127.12 182.42 55.30 43.5%

Office of International Science and Engineering 0.22 1.00 1.05 0.05 5.0%

Office of Polar Programs 25.38 26.24 26.24 - -

Subtotal, Research and Related Activities 471.47 504.55 581.31 76.76 15.2%

Education and Human Resources 20.27 15.02 15.52 0.50 3.3%

Total, Cyberinfrastructure Funding $491.74 $519.57 $596.83 $77.26 14.9%

Totals may not add due to rounding.

Cyberinfrastructure Funding

(Dollars in Millions)

Change over

FY 2006

Examples of FY07 Areas of Emphasis

• Leadership-class HPC system acquisition• Data- and collaboration-intensive software services• Confidentiality protection and user-friendly access for major social and

behavioral science data collections• National STEM Digital Library (NSDL) supporting learners at all levels• CI-TEAM, preparing undergraduates, graduate students, postdocs and

faculty to use cyberinfrastructure in research and education• Support for the Protein Data Bank (PDB), the international repository for

information about the structure of biological macromolecules, and the Arctic Systems Sciences (ARCSS) Data Coordination Center

Principal components

HPCHigh-Performance Computing

Learning and Workforce Development

Collaboratories, Observatories and Virtual Organizations

Data, Data Analysis, and Visualization

HEC-enabled science and engineering

• Impacts in many research fields – E.g. model economies; analysis of multi-sensor astronomical data;

linguistic analysis; QCD & HEP analysis; cosmology; role of dark matter; chemistry; materials science; engineering; geoscience; climate; biochemistry; systems biology; ecosystem dynamics; genomics; proteomics; epidemiology; agent-based models of societies to test policy impacts; optimization; multi-scale, multi-science models

e.g. envt + soc sci, Earth system models, earthquake + structural engineering, …

• Transforming industry– Aircraft manufacturing; pharmaceuticals; engineering (inc nano- & bio-); oil

exploration; entertainment; automobile manufacturing, new industries based on information mining, …

• Part of the American Competitiveness Initiative

Why invest in HEC?

• High-performance computing as a tool of research is becoming ever more important in more areas of research– An inexorable trend over the past few decades– Shows no sign of stopping

• Current examples, future examples– Understanding life– Understanding matter– Understanding the environment– Understanding society

Why invest in HEC?

Understanding life

Satellite tobacco mosaic virus, P. Freddolino et al.

Aldehyde dehydrogenase, T. Wymore and S. Brown

Imidazole glycerol phosphate synthase, R. Amaro et al.

Why invest in HEC?

Understanding matter

I. Shipsey

Why invest in HEC?

Understanding the environment

K. Droegemeier et al.

CCSM

Why invest in HEC?

Understanding society

MoSeS:

A dynamical simulation of the UK population.

http://www.ncess.ac.uk/nodes/moses/BirkinMoses.pdf

M. Birkin et al.

John Q Public:

A computational model that simulates how voters' political opinions fluctuate during a campaign. S.-Y. Kim, M. Lodge, C. Taber.

Fe0.5Pt0.5 random alloy

L10-FePt nanoparticle

● LSMS- locally self-consistent multiple scattering method is a linear scaling ab initio electronic structure method (Gordon Bell prize winner)

● Achieves as high as 81% peak performance of CRAY-XT3

Wang (PSC), Stocks, Rusanu, Nicholson, Eisenbach (ORNL), Faulkner (FAU)

Magnetic NanocompositesWang (PSC)

• Direct quantum mechanical simulation on Cray XT3.

• Goal: nano-structured material with potential applications in high density data storage: 1 particle/bit.– Need to understand

influence of these nanoparticles on each other.

• A petascale problem: realistic simulations for nanostructures of ~ 50nm (~ 5M atoms).

Homogeneous turbulence driven by force of Arnold-Beltrami-Childress (ABC) form

VORTONICSBoghosian (Tufts)

• Physical challenges: Reconnection and Dynamos

– Vortical reconnection governs establishment of steady-state in Navier-Stokes turbulence

– Magnetic reconnection governs heating of solar corona

– The astrophysical dynamo problem. Exact mechanism and space/time scales unknown and represent important theoretical challenges

• Computational challenges: Enormous problem sizes, memory requirements, and long run times

– requires relaxation on space-time lattice of 5-15 Terabytes.

– uses geographically distributed domain decomposition (GD3): DTF, TCS, Lonestar

– Real time visualization at UC/ANL– Insley (UC/ANL), O’Neal (PSC), Guiang

(TACC)

Major Major Earthquakes Earthquakes on the San on the San Andreas Fault, Andreas Fault, 1680-present1680-present

19061906M 7.8M 7.8

18571857M 7.8M 7.8

16801680M 7.7M 7.7

• Largest and most detailed earthquake simulation of the southern San Andreas fault.

• Calculation of physics-based probabilistic hazard curves for Southern California using full waveform modeling.

• Computation and data analysis at multiple TeraGrid sites.• Workflow tools automate the very large number of

programs and files that must be managed.• TeraGrid staff Cui (SDSC), Reddy (GIG/PSC)

Simulation of a magnitude 7.7 seismic wave propagation on the San Andreas Fault. 47 TB data set.

TeraShake / CyberShakeOlsen (SDSU), Okaya (USC)

Searching for New Crystal Structures Deem (Rice)

• Searching for new 3-D zeolite crystal structures in crystallographic space

• Requires 10,000s of serial jobs through TeraGrid.

• Using MyCluster/GridShell to aggregate the computational capacity of the TeraGrid for accelerating search.

• TG staff Walker (TACC) and Cheeseman (Purdue)

HEC Program Elements

• Acquisitions – Track 1 - Petascale– Track 2 - Mid-range supercomputers

• Operations

• HEC System Software Development – Compilers, fault-tolerant OS, fault-survivability tools, system status

monitoring, file-systems, PSEs, …

• HEC Petascale Application Development – Scalable math libraries, scalable algorithms, data exploration tools,

performance profiling and prediction, large application development• Coordinated with other agencies

Acquisition Strategy

FY06 FY10FY09FY08FY07

Sc

ien

ce a

nd

en

gin

eerin

g

cap

ab

ility

(log

rithm

ic s

cale)

Track 3: Typical university HPC systems

Track 1 system(s)

Track 2 systems

Track 2 Acquisitions

• Individual systems - provide capabilities beyond those obtainable with university or state funds

• Collectively, as part of TeraGrid - provide a diverse HPC portfolio to meet the HPC needs of the academic research community

• Annual competition: roughly $30M/year for acquisition costs

• O&M costs via a TeraGrid RP award• Primary selection criterion: impact on science

and engineering research

Track 1 Acquisition (FY07-10)

• A system that will permit revolutionary science and engineering research

• Capable of delivering large numbers of cycles and large amounts of memory to individual problems

• Capable of sustaining at least 1015 arithmetic ops/second on a range of interesting problems

• Have a very large amount of memory and a very capable I/O system

• An architecture that facilitates scaling of codes• Robust system software with fault tolerance and fault

prediction features• Robust program development tools that simplify code

development• A single physical system in a single location

Track 1 Acquisition (FY07-10)

Examples of research problems:• The origin and nature of intermittency in turbulence• The interaction of radiative, dynamic and nuclear physics in stars• The dynamics of the Earth’s coupled carbon, nitrogen and hydrologic

cycles• Heterogeneous catalysis on semiconductor and metal surfaces• The properties and instabilities of burning plasmas and investigation

of magnetic confinement techniques• The formation of planetary nebulae• The interaction of attosecond laser pulse trains with polyatomic

molecules• The mechanisms of reactions involving large bio-molecules and bio-

molecular assemblages• The structure of large viruses• The interactions between clouds, weather and the Earth’s climate

HPC Operations

Track 1 & 2 - O&M for projected useful life awarded with acquisition funds- O&M approach assessed in review process

HPCOPS- An opportunity for universities w/o Track 1 or 2 funding but who

can leverage other funding to acquire large HPC systems- Will provide contribution to O&M in return for provision of HPC

resources to national S&E community- These will be TeraGrid RP awards – aligned w/ TG time frame- Expect to be highly competitive- Funding opportunity this year (Nov 28, 2006); do not anticipate a

similar competition next year

Possible third model?- Provide contribution to acquisition costs if institution picks up

O&M

Principal components

LWDCOVO

High-Performance Computing

Learning and Workforce Development

Collaboratories, Observatories and Virtual Organizations

Data, Data Analysis, and Visualization

DATA

Data CI: - Investments will continue to be prioritized by science and engineering research and education needs - S&E data generated with NSF funds will be accessible & usable - Data CI includes tools to manage, locate, access, manipulate, and analyze data, mechanisms to maintain confidentiality, and tools to facilitate creation and management of metadata- Data CI will involve strong, international, inter-agency and public-private partnerships

Challenges include: - Managing and analyzing very large datasets - Managing, analyzing, and using streaming data - Developing tools to permit research using confidential data

COVO and LWD: To appear (August)

Strategic Plans for Data, COVO and LWD

(FY 2006 – 2010)

Observatories - Based on ability to federate data-sets and data streams, some include instrument control, event detection and response, and some degree of virtualization - Examples: NVO, OOI, EarthScope, NEON, GEOSS

Virtual organizations - A geographically dispersed community with common interests that uses cyberinfrastructure to integrate a variety of digital resources into a common working environment

Supporting technologies - Portals, workflows, data analysis, models, streaming data, event detection, instrument/observatory control, networking, authentication/authorization, digital libraries, …

The growth of observatories and virtual organizations

IRNC

• Components– TransPAC2 (U.S. – Japan and beyond)– GLORIAD, (U.S. – China – Russia –

Korea)– Translight/PacificWave (U.S. – Australia)– TransLight/StarLight, (U.S. – Europe)– WHREN (U.S. – Latin America)

International Research Network Connections

CI-TEAM

• A Foundation-wide effort to foster CI training and workforce devel’t• Started FY05 ($2.5M) - focused on demonstration projects• Anticipated funding in FY06: $10M - small and large activities

FY05: - 70 projects (101 proposals) received -11 projects fundedBroadening participation in CI Alvarez (FIU) – CyberBridges Crasta (VA Tech) – Project-Centric Bioinformatics Fortson (Adler) – CI-Enabled 21st C. Astronomy Training for HS Science Teachers Fox (IU) – Bringing MSI Faculty into CI & e-Science Communities Gordon (OhSU) – Leveraging CI to Scale-up a Computational Science u/g Curriculum Panoff (Shodor) – Pathways to Cyberinfrastructure: CI through Computational Science Takai (SUNY Stonybrook) – High School Distributed Search for Cosmic Ray

Developing & implementing resources for CI workforce development

DiGiano (SRI) – Cybercollaboration between Scientists and Software Developers Figueiredo (U FL) – In-VIGO/Condor-G Middleware for Coastal and Estuarine CI Training Regli (Drexel) – CI for Creation and Use of Multi-disciplinary Engineering Models Simpson (PSU) – CI-based Engineering Repositories for Undergraduates (CIBER-U)

TeraGrid: an integrating infrastructure

TeraGrid

Offers:• Common user environments • Pooled community support expertise• Targeted consulting services (ASTA) • Science gateways• A portfolio of architectures

Exploring: • A security infrastructure that uses campus authentication systems • A lightweight, service-based approach to enable campus grids to federate with TeraGrid

TeraGrid: What is It?

TeraGrid: (1) Provides a unified, user environment to

support high-capability, production-quality cyberinfrastructure services for science and engineering research.

(2) Provides new S&E opportunities – by making possible new ways of using distributed resources and services

Examples of services include:• HPC• Data collections• Visualization servers• Portals

• Integration of services provided by grid technologies

• Distributed, open architecture.• GIG responsible for integration:

• Software integration (including the common software stack, CTSS)• Base infrastructure (security, networking, and operations)• User support• Community engagement (including the Science Gateways activities)

• 9 Resource Providers (with separate awards):

• PSC, TACC, NCSA, SDSC, ORNL, Indiana, Purdue, Chicago/ANL, NCAR• Several other institutions participate in TeraGrid as a sub-awardees of the GIG

• New sites may join as Resource Partners

Science Gateways

• Specific examples of Virtual Organizations• Built to serve communities of practice by bring

together a variety of resources in a customized portal

• Examples include:– NanoHub– NEES– LEAD– SCEC Earthworks Project– NVO

• http://www.teragrid.org/programs/sci_gateways/

Science Gateways

Biology and Biomedicine Science GatewayComputational Chemistry Grid (GridChem)Computational Science and Engineering Online (CSE-Online)GEON (GEOsciences Network)GIScience Gateway (GISolve)Grid Analysis Environment (GAE)Linked Environments for Atmospheric Discovery (LEAD)National Virtual Observatory (NVO)Network for Computational Nanotechnology and nanoHUBNetwork for Earthquake Engineering Simulation (NEES)Neutron Science Instrument GatewayOpen Life Sciences GatewayOpen Science Grid (OSG)SCEC Earthworks ProjectSpecial PRiority and Urgent Computing Environment (SPRUCE)TeraGrid Visualization GatewayThe Telescience Project

Goal: to create and maintain a powerful, stable, persistent, and widely accessible cyberinfrastructure to enable the work of science and engineering researchers and educators across the nation.

NSF-Cyberinfrastructure

Thank you.