George Em KarniadakisDivision of Applied Mathematics

The CRUNCH group: www.cfm.brown.edu/crunch

Cross-Site Simulations on the TeraGrid

spectral elements Micro / Nano-fluidics parallel computing

Grand-Challenge Problem 1: Turbulence – Drag crisis(Tightly-Coupled Problem)

• Turbulence – Last frontier in classical physics

• Climate, environment, transport, energy,…

• Re=300,000 (CPU ~ Re3) requires 20 Billion DOFs

• Memory 4 TBytes

Wave Propagation in a Model of the Arterial Circulation

(Data of 55 main arteries from J.J. Wang and K. Parker, 1997)

Grand-Challenge Problem 2: Human Arterial Tree(Loosely-Coupled Problem)

First ParallelTeraGrid

Paradigm

NCSA IA64SDSC IA64

in-sitecommunication

Cross-sitecommunication

in-sitecommunication

TG Site TG Site

Whole flowDomain

All-to-all

-5/3

DNS versus Experiments: max Re=10,000

DNSExperiments

(Rockwell, 2004)

Energy Spectrum

Black – simulation

Blue - experiment

RMS velocity

Turbulence: Single-Site Performance

Fixed problem size Fixed workload

• PSC: Compaq Alpha EV68, 1 GHz

• 300 Million DOFs, 2-level MPI

• MPICH-G2 and MPI perform similarly (SDSC/IA-64)

• Half processors from NCSA, half from SDSC

• Intel IA-64 processors (Itanium-2, 1.5 GHz)

• Slow-down factor 1.5SDSC TG

NCSA TGFFT Matrix

transposition

Turbulence: Cross-Site Performance

Fixed problem size Fixed workload

P(t)

W1

W2

Ascending aorta

U(t)

Inflow conditions U(t)

P(t)

Thoracic aorta

Femoral

P(t)

U(t) W1

W2

U(t)

Tibial

P(t)

Outflow conditions(Peripheral resistance)

1D Model – Sherwin et al. / Imperial College

Platelet Aggregation in Arterioles and Platelet Aggregation in Arterioles and VenulesVenules

FLOW

Parameters: Vessel diameter - 50 µm, vessel length - 400 µm, blood velocity - 100 µm/s,platelet diameter - 3 µm, platelet concentration - 300000/mm3, platelet density - 1.03 fluid densitySimulation time - 28 s

venules

platelet aggregate

Growth Rate vs. Blood VelocityGrowth Rate vs. Blood Velocity

Experiments: Begent and Born, Nature, Vol. 227, No. 5261, pp. 926-930, 1970

Second Parallel TeraGrid Paradigm

Multiscale Simulation of Arterial Tree

Arterial-Tree: Cross-Site Performance

(Homogeneous Network)

• Three arteries; 4 Million DOFs per artery

• 1CPU/node on ANL; 2CPUs/node on NCSA/SDSC

• No slown-down, full scalabilitySDSC TG

ANL TG

NCSA TG

Fixed problem size Fixed workload

SDSC TG

NCSA TGPSC TG

Arterial-Tree: Cross-Site Performance

(Heterogeneous Network)

•PSC connects to TG via

application gateway (qsockets)

•Two arteries per site

•PSC proc:2 GF vs 6 GF IA-64

New Unique Capability

• Potentially unlimited salability; Enabling technology

– Integrate “real and virtual” in projects like:– Digital human, digital ocean, digital space, …

• Predictability and Uncertainty – Stochastic simulations– Prediction vs. Postdiction– Risk-based/Reliability-based design– Sensitivity analysis – steering of experiments

(e.g., DDDAS concept)

• Inverse Problems– Engineering design– Biomedical sciences– Geological/Climate Modeling

What Users Need

• Debuggers for TG (a la TotalView)• New topology-aware parallel algorithms• Sustained network/cluster performance• TG visualization capability• Middleware

– Robust MPICH-G2– Co-scheduling– Network & Globus diagnostics– Authentication/Security – often in conflict

• Consultants/Referees with TG-Expertise