Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Introduction & PhilosophyIntroduction & Philosophy
Peter Böhm, inuTech GmbH
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Outline Diffpack® -Session
14.00-15.00 Introduction & Philosophy
15.00-15.40 Main Steps Developing Your Own Simulator
16.20-16.40 Easy Use of Advanced Numerical Technologies
16.40-17.40 Complementary Use to Main Stream Analysis
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Outline
• inuTech - Innovative Numerical Technologies
• Diffpack - an overview
• Diffpack - the main philosophy
• Diffpack - how to realize
• Examples
– Heart
– Tsunami
– CFD
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
inuTech
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
FSV Consulting R&D, founded 1995
inuTech GmbH was established in June 2000
Share Holders: Prof. Schittkowski,
CADFEM International AG, F. Vogel
Currently 10 employees
FSV Consulting R&D, founded 1995
inuTech GmbH was established in June 2000
Share Holders: Prof. Schittkowski,
CADFEM International AG, F. Vogel
Currently 10 employees
inuTech - Our History
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
inuTech - Our Objectives
Closing the Gap between
Engineering and Mathematics
to Solve Engineering and
Mathematical Problems
R&D of Numerical Methods Sales and Support of
Software
R&D of Numerical Methods Sales and Support of
Software
Seminars and Training Seminars and Training
Consulting Consulting
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Structural Optimization Topology Optimization Shape Optimization Sizing- and Parameter Optimization Large Scale Optimization Multi-Disciplinary Optimization Design of Experiments Biological Growth Free Material Optimization
(Composite Design)
Structural Optimization Topology Optimization Shape Optimization Sizing- and Parameter Optimization Large Scale Optimization Multi-Disciplinary Optimization Design of Experiments Biological Growth Free Material Optimization
(Composite Design)
inuTech - Our R&D Experience
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Project Partner
Universität Erlangen
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack - in general
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Differential Equations inuTech develops and markets the Diffpack Product
Line for the Numerical Modeling and Solution of Differential Equations
inuTech offers Consulting Services around Diffpack; we can deliver customized turn-key solutions for specialized simulation problems
Attention: Diffpack is a development environment, not an application!It allows you to generate asimulator (or to use an already developed simulator!)
Differential Equations inuTech develops and markets the Diffpack Product
Line for the Numerical Modeling and Solution of Differential Equations
inuTech offers Consulting Services around Diffpack; we can deliver customized turn-key solutions for specialized simulation problems
Attention: Diffpack is a development environment, not an application!It allows you to generate asimulator (or to use an already developed simulator!)
inuTech - Our R&D Experience
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Univ. of Oslo
Univ. of Erlangen
Univ. of Bayreuth
The Diffpack® Team
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Diffpack® Distribution
Diffpack Development Framework
Diffpack Kernel(computing/numerics)
Diffpack Toolboxes
Third-Party Pre-/Postprocessors
OpenSource Pre-/PostprocessorsDiffpack
Utility
Programs
Books, Manpages,Reports
Supportphone/e-mail
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Applications
The Diffpack® Environment
Diffpack Kernel
Ada
ptiv
ity
Par
alle
l LA
Dat
a F
ilter
s
Par
alle
l DD
Mul
tigrid
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Diffpack® Documentation
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
List of Applications
• electrical activity in the heart (and body)
• continuous casting
• simulation of a solid-oxide fuel cell
• chemical reactor flow
• parasitics in high power electronics
• option pricing in finance
• water waves - structure impact
• hyperspherical black hole simulation
• tsunami simulation
• virtual rock physics laboratory
• heat treatment of cancer
• ...
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack - main philosphy
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack Kernel
Diffpack Toolboxes(further extensions)
Diffpack Kernel
Diffpack Toolboxes(further extensions)
The Diffpack® Vision
[ ] in
on
u f
u g
κ−∇ ⋅ ∇ = Ω= ∂Ω
“arbitrary” PDE, ODE,
or othernumerical problem
“arbitrary” PDE, ODE,
or othernumerical problem
+ = ownsolver
ownsolver
( ), ,v tt
∂ =∂s
F s
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Diffpack® Vision (cont’d)
Structuralmechanics
Porous mediaflow
Aero-dynamics
Incompressibleflow
Other PDEapplications
Waterwaves
Electromagnetics
Heattransfer
Field
Grid
MatrixVector
I/O
Ax=b
FEM
FDM
Observation:Methodology basisindependent of applications
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
• Flexible building blocks to develop solvers for non-standard oder specialized simulation problems
• for building highly customized ”trimmed-down” solverswith minimum overhead of unused functionality andunnecessary complexity
• Complementary to standard FEM-programs (i.e.ANSYS, CFX, etc. ...) to fill in the gaps in functionality
• Benefits are:• Applicable to all simulation problems that can be modeled by Differential Equations
• Exact modeling of physical effects• Control of numerical solution algorithm
Diffpack® – Tailored simulation
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems who require the abilityto do self-programming
Solverfor PDE1
STRUCT
• Scenario: Build and test (or use existing) solver for PDE1 (STRUCTURAL)
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT
Solverfor PDE2
EMAG
•Build (or use existing) solver for PDE2 (Electro Magnetics)•Test EMAG Solver standalone •Couple EMAG Solver with Structural Solver
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT EMAG
Add in Heateffects
Heat
•Add in Heat effects to coupled solver
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT EMAG
Solverfor PDE3
CFD
Heat
•Build (or use existing) solver for PDE3 (CFD)•Test CFD Solver standalone •Couple CFD Solver
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT EMAG
CFD
Heat
Solverfor ODEs
ODEs
•Add ODEs (chemical reactions, kinematic, etc. ...) to the coupled system
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT EMAG
CFD
Heat
AddAdaptivity
ADAP
ODEs
•Add Adaptive Functionality
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT EMAG
CFD
Heat ADAP
ODEs
Parallel Funct.
PARALLEL
•Add Parallel Functionality
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
STRUCT EMAG Heat ADAP
PARALLEL
•Do numerical experimentation and adjustments
NA1
NA2
Tune Domain Decomposition
NA4
…
Add Criteriafor elementrefinement
NA5
NonlinearSolver Adjustment
Add TissueMaterial model
NA3
Incorporatenew elementformulation
CFDODEs
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
for complex, new problems …
PrePost
Add advancedPre- and
Postprocessing
STRUCT EMAG
CFD
Heat ADAP
ODEsPARALLEL
NA1
NA2
NA3
NA4NA5
•Add advanced Pre- and Postprocessing
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
PrePost
for complex, new problems …
PrePost
Add GraphicalUser Interface
GUI
WB
Could beANSYS
Workbench!
STRUCT EMAG
CFD
Heat ADAP
ODEsPARALLEL
NA1
NA2
NA3
NA4NA5
•Add GUI
Diffpack® – Flexible Building Blocks
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
At the end we have for example a solver to simulate the electrical activity in the human heart (Courtesy Simula Research, Oslo)
PrePostt
GUI
STRUCT EMAG
CFD
Heat ADAP
PARALLEL
NA1
NA3NA5
NA5
Heart Solver
Example – Heart Simulation
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Diffpack® Intention
satisfy the criteria of numerical software by using Diffpack
• efficiency
• reliability
• robustness
• extensibility
• portability
• user-friendliness
focus on your own problem-specific parts
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Challenge
Software DevelopmentSoftware Development
Num
eric
al M
etho
dsScientific Computing
Speed-Up 1,000,000 over last 50 years
Speed-Up 1,000,000 over last 50 years
Hardw
are Power
getSpeed-Up
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Diffpack® Philosophy
• do not reinvent the wheel
• trust (to a certain extend) former developments
• make use of OOP in numerics -> Object Oriented Numerics
– inheritance
– virtual function
– polymorphism
• support high abstraction level
• offer suitable building blocks (modular design)
– stay as close as possible to the math
– avoid translation errors
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
The Diffpack® Philosophy
• do not patronize the developer
– allow for arbitrary equations / combination of equation
– work on weak formulation
– allow for user defined extensions
• user defined new finite elements
• user defined new preconditioners
• user defined new solvers
• avoid bottle-necks of OOP
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
software specialistwrites a program tosolve the problem
The Diffpack® Philosophy - build a common language
engineerhas a
problem
physicisthas a
physical description
comm
unication
mathematician numeric specialist
has a set of equations / solution
methods
software
available
engineer get datawhich tell him how
to solve theinitial problem
comm
unication
system admin / hardware specialisthas computer to run
on the program
comm
unication
comm
unication
comm
unication
communication
Diffpack: common languageDiffpack: common languagegives
defines
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack - how realized
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack® Implementation - high abstraction level
Example: Solve A*x=b by a CG method:
CALL NSPCP (PREC,ACCEL,NDIM,N,MAXNZ,COEF,JCOEF,P,IP,U,
UBAR,RHS,WKSP,IWKSP,NW,INW,IPARM,RPARM,IER)
Procedural language approach (ITPACK, Netlib):
OON (C++, Diffpack):
//Given Matrix A, Vector x, Vector bLinEqSystem system (A,x,b);LinEqSolver solver (method);
solver.solve(system);
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
C+
+ C
lass
esC
++
Cla
sses
Diffpack® Implementation - inheritance
FEMFEMFEM
Fundamental library class for any simulation
OwnSimulatorOwnSimulatorOwnSimulatorGeneral implementation of the problem. Focus on numerics
Inheritance
SimCaseSimCaseSimCase
Inheritance
Fundamental library class for finite element methods
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack® Implementation - inheritance (cont’d)
FEMFEMFEM
OwnFDMSimulatorOwnFDMSimulatorOwnFDMSimulator
SimCaseSimCaseSimCase
FDMFDMFDMFVMFVMFVM
MxFEMMxFEMMxFEM
OwnMxSimulatorOwnMxSimulatorOwnMxSimulator
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack® Implementation - use weak formulation
[ ] in
on
u f
u g
κ−∇ ⋅ ∇ = Ω= ∂Ω
solve on operator level ×1
, 1,..., in
on
m
j i j ij
N N d u f N d i m
u g
κ= Ω Ω
∇ ∇ Ω + = Ω = Ω
= ∂Ω
∑ ∫ ∫
Use weak formulation
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack® Implementation - important member
global_menu.multipleLoop (S)
S::adm
S::define
S::scan
S::solveProblem
S::integrands
FEM::makeSystem
FEM::calcElmMatVec
FEM::numItgOverElm
S::resultReport
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
bAu
midNfudNNvdNNk i
m
jjijij
=
=Ω=
Ω∇⋅+Ω∇∇ ∫∑ ∫∫
Ω= ΩΩ
,...,1 , 1
Weak Formulation ( ):0=α
ConvDiff:: integrands (numerical kernel)
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Apply Boundary Conditions:
2 3 4( , ) 0, , , ,u x y x y= ∈Γ Γ Γ “essential conditions”
ConvDiff :: fillEssBC (apply essential boundary Conditions)
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Examples
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Courtesy of Simula Research Lab
Accurate 2D solution: 1,000,000 elements, 32 proc essors, 4 hours, 1 Gb Accurate 3D solution: 900.000.000 unknowns, 64 proc essors, 1000s per time step , 312Gb
Accurate 2D solution: 1,000,000 elements, 32 proc essors, 4 hours, 1 Gb Accurate 3D solution: 900.000.000 unknowns, 64 proc essors, 1000s per time step , 312Gb
4 3 coupled PDEs - 1 in torso, 2 in heart4 12 coupled ODEs sitting in each node4 Solved by finite elements using
4Diffpack standard FEM tools4multigrid methods4adaptive gridding (wave front)4Parallel solution using Domain Decomposition
/ODEs solved in parallel4 Sub-problem simulators build and tested
separately. Joined by administration class.4 Dimension independent code4 Around 10,000 lines of code
4 3 coupled PDEs - 1 in torso, 2 in heart4 12 coupled ODEs sitting in each node4 Solved by finite elements using
4Diffpack standard FEM tools4multigrid methods4adaptive gridding (wave front)4Parallel solution using Domain Decomposition
/ODEs solved in parallel4 Sub-problem simulators build and tested
separately. Joined by administration class.4 Dimension independent code4 Around 10,000 lines of code
Diffpack® - Electrical Activity in the Heart
3D Case:440.000.000 nodes in the heart41.000.000 nodes in the body4900.000.000 unknowns update
every time step4About 1000 sec per time step4Optimal preconditioning, 64
processors: 15 days
3D Case:440.000.000 nodes in the heart41.000.000 nodes in the body4900.000.000 unknowns update
every time step4About 1000 sec per time step4Optimal preconditioning, 64
processors: 15 days
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
( )
( ) ( ) ( )( ) ( )( )eeii
eiim
uMMvM
uMvMsvIt
vC
svFt
s
∇+⋅∇+∇⋅∇=
∇⋅∇+∇⋅∇=+∂∂
=∂∂
0
,
,
ionχχ
ty tensorconductivicellular Extra
ty tensorconductivicellular Intra
ratio volume toarea surface Membrane
ecapacitanc Membrane
current ionic Total
functions Rate
gates) and tions(concentra vector State
potential, aneTransmembr
potentialcellular Extra
where
ion
e
i
m
ei
e
M
M
C
I
F
s
uuvv
u
χ
−=
Courtesy of Simula Research Lab
Diffpack® - Electrical Activity in the Heart
3D Case:440.000.000 nodes in the heart41.000.000 nodes in the body4900.000.000 unknowns update
every time step4About 1000 sec per time step4Optimal preconditioning, 64
processors: 15 days
3D Case:440.000.000 nodes in the heart41.000.000 nodes in the body4900.000.000 unknowns update
every time step4About 1000 sec per time step4Optimal preconditioning, 64
processors: 15 days
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Tsunami Simulation - Storegga (Norway)
• Slides/impact
• Large destructive water waves
Courtesy of International Centre for Geohazards
Diffpack® - Tsunami Simulation
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Large-eddy simulation of flow around two cylinders in atandem arrangement, Re=22000,
velocity isosurface and pressure distribution
Viscous 3D flow around a cylinder, Re=265,velocity isosurface
Diffpack® - Computational Fluid Dynamics
Courtesy of SINTEF, Applied Mathematics
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Diffpack® Benchmarks
-251197/
953 s
Origin3800/ Itan.-Clu.(2)
32 /
16
10.4 Mio+ 150 Mio
Electro-cardiac(1)
Simula
(2005)Heart
1 / proc.
1
8
200
RAM
[Gb]
--Origin 3800963 MioVISTA-
Marine CFD
SINTEF
(2004)Flow
23.9/
23.7
725 /
563
Origin2000/
PIII-Clu.
32 /
321 Mio
3D-Nonlin. acoustics
Simula
(2000)Sound
19.4780 sItan.-Clu.2011 MioBoussinesqSimula
(2005)Tsunami
-5263 /
2840 sOrigin3800
32 /
64
84 Mio +
1260 Mio
Electro-cardiac
Simula
(2004)Heart
Speedup
WT(3)
[s]SystemCPU
un-
knownsEquation
Author
(Year)Project
(1)2 PDEs & 30 ODEs, (2)Linux Itanium2 Cluster (1.3 GHz, 64bit), (3)wall clock time (per time-step)
Innovative Numerical
Technologies
23. CADFEM Users´ Meeting, Bonn 09.-11. November 2005
Thank you for attention !Thank you for attention !