A massively parallel solid mechanics solver in ALYA
A massively parallel solid mechanics solver in ALYA
E. Casoni, G. Houzeaux, M. Vázquez and X. Sáez
JOCS - Càlcul d’altes prestacions en engineria
Barcelona, November 21th, 2013
CASE Department
CASE Department - Objectives
Computer Applications in Scienceand Engineering (CASE)Department
Interdisciplinary research unit of theBSC-CNS
Our mission:
To develop computational tools to simulate highly complex problemsseamlessly adapted to run onto high-end parallel supercomputers
Alya system - Large Scale Computational Mechanics
Alya is one of the two CFD codes of the PRACE benchmark suite.
Supercomputer Physical model CPUʼs
Lindgren (Sweden) NSI 12288
Huygens (The Netherlands) NSI 2128
Jugene BG (Germany) NSI, FSI 16384
Fermi BG (Italy) NSI + particles + species transp 16384
Curie Bullx (France) NSI, thermal flow 22528
Marenostrum (Spain) NSI , thermal flow, compressible 5000
Alya in PRACE
Impact for company Impact for BSC
- Simulation of large and complex tests: experimental tests unavailable!
- Demonstration of value of HPC to real world problems.
- Reduction of development cycle and cost. - Sensitivity analysis: more accurate results.
- Support for future product design and development. - Evolutions of simulation depending on the
industry requirements: need feedback to improve!- Analyze hypothetical scenarios.
HPC in Industry
Alya-SOLIDZ module
- Alya was initially designed for CFD: large meshes and expensive computations.
- The objective in Alya design is twofold:- To run with the highest efficiency standards in large scale supercomputing facilities.- To solve multi-physics problems in a coupled way.
Why Alya-SOLIDZ?
ITER vacuum vessel, property of F4E
- New challenges in solid mechanics: large structures, small details thatrequire fine meshes, ...
- Well-known robust commercial codes.. but need to be adapted formassively parallel architectures.
- No OpenSource codes.
- Need to solve more involved problems: coupling FSI, electro-mechanical,termo-mechanical, ...
The Alya-SOLIDZ module
ITER vacuum vessel, property of F4E
Main Alya
Solidz
Turnon DoIter Output
Initialize problem Set time step
METIS
Parallel service
Solver: Au = b
ImplicitExplicit
Constitutive model
Temper Nastin
SOLIDZ module
- Total Lagrangian formulation with FEM numerical method.
- Large deformations.
- Dynamic and static solution.
- Implicit (BCGStab, GMRES, Deflated CG) and Explicit schemes.
- Built-in material models (linear isotropic, neo-Hookean, Holzapfel-Ogden, inclusion of fibers, ...)
- Anisotropic materials (including orthotropic models).
- Plug-in interface to ABAQUS user-built subroutines.
- Overset meshes.
- Multiplier in order to refine the mesh.
Main features
Alya-SOLIDZ module
Fracture simulation
- Extended finite Element method (XFEM).
- Modeling the propagation of discontinuities (cracks) such that the path is independent of the mesh.
- Easily adapted to the code: does not modify the FEM routines.
- Open door to contact problems, fatigue...
Simulation of fracture process
Simulation of fracture process
- Implementation of fracture models for large scale simulations in composite laminate, focusing on the scalability of algorithms.
- Fracture simulation at the intra-laminar (XFEM) and inter-laminar (CZM) level.
- Validate models implementation against experiments.
-XFEM and CZM have been integrated in Alya with the aim of performing simulations of fracture in fiber laminates with different sequence.
Work done in collaboration with IMDEA materials and Oxford University
Alya-SOLIDZ module
Hybrid parallelization
- Multicore Processor is a processing system composed of two or more independentcores.
- Multicores run at lower frequencies, are more power efficient and get betterperformances than single cores, but their programming is difficult.
Recently, the most important trend in computer architecture is adding multiple cores inside a socket.
Parallelization: hybrid OpenMP / MPI
Hybrid parallel programming: mixing some parallel programming paradigms in order to exploit different levels of parallelism.
- Why? To exploit the task-level and the thread-level parallelism of multicore architecture.
- How? Assign a MPI-task to each node and an OpenMP thread per core.
- What do we win? Reduce the communication cost inside the node.
TaskShared memory
Parallelization: hybrid OpenMP / MPI
Real Industry case
Visualization team of CASE: Fernando Cucchietti and Guillermo Marín
Shear stress of a milling cutter punch (10M elements and 3000 cpu’s)
- Initial mesh of 500000 elements (multiplier)
- 3.9 Milion elements solved on 1000 cpu’s
- Hybrid mesh
- Run on Marenostrum III (48896 Inter Sandy Bridge processors in 3056 nodes)
Real Industry case: fusion reactor with magnetic loads
ITER vacuum vessel, property of F4E
Alya-SOLIDZ module
Coupled problems
- Avoid MPI barriers in order to reduce total time. - Improve the baseline speedup of 50%.- Reduce the communication between zones.
FSI coupling
FSI coupling: parallel strategy
force
The coupling code
- Alya architecture is modular: kernel, modules, and services. - Different solvers can be used on each module.- Coupling between modules by imposing continuity of variables.
Zonal strategy
- Load balancing between processors. - All the processors are working. - Define local communicators between zones for assembly and a global one for convergence check.
FSI coupling: scalability of the modules
FSI coupling: example
Pressure pulse in a compliant vessel
FSI coupling: example
Pressure wave propagation with the fluid deformed configuration.
Velocity of the fluid
FSI coupling: parallel strategy
Electro-mechanical model of the heart using medical images and mathematical models for the electric and mechanical coupling.
Cardiac- fluid-electro-mechanical coupling
Cardiac- fluid-electro-mechanical coupling
Cardiac- fluid-electro-mechanical coupling
- Using HPC systems will make numerical simulation closer to commercial computing and real-life simulations.
- Alya-Solidz module to solve large problems for nonlinear finite elements for continua and structures.
- New improvements to fit industry requirements: simulation of fracture process, coupling multi-physics, ...
- Hybrid version will take profit of the massively multicore architectures.
Concluding remarks
Thank you for your attention!