Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Large-Scale Granular and Fluid (DEM/SPH) Simulations using Particles
Large-Scale Granular and Fluid (DEM/SPH) Simulations using Particles
1
Takayuki Aoki
Global Scientific Information and Computing CenterTokyo Institute of Technology
SC14 NVIDIA booth talk, November 19, 2014, New Orleans
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Compute Node(3 Tesla K20X GPUs)
Performance: 4.08 TFLOPSMemory: 58.0GB(CPU)
+18GB(GPU)
Rack (30 nodes)
Performance: 122 TFLOPSMemory: 2.28 TB
System (58 racks)1442 nodes: 2952 CPU sockets,
4264 GPUs
Performance: 224.7 TFLOPS (CPU) ※ Turbo boost5.562 PFLOPS (GPU)
Total: 17.1 PFLOPS
TSUBAME 2.5TSUBAME 2.5
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
3
TSUBAME SupercomputerTSUBAME Supercomputer
Graph 500No. 3 (2011)
wire
文部科学大臣表彰
(2012)
Gordon Bell Prize (2011)
Tesla S1070X170(680GPU)
Tesla K20XTesla M2050
CUDACOE
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
4
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Weak Scalability: 2.0000 PFLOPS on 4,000 TSUBAME2.0, 330 billion cells44.5 % the peak performance
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Granular Material Simulationsusing Discrete Element Method
Granular Material Simulationsusing Discrete Element Method
7
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
8
Golf Bunker ShotsGolf Bunker Shots
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Contact interaction
Normal direction
Tangential direction
SpringViscosity
Viscosity
Friction
Spring
ijijij xkxF
Simulation for Granular MaterialsSimulation for Granular MaterialsDEM (Discrete Element Method)DEM (Discrete Element Method)
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
10
In 2005In 2005
■ kn = 5×108 dyn/cm■ Time Integration:
2-stage Ruge-Kutta
■ = 8×104 dyn・sec/cm■ t = 4×10-7 sec
DEM (Discrete Element Method) 76,000 Particles: 48 hours
Future work:
CPU 0 CPU 1 CPU 2
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
2 dimensional slice-grid method
Dynamic Load Balance Dynamic Load Balance
Many particles
no particle
2. Move boundary
1. Move boundary
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
2 dimensional slice-grid method
Dynamic Load Balance Dynamic Load Balance
Many particles
no particle
2. Move boundary
1. Move boundary
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Computational domain is dynamically decomposed into 64 sub-domains.
Slice grid
Dynamic Domain DecompositionDynamic Domain Decomposition
KD-tree Octree
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
• Particle Collision detection of particles with complex shapes described by CAD data is efficiently carried out by using Level Set Function.
Collision Detection using Level Set FunctionCollision Detection using Level Set Function
Particle
Polygon ofCAD data Φ > 0 Φ < 0Negative areaPositive area
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPULevel Set Function describing CAD surfaceLevel Set Function describing CAD surface
Surface patches of CAD data Level Set Function
negative distance area far from the surface
positive distance area far from the surface
• Generation from 3D CAD data on the uniform mesh• Fast generation algorithm and inside/outside judgment
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Neighbor Particle ListNeighbor Particle List
Local domain0 6
3
0
0 6 3
NULL
87 percent of memory usage is reduced compared to regular neighbor list.
Linked-list methodLinked-list method
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUDEM using non-spherical particles
Considering more realistic shapes of rocks, non-spherical particles are used in DEM.
Many spherical particles with rigid body connections
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Using spherical particles,
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Using non‐spherical tetrapod particles,
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUMultiple GPU ScalabilityMultiple GPU Scalability
• Conditions Particles : 2 × 106, 1.6 × 107, 1.29 × 108
Domain Decomposition: Dynamic load Balance using Slice Grid Method Time-Integration : 2-stage
Runge-Kutta
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUSPH for Fluid DynamicsSPH for Fluid Dynamics
: Kernel function
First derivatives
h
h : Kernel radius
Particle interaction within a kernel radis
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
A list of Particle Difference Operators
Improved SPHImproved SPH
Interpolation
Gradient
Divergence
Laplacian
2nd polynomial function (Spiky shaped):
r
Generalization of Finite Difference Operators (Imoto, Tagami 2014)
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
• Explicit Time-integration using Predictor-corrector Method
Improved SPHImproved SPH
Predicator
Collector
Temporary pressures are calculated from Birch-Murnaghan’s equations:
Positions are computed as follows:
Pressures are computed as follows:
(1)
(2)
(3)
(4)
(5)
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUA Dam Break SimulationA Dam Break Simulation
• Initial setting and Parameters
12 m
2.2 m
0.8 m
Water
Object 6 m10 m
4.8 m
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUDescription of the Object ShapeDescription of the Object Shape
• A object is represented by particles arrangement generated from CAD data
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPUA Dam Break SimulationA Dam Break Simulation
72 M particles with 80 GPUs
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
Fluid-Structure Interaction
Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology
GP GPUGP GPU
33
SUMMARYSUMMARY
Particle Method (DEM/SPH) based on short-range interaction are also suitable for GPU computing as well as stencil computation.
Successful many granular simulations GPU-based supercomputer TSUBAME 2.0/2.5 have been shown.
Fluid simulations using SPH is suitable to describe free-surface flows.
Particle methods can be applied to Fluid-Structure Interaction easily.