A Particle Simulation With Accelerated Gyrokinetic Electron and Fully-kinetic Ion
(GeFi) CodeSpeaker:Wei Kong
(孔伟 )
Nankai university
Xueyi Wang1, Yu Lin1, Liu Chen2 , Huasheng Xie2 and Peng Wang3
1: Auburn university2: Zhejiang university
3: NVIDIA
Hangzhou, April 2012
ContentsContents
Introduction to GeFi Accelerated GeFi with CUDA electron beam instability in the center of Harris current
sheet
GeFi Simulation 2
Introduction to GeFi Introduction to GeFi Motivation
Important physics of kinetic process involves both electron and ion scales. (collisionless magnetic reconnection ranges from short electron scale to global Alfven scale).
Fully particle simulation often employ artificial mass ratio in order to accommodate limited computing resources. also suffers the numerically unstable due to rather more steps.
Hybrid simulation based on fully-ion & fluid-electron mainly solve problems related with only ion dynamics. Hybrid simulation based on fully-ion and drift kinetic-electron has a salient drawback that drops the important electron gyro-radius & polarization effects.
GeFi particle simulation aims at including both electron and ion kinetics, meantime computing efficiently. (First published by Yu Lin, Xueyi Wang, Zhihong Lin and Liu Chen at 2005)
GeFi Simulation 3
Introduction to GeFi Introduction to GeFi
ce
k k
GeFi Simulation 4
Advantages of GeFi simulation
Orders of magnitude improvements in time step and grid space are achieved by removing the rapid cyclotron electron-motion with Larmor radius retained, such that the super-computer can be greatly utilized to study a global problem such as collisionless magnetic reconnection.
Problems with disparate temporal and spatial scales (modes ranging from magnetohydrodynamic wave to kinetic wave) can be solved at an equal footing.
What GeFi could resolve:
Frequency with due to the gyro-averaging of electron motion.
Wave number with due to eliminate the high-frequency Langmuir oscillation along magnetic field.
Introduction to GeFi Introduction to GeFi GeFi kernals —— ion equations of motion
To keep symmetry in driving ions and particles:
then have
where and is the background magnetic field, is the perturbed scalar potential, and is the perturbed vector potential.
2
/
1/ 2 | / |
p mv qA c
H m p eA c e
( / )( / ) / i
p q v A cx p q A c m
A A A A A
GeFi Simulation 5
Introduction to GeFi Introduction to GeFi GeFi kernals —— electron equations of motion
Employ gyro-kinetic ordering for electrons
where and <…> indicates the gyro-averaging.
1
ee
ce
e
BkL B
k
* *
* *
[ ]
[ ]
p b q B
cR v b b q BqB
GeFi Simulation 6
* *( / ) ( ) , /e ceb b v b b b v A c
Introduction to GeFi Introduction to GeFi GeFi kernals —— potential equations (1) generalized Poisson’s equation
Ampere’s law
electron force balance equation
and were intentionally placed at left-hand side or right-hand side to calculate conveniently.
2 2*
2 * 2
2
* 2
4[(1 ) ] ( )(1 )
(4 )(1 )
pe e ee
ce e
e ep
e
n m cB
n qR HB
2 22
2 2
4( )( ) ( )pi peA A b b J Jc c c
2 2
2 * 2
2*
2
(4 )1[(1 ) ](1 )
1[(1 ) ( )]
pe e e
ce e e e
pep w B B
ce e e
n qn q B
R H H S Hn q
GeFi Simulation 7
Introduction to GeFi Introduction to GeFi
4 [ ]p i i i eR q n q N
( ) ,e e i iJ p F dp p f dp b
[ / ]B eg iS P J B c
2 2
44 [ ( ) ( )]e ep e i
n nH q J A A BB B
2*
2 *
1 [ ]1
pew e
ce e
H
* * *1 4[ ( ) ( ) ]4B e e B
JH A B A Hc
21 { [( ) ] / 2}4BH B B B
( )4cJ B b
,eg e e e eP m p p F dp
GeFi kernals —— potential equations (2)
* / 2e e
GeFi Simulation 8
* 2*(1 ) ( )
4 4e e
ee
B m cBq
Accelerated GeFi with CUDA Accelerated GeFi with CUDA
GeFi & CUDA A typical simulation with 256x64 grids, 100 particles per cell
and 4000 time steps (call 32 cores) will cost about 3 hours . For the 3D global simulation, much more longer. Is there a way to run faster?
Yes, with CUDA!
Get to know CUDA CUDA refers to a parallel computing architecture, which
mainly includes ISA instruction set(PTX) and hardware for graphics and computing(NVIDIA GPU). Through “the most-intensive computing” in GPU.
GeFi Simulation 9
Accelerated GeFi with CUDA Accelerated GeFi with CUDA
GeFi Simulation 10
(1) Diagnose to find which should be optimized:
……………….
Accelerated GeFi with CUDA Accelerated GeFi with CUDA
GeFi Simulation 11
(2) Redefine variables and functions in .cu file, as soon as possible avoid calling the outer
__forceinline__ __device__ void A_CROSS_B(float *A, float *B, float *C, float &ABSC…){ C[0]=A[1]*B[2]-A[2]*B[1]; C[1]=A[2]*B[0]-A[0]*B[2]; C[2]=A[0]*B[1]-A[1]*B[0]; ABSC = sqrt(powf(C[0],2)+powf(C[1],2)+powf(C[2],2));}
__global__ void acce_gpu_loop_kernel(float *QVE, float *QVE0, float *VGCE, int NE, float XMIN, float YMIN, float ZMIN)
Just copy source code!! ……
Accelerated GeFi with CUDA Accelerated GeFi with CUDA
GeFi Simulation 12
(3) At last wrapper the kernal to run at GPU, then transfer back to CPU
extern "C"{ void acce_gpu_loop_(float *QVE, float *QVE0, float *VGCE, int *NE, float *XMIN, float *YMIN, float *ZMIN, float *DX, float *DY, float *DZ, int *I0A, int *J0A, int *K0A, ……..}
souce code …CALL acce_gpu_loop( QVE, QVE0, VGCE, NE, XMIN, YMIN, ZMIN, DX, DY, DZ, I0A, J0A, K0A, I1A, J1A, K1A, NX, NY, NZ, BTOT_ALL, B_TOT_BAR …….)source code …
Accelerated GeFi with CUDA Accelerated GeFi with CUDA
Performance of upgraded GeFi
• Initial port of acce: ~ 30x– (NXT,NYT,NZT)=(1,65,257)– CPU acce time: 14 sec/step (PGI Fortran compiler, -O2)– GPU acce time: 0.47 sec/step (kernel: 0.15 sec)
optimizated by Dr. P. Wang (NVIDIA)
• Second port of accp: ~ up to 45x– (NXT,NYT,NZT)=(1,65,257)– CPU acce time: 14 sec/step (PGI Fortran compiler, -O2)– GPU acce time: 0.30 sec/step
GeFi Simulation 13
Electron beam plasmaElectron beam plasma Beam instability Exists in the laboratory and space plasma, may contributes to
the fast magnetic reconnection. Variations: cold and warm , weak and strong, isotropic and
anisotropic, linear and nonlinear… We more concerns about the ion beam plasma due to an
instability localized at the center of Harris current sheet.(Wang PoP et al., 2008).
GeFi Simulation 14
Electron beam plasma Electron beam plasma
GeFi Simulation 15
For benchmark, we begin with an electron beam plasma:
Cold dispersion relation(cold ion and cold electron beam, Verdon et al., PoP, 2011):
While our GeFi simulation includes the ion and electron thermal effects.
22 2 2 2 22 2
2 2 4 2 2 2 2 2
22 22
22 2 2
1 1 1
cos 0,
pe pe pe d
e e
pee e
i e e i
k vk c k c k c
m mm m k c
/ / dk v
Electron beam plasma Electron beam plasma
GeFi Simulation 16
Design the simulation: Y
B
beam X wave
Electron beam plasma Electron beam plasma
GeFi Simulation 17
Electron beam FK theory(line) & cold theory(line) & GeFi simulation(contour) GeFi simulation(circle)
23, 0.04, 0, 1.553 (88.98 )pei e de B wave
ce
V rad
0deV
Electron beam plasma Electron beam plasma
GeFi Simulation 18
Weak electron beam
(1) Consistent with higher- ranch (2) zero frequency branch,not grows. (up to 10 )
23, 0.04,
1.553 (88.98 )
pei e
ce
B wave rad
1i
70de iV V
Electron beam plasma Electron beam plasma
GeFi Simulation 19
Strong electron beam
(1) Forward propagating(2) Lower growth rate
Resonance with ion-Cyclotron motion?
23, 0.04,
1.553 (88.98 )
pei e
ce
B wave rad
300de iV V
Electron beam plasma Electron beam plasma
GeFi Simulation 20
More strong electron beam (Verdon et al. PoP, 2011 )
For such a distribution, Thermal nearly not play effects?
23, 0.04,
1.553 (88.98 )
pei e
ce
B wave rad
500de iV V
Electron beam plasma Electron beam plasma
GeFi Simulation 21
GeFi simulation (nonlinear)
Electron beam plasma Electron beam plasma
GeFi Simulation 22
Analyze(1):By , Bz, fluctuates at the same level. Ex >> Ey and Ez
Note: the main magnetic field is designed to Y direction.
Electron beam plasma Electron beam plasma
GeFi Simulation 23
Analyze(2):EM or ESES ?
610E
910E
Electron beam plasma Electron beam plasma
GeFi Simulation 24
Analyze(3):Resonance?
Which gives , so A large gap between and , even considers the width
of electron velocity distribution (300 162) Anyway ,next we show the phase condition of electrons and
ions(in the frame of moving wave with ).
23, 0.04, 300 , 1.553 (88.98 )
3.51, 3.75
pei e de i B wave
ce
total i ci
V V rad
k
0.1331ik 60p iV V
pV deV
60p iV V
iV
Electron beam plasma Electron beam plasma
GeFi Simulation 25
Analyze(4):eles phase condition(at the linear stage)
1 10.268 1.36i i
Electron beam plasma Electron beam plasma
GeFi Simulation 26
Analyze(5):eles phase condition(at the nonlineaer stage)
1 12.4 3.5i i
Electron beam plasma Electron beam plasma
GeFi Simulation 27
Analyze(6):Ions phase condition(at the linear stage)
1 10.268 1.36i i
Electron beam plasma Electron beam plasma
GeFi Simulation 28
Analyze(7):Ions phase condition(at the nonlineaer stage)
1 12.4 3.5i i
GeFi Simulation 29
Seed :strong electron beams(with thermal effects) (check the polarization) generates perturbed Ex(dominant)
acclerates ion (Vx) resonance with wave along x
ConclusionsConclusions
GeFi Simulation 30
A CUDA-version GeFi pic code was given. We studied the weak and strong electron beam plasma
with GeFi simulation, and compared with the cold beam theory. One typical nonlinear case with considering particles thermal effects and strong electron beam was analyzed briefly.
Basically, Our GeFi simulation shows that the thermal effects of the electron beam plays an important influence on the plasma instability.
To be continued…
GeFi Simulation 31
Thanks!
To be continuedTo be continued
GeFi Simulation 32
and V. S. Background effects Tp/Te effects Beta effects Anisotropic effects Saturation level Boundary effects To benchmark the weak-beam nonlinear behavior with the results of Kainer
et al. (S. Kainer, J. Dawson, R. Shanny and T. Coffey, Phys. Fluids 15 (1972) 493.).
Ion beam plasma Beam plasma in Harris current sheet.
B wave maxdeV