Particle Tracking with Space Charge Effect using GPU
Yoshi Kurimoto J-PARC/KEK
J-PARC Main Ring Power Converter Group
1
Contents
โข Introduction about GPU
โข Single Particle Mechanics
โข Space Charge Effect
โข Application to J-PARC Main Ring
โข Current Issues
2
References
3
โข K. Ohmi et al., โStudy of Halo Formation in J-PARC MRโ, in Proc. PAC07, Albuquerque, NM, USA, Jun. 2007, paper THPAN040, pp. 3318-3320.
โข M. Ferrario et al., โSpace Charge Effectsโ 26 pages, contribution to the CAS - CERN Accelerator School: Advanced Accelerator Physics Course, Trondheim, Norway, 18-29 Aug 2013 10.5170/CERN-2014-009.331
โข Laurent S. Nadolski โSymplectic Integrators applied to Beam Dynamics in Circular Accelerators Basis for Constructing Integration Schemes for Hamiltonian Systemsโ NPAC 2010โ2011 (v1.4)
โข C.K. Birdsall, A.B Langdon, โPlasma Physics via Computer Simulationโ DOI: https://doi.org/10.1201/9781315275048โข R. De Maria, et al., โSixTrack Physics Manualโโข F. Christoph Iselin, โThe MAD Program Physical Methods Manual โ Version 8.13
โข S. Machida โ็ฉบ้้ป่ทๅนๆโ OHO seminar 2000โข K. Oide โใใผใ ๅๅญฆๅ ฅ้โ 1992
โข Private Communication to Ohmi-san (KEK) Hochi-san (JAEA)
This work is also supported by NVIDIA Corporation via NVIDIA GPU Grant Program
GPU (Graphic Processing Unit)
4
โฆGlobal Memory
Thread 1
โฆ
Shared
Memory
Block M
Thread 2
Thread N
โข Each operation can be assigned to each thread.
โข Execution of each thread can be parallelly done.
โข Threads in a common block can access the shared memory.
โข Shared memory is limited (12288 double words) but very fast
Thread 1
โฆ
Shared
Memory
Block 2
Thread 2
Thread N
Thread 1
โฆ
Shared
Memory
Block 1
Thread 2
Thread N
Example
5
๐=1
๐
๐๐๐ฅ๐ = ๐1๐ฅ1 + ๐2๐ฅ2 +โฏ+ ๐๐๐ฅ๐
1. Each term can be calculated in parallel
a1x1 a2x2 a3x3 a4x4 a5x5 a6x6 a7x7 a8x8
+ + + +
2. Summation also can be partially parallel.
a1x1 +a2x2 a3x3 +a4x4 a5x5 +a6x6 a7x7 +a8x8
+ +
No
t Para
llel
Parallel
Matrix multiplication, Fourier transformation โฆ
a1 x1ร a2 x2ร aN xNรโฆ
Parallel
CUDA (Compute Unified Device Architecture)
โข C/C++ based Programing Language
โข Compiler : โnvccโ (instead of gcc)
โข FFT Library : โcuFFTโ
โข Linear Algebra Library : โcublasโ
6
Hardware
7
Quadro P6000 TESLA V100
Single Precision (float) 12.63 TFLOP 14 TFLOP
Double Precision (double) 394.8 GFLOP 7 TFLOP
How I Got NVIDIA gave me Amazon Web Service 3-4$/hour
How I use Checking if my code works correctly
Massive calculations
NVIDIA Quadro P6000 TESLA V100
Overview of the Particle Tracking Simulation
8
Particle Generation Drift, Magnets, Cavity Space Charge Effect
Single Particle Mechanics : Maps between two locations can be calculated in parallel
PIC simulation : Maps requires spatial distributions of particles
Single Particle Mechanics
9
Implemented Components
โขBEND, DRIFT
10
โข Find the exact solutions of particle motions in uniform magnetic fields
โขQUAD, SEXT, thin multipole, Cavity
โข Using Hamiltonianโข Approximated by sequential symplectic transformations
Hamiltonian
11
๐ป ๐ฅ, ๐๐ฅ, ๐ฆ, ๐๐ฆ, ๐, ๐๐; ๐ = ๐๐ โ 1 + โ๐ฅ 1 + ๐ฟ 2 โ ๐๐ฅ2 โ ๐๐ฆ
2 โ ๐๐ด๐ (๐ฅ,๐ฆ)
๐0
๐๐ =๐ธโ๐ธ0
๐ฝ0๐0๐, ๐ = ๐ โ ๐ฝ0๐ก, ๐ฟ =
๐โ๐0
๐0โ ๐๐ โ
1
2๐พ02 ๐๐
2,
โ๐๐ฅ
2+๐๐ฆ2
2+
๐๐2
2๐พ02 โ โ๐ฅ โ โ๐ฅ๐๐ โ
๐๐ฅ2+๐๐ฆ
2
2๐๐ โ ๐
๐ด๐ (๐ฅ,๐ฆ)
๐0
Then just have to solve the Hamilton equations. But not always solvable
Symplectic Map
12
In case that a Hamilton equation is difficult to be solved,
๐๐๐ป0๐ฟ/2๐๐๐๐ฟ ๐๐๐ป0๐ฟ/2๐๐๐ป0๐๐ฟ๐โ๐๐๐๐ฟ ๐โ๐๐ป0๐๐ฟ ๐๐๐๐๐ฟ๐โ๐๐ป0๐๐ฟ
๐ =1
21 โ
1
3, ๐ =
1
2, ๐ =
1
3
Both symplectic integrators ( ๐๐๐ป0๐ and ๐๐๐๐ ) are integrable๐ป = ๐ป0 + ๐ :
or
The map between the entrance and exit of a components (Length = ๐ฟ ) :
๐๐ด๐ (๐ฅ) = โ1
2๐1(๐ฅ
2 โ ๐ฆ2)
๐ป โ๐๐ฅ
2+๐๐ฆ2
2+
1
2๐1 ๐ฅ2 โ ๐ฆ2 โ
๐๐ฅ2+๐๐ฆ
2
2๐๐ +
๐๐2
2๐พ02
h
๐ป0 ๐
โข ๐๐๐ป0๐ โข ๐๐๐๐
๐ฅ ๐ = ๐ฅ 0 cos ๐1๐ +๐๐ฅ(0)
๐1sin ๐1๐
๐๐ฅ ๐ = ๐๐ฅ 0 cos ๐1๐ โ ๐1๐ฅ(0) sin ๐1 ๐
๐ ๐ = ๐ 0
๐๐ ๐ = ๐๐ 0
๐ฅ ๐ = โ๐๐ฅ(0)๐๐ 0 ๐ + ๐ฅ(0)
๐ฆ ๐ = โ๐๐ฆ(0)๐๐ 0 ๐ + ๐ฆ(0)
๐ ๐ = ๐ 0 +๐๐ 0
๐พ02
โ๐๐ฅ 0 2 + ๐๐ฆ 0 2
2๐
๐๐ฅ ๐ = ๐๐ฅ 0
๐๐ฆ ๐ = ๐๐ฆ 0
๐๐ ๐ = ๐๐ 0
๐ฆ ๐ = ๐ฆ 0 cosh ๐1๐ +๐๐ฆ(0)
๐1sinh ๐1๐
๐๐ฆ ๐ = ๐๐ฆ 0 cosh ๐1๐ + ๐1๐ฆ(0) sinh ๐1 ๐
Ex1 Quad Magnets
13
Particle Motions of Uniform Magnetic Fields
14
๐ฟ
๐
Reference orbit๐ฅ
ว๐ง
( ๐ฅ ๐ก , ว๐ง ๐ก )
๐ฅ ๐ก =๐๐ฅ(0)
๐๐ตsin
๐๐ต
๐๐พ๐ก +
๐๐ง (0)
๐๐ตcos
๐๐ต
๐๐พ๐ก + ๐ฅ 0 โ
๐๐ง(0)
๐๐ต
ว๐ง ๐ก =๐๐ง(0)
๐๐ตsin
๐๐ต
๐๐พ๐ก โ
๐๐ฅ(0)
๐๐ตcos
๐๐ต
๐๐พ๐ก +
๐๐ฅ(0)
๐๐ต
1. Find ๐ก when ว๐ง ๐ก = tan๐ฟ
๐๐ฅ ๐ก
๐ตโจ
๐ฅ ๐ฟ = ๐ฅ2 ๐ก + ว๐ง2 ๐ก โ ๐
2. Convert canonical valuables
๐๐ฅ ๐ก = ๐๐ฅ(0)cos๐๐ต
๐๐พ๐ก โ ๐๐ง(0)sin
๐๐ต
๐๐พ๐ก
๐๐ง ๐ก = ๐๐ง(0)cos๐๐ต
๐๐พ๐ก + ๐๐ฅ(0)sin
๐๐ต
๐๐พ๐ก
๐๐ฅ ๐ฟ =๐๐ฅ ๐ก cos
๐ฟ๐ + ๐๐ง(๐ก) sin
๐ฟ๐
๐0
๐๐ง 0 = ๐0 1 + 2๐๐ + ๐ฝ2๐๐2 โ ๐๐ฅ 0 2 โ ๐๐ฆ(0)
2
๐๐ฅ 0 = ๐0๐๐ฅ 0
, where
Solution
Ex2 Bend
๐๐(๐ก)
๐๐ก= ๐ ิฆ๐ฝ๐ ร ๐ต
Equation of Motion
( ๐ฅ 0 , 0)
Space Charge Effect
15
โข Overview
โข Histogram Making (Charge weighting)
โข Poisson Solver
โข Electric field
Overview
16
(๐2
๐๐ฅ2+
๐2
๐๐ฆ2+
๐2
๐๐ง2)๐(๐ฅ, ๐ฆ, ๐ง) = โ
๐(๐ฅ, ๐ฆ, ๐ง)
ํ0๐(๐ง)
๐2
๐๐ฅ2+
๐2
๐๐ฆ2๐ข(๐ฅ, ๐ฆ) = โ
๐ ๐ง ๐(๐ฅ, ๐ฆ)
ํ02D approximation๐๐ฅ, ๐๐ฆ โช ๐๐ง
1. Make histogram ๐ ๐ฅ, ๐ฆ ๐๐๐ ๐ ๐ง
2. Solve 2D Poisson equation ๐ข ๐ฅ, ๐ฆ with boundary conditionsa
3. Calculate gradient (kick) โ๐
๐๐ฅ,๐
๐๐ฆ,๐
๐๐ง๐ด๐ ๐ง ๐ข(๐ฅ, ๐ฆ) ร ๐ฟ
The most time-consuming part
๐ด =๐
๐๐๐พ๐3๐ฝ๐2๐2
๐, ๐ฟ = ๐๐๐ฃ๐๐๐ ๐ท๐๐ ๐ก๐๐๐๐
a. Using โpolarโ or โrectangularโ boundary condition depending on the duct shape.b. This coefficient A is not always correct (come back later)
Space Charge Effect
17
โข Overview โข Histogram Making (Charge weighting)โข Poisson Solverโข Electric field
Cartesian Coordinate
๐ฅ๐ , ๐ฆ๐ ๐ฅ๐+1, ๐ฆ๐
๐ฅ๐+1, ๐ฆ๐+1๐ฅ๐ , ๐ฆ๐+1
๐ฅ, ๐ฆ
๐
๐ ๐ฅ๐ , ๐ฆ๐ += ๐(๐ฅ๐+1 โ ๐ฅ)(๐ฆ๐+1 โ ๐ฆ)
๐ฟ๐ฅ๐ฟ๐ฆ
๐ ๐ฅ๐+1, ๐ฆ๐ += ๐(โ๐ฅ๐ + ๐ฅ)(๐ฆ๐+1 โ ๐ฆ)
๐ฟ๐ฅ๐ฟ๐ฆ
๐ ๐ฅ๐ , ๐ฆ๐+1 += ๐(๐ฅ๐+1 โ ๐ฅ)(โ๐ฆ๐ + ๐ฆ)
๐ฟ๐ฅ๐ฟ๐ฆ
๐ ๐ฅ๐+1, ๐ฆ๐+1 += ๐(โ๐ฅ๐ + ๐ฅ)(โ๐ฆ๐ + ๐ฆ)
๐ฟ๐ฅ๐ฟ๐ฆ
Making Histograms
18
Sub-histograms using shared memory
19
โฆ
๐ฅ0, ๐ฆ0
๐ฅ1, ๐ฆ1
๐ฅ2, ๐ฆ2
๐ฅ(๐โ1)๐, ๐ฆ(๐โ1)๐
๐ฅ(๐โ1)๐+1, ๐ฆ(๐โ1)๐+1
๐ฅ(๐โ1)๐+2, ๐ฆ(๐โ1)๐+2
โฆ
1. Divide (macro) particles into N groups
๐ฅ๐โ1, ๐ฆ๐โ1
โฆ
๐ฅ๐๐โ1, ๐ฆ๐๐โ1
โฆ
๐ฅ๐, ๐ฆ๐
๐ฅ๐+1, ๐ฆ๐+1
๐ฅ๐+2, ๐ฆ๐+2
๐ฅ2๐โ1, ๐ฆ2๐โ1
GROUP1 GROUP2 GROUPN
Sub-histograms using shared memory
20
Thread
Thread
Threadโฆ
Shared Memory
Block 1๐ฅ0, ๐ฆ0
๐ฅ1, ๐ฆ1
๐ฅ2, ๐ฆ2
โฆ
Thread
Thread
Threadโฆ
Shared Memory
Block N๐ฅ(๐โ1)๐, ๐ฆ(๐โ1)๐
๐ฅ(๐โ1)๐+1, ๐ฆ(๐โ1)๐+1
๐ฅ(๐โ1)๐+2, ๐ฆ(๐โ1)๐+2
1. Divide (macro) particles into N groups 2. Assign each group to each block and each particle to each thread
Sub-histograms using shared memory
21
Thread
Thread
Threadโฆ
Sub-histogram 1
Shared Memory
Block 1
AtomicAdd
๐ฅ0, ๐ฆ0
๐ฅ1, ๐ฆ1
๐ฅ2, ๐ฆ2
โฆ
Thread
Thread
Threadโฆ
Sub-histogram N
Shared Memory
Block N
AtomicAdd
๐ฅ(๐โ1)๐, ๐ฆ(๐โ1)๐
๐ฅ(๐โ1)๐+1, ๐ฆ(๐โ1)๐+1
๐ฅ(๐โ1)๐+2, ๐ฆ(๐โ1)๐+2
Can be done in parallel
Note: Each thread in a common block adds entries in shared sub-histogram. But use โexclusive addโ operation (called AtomicAdd function)
1. Divide (macro) particles into N groups 2. Assign each group to each block and each particle to each thread3. Sub-histogram in each shared memory be made in parallel.
Sub-histograms using shared memory
22
Thread
Thread
Threadโฆ
Sub-histogram 1
Shared Memory
Block 1
AtomicAdd
๐ฅ0, ๐ฆ0
๐ฅ1, ๐ฆ1
๐ฅ2, ๐ฆ2
โฆ
Sub-histogram 1
Sub-histogram N+ +
โฆ
Global Memory
Total Histogram =
Thread
Thread
Threadโฆ
Sub-histogram N
Shared Memory
Block N
AtomicAdd
๐ฅ(๐โ1)๐, ๐ฆ(๐โ1)๐
๐ฅ(๐โ1)๐+1, ๐ฆ(๐โ1)๐+1
๐ฅ(๐โ1)๐+2, ๐ฆ(๐โ1)๐+2
1. Divide (macro) particles into N groups 2. Assign each group to each block and each particle to each thread3. Sub-histogram in each shared memory be made in parallel.4. Copy sub-histograms to the global memory and make summation
Space Charge Effect
23
โข Overview โข Histogram Making (Charge weighting)โข Poisson Solverโข Electric field
Poisson Solver (Rectangular Coordinates)
24
๐2๐ข
๐๐ฅ2+๐2๐ข
๐๐ฆ2= ๐(๐ฅ, ๐ฆ)
๐๐ฆ
๐๐ฅ=๐ฆ๐+1 โ ๐ฆ๐
โ๐ฅ
Boundary Condition : ๐ข ๐ฅ, 0 = ๐ข ๐ฅ, ๐ฟ๐ฅ = ๐ข ๐ฆ, 0 = ๐ข ๐ฆ, ๐ฟ๐ฆ = 0
2D Poisson Equation (Rectangular Coordinates)
Discretization with
๐ข๐โ1,๐ โ 2๐ข๐,๐ + ๐ข๐+1
โ๐ฅ2+๐ข๐,๐โ1 โ 2๐ข๐,๐ + ๐ข๐,๐+1
โ๐ฆ2= ๐๐,๐
Odd Extension for x
๐๐,๐ = (0, ๐ข1,๐, ๐ข2,๐ , โฏ , ๐ข๐,๐ , 0, โ๐ข๐,๐ , โ๐ข๐โ1,๐ , โฏ , โ๐ข1,๐)
๐, ๐ = 1,โฏโฏ ,๐
๐ = 0,โฏโฏ , 2(๐ + 1) โ 1
Poisson Solver (Rectangular Coordinates)
25
๐ท๐น๐๐ฅ(๐๐โ1,๐ โ 2๐๐,๐ + ๐๐+1,๐
โ๐ฅ2)๐ = ๐ข1,๐ + ๐
โ๐๐
2 ๐+12๐
๐ข2,๐ โ 2๐ข1,๐ + ๐โ๐
2๐2 ๐+1
2๐๐ข3,๐ + ๐ข1,๐ โ 2๐ข2,๐
= 0 + 1 โ2๐โ๐
๐2 ๐+1
2๐+๐
โ๐2๐
2 ๐+12๐
๐ข1,๐ + ๐โ๐
3๐2 ๐+1
2๐โ 2๐
โ๐2๐
2 ๐+12๐+ ๐
โ๐3๐
2 ๐+12๐
๐ข2,๐โฏ
= ๐๐
๐2 ๐+1
2๐+ ๐
โ๐๐
2 ๐+12๐โ 2 0 + ๐
โ๐๐
2 ๐+12๐๐ข1,๐ + ๐
โ๐2๐
2 ๐+12๐๐ข2,๐ +โฏ
= โ4๐ ๐๐2๐๐
2(๐ + 1)
๐=0
2 ๐+1 โ1
๐โ๐
๐๐2 ๐+1
2๐๐๐,๐
Odd Extension for y
๐๐,๐โฒ = (0, ๐๐,1, ๐๐,2, โฏ , ๐๐,๐, 0, โ๐๐,๐, โ๐๐,๐โ1, โฏ , โ๐๐,1)
๐โฒ = 0,โฏโฏ , 2(๐ + 1) โ 1
๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐๐โ1,๐โฒ โ 2๐๐,๐โฒ + ๐๐+1,๐โฒ
โ๐ฅ2)๐)๐ = โ
4
โ๐ฅ2๐ ๐๐2
๐๐
2(๐ + 1)
๐=0
2 ๐+1 โ1
๐โฒ=0
2 ๐+1 โ1
๐โ๐
๐๐2 ๐+1
2๐๐โ๐
๐๐โฒ
2 ๐+12๐๐๐,๐โฒ
๐น๐,2 = (0, ๐1,2, ๐2,2, โฏ , ๐๐,2, 0, โ๐๐,2, โ๐๐โ1,2, โฏ ,โ๐1,2)
๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐2๐ข
๐๐ฅ2+๐2๐ข
๐๐ฆ2)๐)๐ โ โ4
1
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐ + 1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐ + 1
๐=0
2 ๐+1 โ1
๐โฒ=0
2 ๐+1 โ1
๐โ๐
๐๐2 ๐+1
2๐๐โ๐
๐๐โฒ
2 ๐+12๐๐๐,๐โฒ
= โ41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐ + 1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐ + 1๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐)๐)๐
๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐ โ
๐=0
2 ๐+1 โ1
๐โฒ=0
2 ๐+1 โ1
๐โ๐
๐๐2 ๐+1
2๐๐โ๐
๐๐โฒ
2 ๐+12๐๐น๐,๐โฒ
Odd Extension of ๐๐,๐
๐น๐,0 = (0, 0,0,โฏ , 0, 0,0,0,โฏ , 0)
๐น๐,1 = (0, ๐1,1, ๐2,1, โฏ , ๐๐,1, 0, โ๐๐,1, โ๐๐โ1,1, โฏ ,โ๐1,1)
๐น๐,๐ = (0, ๐1,๐, ๐2,๐ ,โฏ , ๐๐,๐ , 0, โ๐๐,๐, โ๐๐โ1,๐ ,โฏ ,โ๐1,๐)
๐น๐,๐+1 = (0, 0,0,โฏ , 0, 0,0,0,โฏ , 0)
๐น๐,๐+2 = (0, โ๐1,๐, -๐2,๐ ,โฏ ,โ๐๐,๐ , 0, ๐๐,๐, ๐๐โ1,๐ ,โฏ , ๐1,๐)
โฎ
Poisson Solver (Rectangular Coordinates)
26
( FFT result for the left-hand side of the equation )
( FFT result for the right-hand side of the equation )
Poisson Solver (Rectangular Coordinates)
27
๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐2๐ข
๐๐ฅ2+๐2๐ข
๐๐ฆ2)๐)๐ = ๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
โ41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐ + 1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐ + 1๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐)๐)๐ = ๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐)๐)๐ = โ๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41โ๐ฅ2
๐ ๐๐2๐๐
2 ๐ + 1+
1โ๐ฆ2
๐ ๐๐2๐๐
2 ๐ + 1
๐๐,๐โฒ โ ๐ข๐,๐
๐ โ 0๐ โ 0 ๐๐
๐ = 0 ๐๐๐ ๐ = 0 ๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐)0)0 =
๐=0
2 ๐+1 โ1
๐โฒ=0
2 ๐+1 โ1
๐๐,๐โฒ = 0 (๐๐,๐โฒ is odd extension of ๐ข๐,๐ )
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ 4 FFTs using GPU
Poisson Solver (Rectangular Coordinates)
28
Summary
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
1. Odd extension of charge distributions
Poisson Solver (Rectangular Coordinates)
29
Summary
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
1. Odd extension of charge distributions
2. Apply 2D FFT to the extended distributions
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
Poisson Solver (Rectangular Coordinates)
30
Summary
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
1. Odd extension of charge distributions
2. Apply 2D FFT to the extended distributions
3. Divide the FFT results by โ41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
Poisson Solver (Rectangular Coordinates)
4. Apply 2D iFFT to 3
31
Summary
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
1. Odd extension of charge distributions
2. Apply 2D FFT to the extended distributions
3. Divide the FFT results by โ41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
Poisson Solver (Rectangular Coordinates)
4. Apply 2D iFFT to 3
32
Summary
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
1. Odd extension of charge distributions
2. Apply 2D FFT to the extended distributions
3. Divide the FFT results by โ41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
5. Extracted the solutions from the extended solutions.
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
๐๐,๐โฒ โ ๐ข๐,๐
๐๐,๐โฒ = โ๐๐ท๐น๐๐ฆ(๐๐ท๐น๐๐ฅ(๐ท๐น๐๐ฆ(๐ท๐น๐๐ฅ(๐น๐,๐)๐)๐
41
โ๐ฅ2๐ ๐๐2
๐๐
2 ๐+1+
1
โ๐ฆ2๐ ๐๐2
๐๐
2 ๐+1
)๐)๐โฒ
Space Charge Effect
33
โข Overview โข Histogram Making (Charge weighting)โข Poisson Solverโข Electric field
Rectangular Coordinate
Electric Field
34
Interpolate potential using Bezier Surface
๐ ๐ฅ, ๐ฆ =
๐=0
3
๐=0
3
๐(๐ฅ๐+๐โ1, ๐ฆ๐+๐โ1)3!
๐! (3 โ ๐)!(๐ฅ โ ๐ฅ๐๐ฟ๐ฅ
)๐(๐ฅ๐+1 โ ๐ฅ
๐ฟ๐ฅ)3โ๐
ร3!
๐! (3 โ ๐)!(๐ฆ โ ๐ฆ๐
๐ฟ๐ฆ)๐(
๐ฆ๐+1 โ ๐ฆ
๐ฟ๐ฆ)3โ๐
๐ฅ๐ , ๐ฆ๐ ๐ฅ๐+1, ๐ฆ๐
๐ฅ๐+1, ๐ฆ๐+1๐ฅ๐ , ๐ฆ๐+1
-1
๐ฅ๐โ1, ๐ฆ๐
๐ฅ๐โ1, ๐ฆ๐+1
๐ฅ๐+2, ๐ฆ๐
๐ฅ๐+2, ๐ฆ๐+1
๐ฅ๐ , ๐ฆ๐+2 ๐ฅ๐+1, ๐ฆ๐+2
๐ฅ๐ , ๐ฆ๐โ1 ๐ฅ๐+1, ๐ฆ๐โ1
๐ธ ๐ฅ, ๐ฆ = โ๐๐๐๐ ๐ ๐ฅ, ๐ฆ = โ(๐
๐๐ฅ,๐
๐๐ฆ)๐(๐ฅ, ๐ฆ)
Application to J-PARC Main Ring
35
J-PARC Main Ring (MR)
36
Circumference 1568 m
Injection Energy 3 GeV
Extraction Energy 30 GeV
Transverse Tune (21.35, 21.45)
Synchrotron Tune 0.002 โ 0.00015
Transition gt j32.5
RF frequency 1.67โ1.72 MHz
Harmonics 9
J-PARC MR
Extraction to NU
Injection from RCS
2.48 s
Specifications of J-PARC MR
Present Operation Cycle for NU
Intensity ~ 500 kW (8 bunches)Protons per bunch > 3x1013
Simulation of J-PARC Main Ring
37
2700 sec /10000 turns using NVIDIA TESLA V100
At least 10 times faster than CPU simulation that we usually use for J-PARC MR
โข (nx, ny) = (21.35, 21.45)โข 3x1013 ppbโข 200000 macro-particles โข Rf 1st 160 kV 2nd 110 kVโข Bunch factor @ Inj = 0.2โข 2s emittance @ Inj 16p for horizontal and verticalโข Aperture 65p for horizontal and verticalโข Number of Components๏ผ~1600 / turnโข Number of Locations for SC calculations๏ผ~1000 points /turn
Conditions
Preliminary Result
38
Beam Survival
Bunch Factor Iaverage/Ipeak
Simulation Measurement
Transverse Tunes
๏ฝ1 %
Operating Point๏ผ21.35,21.45๏ผ
Tune SpreadDn = 0.3-0.4
0.2 @ Injection ~0.3 after 1000 turns
Seems correctly working.
Current Issues
39
The calculations here were already implemented in my code. But they are not included in the results shown in the last slide. This is because they does not reproduce our measurements so far.
2D Approximation (๐๐ง โซ ๐๐ฅ,๐ฆ)
โข ๐๐๐ฃ๐ธ =๐(๐ฅ,๐ฆ)
๐0, ๐ธ = โ๐๐๐๐ ๐๐ธ(๐ฅ, ๐ฆ)
โ๐๐ธ ๐ฅ, ๐ฆ = โ๐ ๐ฅ, ๐ฆ
ํ0
Electric Field
Magnetic Field
โข ๐๐๐ก๐ต =๐(๐ฅ,๐ฆ)
๐0๐2 , ๐ต = ๐๐๐ก ิฆ๐ด ๐ฅ, ๐ฆ , ๐ ๐ฅ, ๐ฆ = ๐ฝ0๐๐(๐ฅ, ๐ฆ)
โ๐ด๐ง ๐ฅ, ๐ฆ = โ๐ ๐ฅ, ๐ฆ ๐ฝ0
ํ0๐โ โ๐๐ต ๐ฅ, ๐ฆ = โ
๐ ๐ฅ, ๐ฆ
ํ0(๐๐ต ๐ฅ, ๐ฆ =
๐
๐ฝ0๐ด๐ง(๐ฅ, ๐ฆ))
In free space, ๐๐ธ(๐ฅ, ๐ฆ) and ๐๐ต ๐ฅ, ๐ฆ are same. (satisfy same equation )
Origin of the coefficient ๐
๐๐๐พ๐3๐ฝ๐
2๐2(1)
40
(Using Coulomb Gauge)
Using ิฆ๐น ๐ฅ, ๐ฆ = ๐น๐ธ ๐ฅ, ๐ฆ + ๐น๐ต ๐ฅ, ๐ฆ =๐
๐๐ก๐ = ๐0๐ฝ0๐
๐
๐๐ แถ๐ฅ ๐ , แถ๐ฆ ๐ (Equation of Motion),
๐
๐๐ แถ๐ฅ ๐ , แถ๐ฆ ๐ = โ
๐
๐๐ฝ02๐พ0
3๐2
๐
๐๐ฅ,๐
๐๐ฆ๐(๐ฅ, ๐ฆ)
๐น๐ธ ๐ฅ, ๐ฆ = โ๐๐
๐๐ฅ,๐
๐๐ฆ๐๐ธ(๐ฅ, ๐ฆ)
๐น๐ต ๐ฅ, ๐ฆ = ๐ 0,0, ๐ฝ0๐ ร๐
๐๐ฅ,๐
๐๐ฆ,๐
๐๐งร 0,0, ๐ด๐ง ๐ฅ, ๐ฆ = โ๐
๐
๐๐ฅ,๐
๐๐ฆ(โ๐ฝ0
2๐๐ต ๐ฅ, ๐ฆ )
This is correct only when the beam is in infinite free space. ๐๐ธ ๐ฅ, ๐ฆ = ๐๐ต ๐ฅ, ๐ฆ
In general, however, electric and magnetic fields satisfy different boundary conditions
The forces on a test particle by electric and magnetic fields are written as
๐๐ฌ ๐, ๐ โ ๐๐ฉ ๐, ๐
Origin of the coefficient ๐
๐๐๐พ๐3๐ฝ๐
2๐2(2)
41
Boundary Conditions with Beam Ducts
42
Electric Field Magnetic Field
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const.
๐/2
๐/2
Skin depth ๐ฟ(frequency of the beam) << Duct depth ๐ค
Same potential at duct surfaces
Yes (fast)
๐ต๐ฆ ๐ฅ, ยฑ๐
2= โ
๐
๐๐ฅ๐๐ต(๐ฅ, ยฑ
๐
2) = 0
๐ต๐ฅ ยฑ๐
2, ๐ฆ =
๐
๐๐ฆ๐๐ต(ยฑ
๐
2, ๐ฆ) = 0
B is tangential to duct surfaces
๐ค
Boundary Conditions with Beam Ducts
43
Electric Field Magnetic Field
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const.
๐/2
๐/2
Skin depth ๐ฟ(frequency of the beam) << Duct depth ๐ค
Same potential at duct surfaces
Yes (fast)
๐ต๐ฆ ๐ฅ, ยฑ๐
2= โ
๐
๐๐ฅ๐๐ต(๐ฅ, ยฑ
๐
2) = 0
๐ต๐ฅ ยฑ๐
2, ๐ฆ =
๐
๐๐ฆ๐๐ต(ยฑ
๐
2, ๐ฆ) = 0
B is tangential to duct surfaces
๐๐ต ยฑ๐/2, ๐ฆ = ๐๐ต ๐ฅ, ยฑ๐/2 = Const.๐๐ฉand ๐๐ฌ can be commonly calculated
๐ค
Boundary Conditions with Beam Ducts
44
Electric Field Magnetic Field
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const.
๐/2
๐/2
Skin depth ๐ฟ(frequency of the beam) << Duct depth ๐ค
๐๐ต ยฑโ = 0
Same as free space
Same potential at duct surfaces
No (slow)Yes (fast)
๐ต๐ฆ ๐ฅ, ยฑ๐
2= โ
๐
๐๐ฅ๐๐ต(๐ฅ, ยฑ
๐
2) = 0
๐ต๐ฅ ยฑ๐
2, ๐ฆ =
๐
๐๐ฆ๐๐ต(ยฑ
๐
2, ๐ฆ) = 0
B is tangential to duct surfaces
๐๐ต ยฑ๐/2, ๐ฆ = ๐๐ต ๐ฅ, ยฑ๐/2 = Const.๐๐ฉand ๐๐ฌ can be commonly calculated
๐ค
Boundary Conditions with Beam Ducts
45
Electric Field Magnetic Field
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const.
๐/2
๐/2
Skin depth ๐ฟ(frequency of the beam) << Duct depth ๐ค
๐๐ต ยฑโ = 0
Same as free space
Same potential at duct surfaces
No (slow)Yes (fast)
๐ต๐ฆ ๐ฅ, ยฑ๐
2= โ
๐
๐๐ฅ๐๐ต(๐ฅ, ยฑ
๐
2) = 0
๐ต๐ฅ ยฑ๐
2, ๐ฆ =
๐
๐๐ฆ๐๐ต(ยฑ
๐
2, ๐ฆ) = 0
B is tangential to duct surfaces
๐๐ต ยฑ๐/2, ๐ฆ = ๐๐ต ๐ฅ, ยฑ๐/2 = Const.
๐๐ฉand ๐๐ฌ must be separately calculated
๐๐ฉand ๐๐ฌ can be commonly calculated
๐ค
Free Space 2D Poisson Equation
46
๐(๐ฅ, ๐ฆ) =1
2๐เถฑ๐(๐ฅโฒ, ๐ฆโฒ) ln
1
(๐ฅ โ ๐ฅโฒ)2+(๐ฆ โ ๐ฆโฒ)2๐๐ฅโฒ๐๐ฆโฒ
1. Find potential at the boundary using the Green function*
2. Modify the differential equation
Obtained by the Green function
โ2๐ข1,๐ + ๐ข2,๐
โ๐ฅ2+๐ข1,๐โ1 โ 2๐ข1,๐ + ๐ข1,๐+1
โ๐ฆ2= ๐1,๐ โ
๐ข0,๐
โ๐ฅ2
Modifying charge distribution near the boundary
* Calculating whole region using the Green function costs a lot since it needs O(n2) operations
๐ข0,๐ โ 2๐ข1,๐ + ๐ข2,๐
โ๐ฅ2+๐ข1,๐โ1 โ 2๐ข1,๐ + ๐ข1,๐+1
โ๐ฆ2= ๐1,๐
In case of J-PARC Main Ring (1)
47
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const. ๐๐ต ยฑ๐/2, ๐ฆ = ๐๐ต ๐ฅ, ยฑ๐/2 = Const.1. Same Boundary Condition
โข Comparison with measurement of coherent tune shift
BM duct height 100mm(real value)
BM duct height 60mm(good agreement with measurement )
Bending Magnet Duct
131 mm
100 mm
60 mm
Simulation Results
Need to make duct height unreasonably smaller
๐
In case of J-PARC Main Ring (2)
48
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const.2. ๐๐ต ยฑโ = 0
๐๐ธ ๐๐ต ๐๐ธ โ ๐ฝ02๐๐ต
Potential Distribution in a BM Duct
Incoherent tune shifts to vertical direction are so large and cause huge beam loss within a few turnsโฆ
Need to consider other physics
One Possibility Boundary with Magnetic Material
49
โข ๐ต at the surface of ferromagnetic material must be perpendicular to the surface ( ๐ฟ โช ๐ค)
โข The cores of the BMs touch with all horizontal surface of the duct. (J-PARC MR)
โข Use image current density so that the tangential components of ๐ต can be eliminated
๐๐๐๐๐๐(๐ฅ, ๐ฆ) =[๐ ๐ฅ, โ1 ๐๐ฆ + ๐๐ ]
๐(๐ฅ, ๐ฆ) =1
2๐เถฑ(๐ ๐ฅโฒ, ๐ฆโฒ + ๐๐๐๐๐๐ ๐ฅโฒ, ๐ฆโฒ )ln
1
(๐ฅ โ ๐ฅโฒ)2+(๐ฆ โ ๐ฆโฒ)2๐๐ฅโฒ๐๐ฆโฒ
1. Find potential including image current at the boundary using the Green function*
2. Modify the differential equation (same as for free space )
Finding Potential with Image Current
50
๐๐๐๐๐๐(๐ฅ, ๐ฆ) =[๐ ๐ฅ, โ1 ๐๐ฆ + ๐๐ ]
In case of J-PARC Main Ring (3)
51
๐๐ธ ยฑ๐/2, ๐ฆ = ๐๐ธ ๐ฅ, ยฑ๐/2 = Const.2. ๐ต๐ฅ ๐ฅ, ยฑ๐/2 = ๐ต๐ฆ ยฑ๐/2, ๐ฆ = 0
๐๐ธ ๐๐ต ๐๐ธ โ ๐ฝ02๐๐ต
Potential Distribution in a BM Duct
Incoherent tune shifts to vertical direction are still so large and cause huge beam loss within a few turnsโฆ
The potential in the BM duct can not be simply modeled. May need combinations of phenomena shown here ??
Summary
โข Particle tracking simulation code with space charge effect has been developed for GPU usage.
โข Not only single particle mechanics but also space charge calculation can be accelerated using FFT.
โข For the simulation of J-PARC MR, the developed code with GPU is at least 10 times faster than our official(?) code with CPU.
โข Currently working on the calculation of SC in thin rectangular ducts.
52
Backup
53
Canonical Valuables
54
๐ฅ : horizontal coordinate of a plane perpendicular to the reference orbit
๐ฟ =๐โ๐0
๐0โ ๐๐ โ
1
2๐พ02 ๐๐
2,
๐๐ฅ : momentum of ๐ฅ direction normalized by the reference momentum ๐0
๐ฆ : vertical coordinate of a plane perpendicular to the reference orbit
๐๐ =๐ธโ๐ธ0
๐ฝ0๐0๐: canonical conjugate of ๐
๐๐ฆ : momentum of ๐ฆ direction normalized by the reference momentum ๐0
๐ = ๐ โ ๐ฝ0๐ก
Implemented Components
Ex2 Acceleration
โข Cavity
55
๐๐ ๐น : Amplitude of RF voltage, ๐๐ ๐น : RF frequency, ๐ : n-th order harmonics, ๐๐ : Synchrotron phase
๐๐ฅ(๐ ) = ๐๐ฅ 0 ๐๐ฆ(๐ ) = ๐๐ฆ 0 ๐๐ ๐ = ๐๐ 0 +๐๐๐ ๐น
๐๐พ๐ฝ๐2(sin
2๐๐๐ ๐น
๐ฝ๐๐ 0 + ๐๐ โ sin๐๐ )
๐ฆ ๐ = ๐ฆ 0 ๐ ๐ = ๐ 0๐ฅ ๐ = ๐ฅ 0
โข Adiabatic Damping (turn by turn)
๐๐ฅ,๐+1 = ๐๐ฅ,๐(1 โโ๐ธ
๐๐พ๐๐ฝ๐2)
๐๐ฆ,๐+1 = ๐๐ฆ,๐(1 โโ๐ธ
๐๐พ๐๐ฝ๐2)
๐๐,๐+1 = ๐๐,๐(1 โโ๐ธ(1 +
1๐พ๐
2)
๐๐พ๐๐ฝ๐2 )