Implementing molecular dynamics on hybrid high performancecomputers—Three-body potentials
W. Michael Brown a,∗, Masako Yamada b
a National Center for Computational Sciences, Oak Ridge National Laboratory, Oak Ridge, TN, USAb GE Global Research, 1 Research Circle K1-3A17A, Niskayuna, NY, USA
a r t i c l e i n f o
Article history:Received 28 May 2013Received in revised form2 August 2013Accepted 5 August 2013Available online 13 August 2013
The use of coprocessors or accelerators such as graphics processing units (GPUs) has become popularin scientific computing applications due to their low cost, impressive floating-point capabilities, highmemory bandwidth, and low electrical power requirements. Hybrid high-performance computers,defined as machines with nodes containing more than one type of floating-point processor (e.g. CPU andGPU), are now becoming more prevalent due to these advantages. Although there has been extensiveresearch into methods to use accelerators efficiently to improve the performance of molecular dynamics(MD) codes employing pairwise potential energymodels, little is reported in the literature formodels thatincludemany-body effects. 3-body terms are required formany popular potentials such asMEAM, Tersoff,REBO, AIREBO, Stillinger–Weber, Bond-Order Potentials, and others. Because the per-atom simulationtimes are much higher for models incorporating 3-body terms, there is a clear need for efficientalgorithms usable on hybrid high performance computers. Here, we report a shared-memory force-decomposition for 3-body potentials that avoids memory conflicts to allow for a deterministic code withsubstantial performance improvements on hybridmachines.We describemodifications necessary for usein distributed memory MD codes and show results for the simulation of water with Stillinger–Weberon the hybrid Titan supercomputer. We compare performance of the 3-body model to the SPC/E watermodel when using accelerators. Finally, we demonstrate that our approach can attain a speedup of 5.1with acceleration on Titan for production simulations to study water droplet freezing on a surface.
Issues with power consumption, heat dissipation, and highmemory access latencies have made heterogeneous architecturesa popular idea for increasing parallelism with electrical powerand cost efficiency. Basic heterogeneous architectures include hy-brid systems that combine a traditional CPU with a coprocessoror accelerator such as a graphics processing unit (GPU), digitalsignal processor, field-programmable gate array, or other many-core chip. These architectures are becoming more popular inhigh-performance computers due to significant advantages in theperformance to electrical power ratio; for example, the upgradefrom the CPU-only Jaguar Cray XT5 at Oak Ridge National Labora-tory to the hybrid Titan CrayXK7 resulted in ten times the observedperformance while requiring only 19% more electrical power [1].Not only was Titan the fastest ranked supercomputer at the time
of writing, it was also ranked number 3 in terms of power effi-ciency [2,3].
In order to make effective use of hybrid machines, changes tothe models, algorithms, and/or code are typically required. For thelatter, changes are often required in order to (1) efficiently useshared memory parallelism, (2) increase concurrency with fine-grain parallelism, and (3) improve data locality, often with ex-plicit code to improve hierarchical memory use. There has beenextensive research along these lines to demonstrate significantperformance improvements formolecular dynamics onhybridma-chines [4,5]. Most of this work has been focused on pairwise po-tentials. Although these potentials are commonly employed in thesimulation of polymers and biomolecules, many materials such asmetals, covalent solids, and carbon nanotubes, as well as chemi-cal reactions, are typically simulated with potential energy mod-els that incorporate many-body effects. These potentials typicallyhave a much higher computational cost per atom when comparedto pairwise potentials. The simulation of materials with many-body potentials has been described in the context of the ‘‘Lawof Constancy of Pain’’ [6]—the trend in the development of newmany-body potentials has been to use increased CPU speeds and
core counts not for faster simulations, but to simulate with morecomplex models that have improved accuracy and transferability.
For these reasons, it is clearly desirable to develop algorithmsand code for simulation of many-body potentials on accelera-tors [6,7]. Despite their importance, very little has been reportedin the literature describingmethods or performance gains from ac-celeration ofmany-body potentials. In part, this could be due to theincreased complexity of these models—the models require multi-ple and/or nested loops that increase data dependencies, requirechanges to the standard neighbor list used in pairwise models, andcan require additional communications in parallel codes [6]. Im-plementations of the embedded atommethod (EAM) [8], for use onaccelerators and coprocessors have been described that led to sig-nificant performance improvements [9,10]. The EAM potential in-corporates the energy from embedding an atom into the electrondensity produced by its neighbors. In this sense, the EAM potentialis many-body because the electron charge density at each neigh-boring atom position must be calculated with a loop over sur-rounding atoms within some cutoff. However, this model issomewhat unique among many-body potentials in that it can stillbe computed using only pairwise summations. While this requiresadditional interprocess communications during the force compu-tation, we have shown that parallel implementations on hybridmachines can maintain significant performance improvements upto the entire 900 nodes available at the time of study [10].
For other many-body potentials, the data dependencies aremore complex. 3-body interactions are commonly used and re-quire terms calculated for every triplet of atoms in addition toevery pair. 3-body terms are required for many popular poten-tials such as MEAM [11], Tersoff [12], REBO [13], AIREBO [14],Stillinger–Weber [15], Bond-Order Potentials [16], and others. Al-though the nested loops required for 3-body terms are simple toimplement for serial calculations, their implementation for many-core accelerators/coprocessors results in some complications. Theproblems arise because non-uniform memory access and limitedper-core memory typically favor shared-memory atom- or force-decomposition for parallelism. The naïve implementations of thesedecompositions result in data dependencies; the evaluation of eachenergy term in the summation is used to update the force of threedifferent atoms. Therefore, naïve implementations require the useof atomic operations to prevent memory collisions—erroneousresults caused by simultaneous update of the same location inmemory from multiple threads. Atomic operations are generallyundesirable because of the high latencies and because they intro-duce randomness into the code. Many experienced developers willprefer deterministic code whenever possible to make debuggingfeasible on high performance computers that are constantly chang-ing out hardware and often changing the software stack.
Alternative implementations can alter the force computationloops to avoid data dependencies in exchange for increased com-putation. These implementations can reduce globalmemory accessand allow for deterministic code, but the potential performancegains become limited due to substantial increases in the amount offloating point operations required. Although an elegant approachfor implementing the Stillinger–Weber potential on GPUs has beendescribed with impressive performance [17], the approach is onlyapplicable for simulations of solid crystals where the neighbors ofany given atomdonot change. Therefore, the approach is not appli-cable tomanyproblems such as vacancy diffusion or the simulationof liquids.
In this paper, we present a simple approach for computing3-body interactions using an atom or force decomposition inshared memory. The approach avoids data dependencies allow-ing for a deterministic code. We present the changes necessaryfor implementation in parallel molecular dynamics codes using aspatial decomposition. We provide benchmark results on a hybrid
Cray XK7 supercomputer for a 3-body implementation building onour previous work in the LAMMPS molecular dynamics package[18,19,10]. We evaluate performance using the mW water model.ThemWwatermodel is comprised of a single effective particle thatpreferentially forms four tetrahedral bonds. The model has no ex-plicit charges, andhencenohydrogen-bonds or long-range electro-static terms, but it reproduces the quantitative behavior of wateras well as or better than conventional 3, 4 or 5 point charge mod-els. Simulation rates have been reported that are 180 times fasterthan the least expensive 3 point charge model (specifically SPC/E)while the quantitative agreement of the melting temperature, en-thalpy of melting, liquid–vacuum surface tension and liquid den-sity as a function of temperature have been shown to be superiorto the SPC, SPC/E, TIP3P, TIP4P and TIP5P models [20]. The orders-of-magnitude speedup relative to SPC/E has been attributed to:(a) a three-fold reduction in number of atoms, (b) the elimination ofexpensive k-space solvers and (c) the enabling of longer timestepsdue to the lack of internal bonds. In particular, the mW model hasbeen shown to facilitate the observation of spontaneous freezingin water [21] with much fewer timesteps relative to well-knowntraditional point-charge potentials [22,23] while still reproducingmany quantitative water properties of interest. This makes molec-ular dynamics amore attractive tool to probe phenomena that spanmany orders of magnitude of space and/or time, such as our par-ticular area of interest, which is the study of ice formation in thepresence of surfaces.
Here, we evaluate performance of the mW model with accel-eration compared to both the standard CPU implementation forStillinger–Weber in LAMMPS and simulation with the SPC/E watermodel. Our benchmark simulations include periodic water boxesand production simulations that are used to study the microscopicmechanism of droplet freezing on a surface. For the latter, simula-tion sizes of onemillionwatermolecules are used in order to probethe types of complex crystallization behaviors [24] we have ob-served experimentally for water droplets freezing on surfaces[25,26].
2. Methods
2.1. LAMMPS
Our implementation for 3-body potentials has been performedwithin the LAMMPS molecular dynamics package [27]. LAMMPS isparallelized via MPI, using spatial-decomposition techniques thatpartition the 3D simulation domain into a grid of smaller 3D sub-domains, one per MPI process. The algorithms we have previouslydeveloped for pairwise potentials and long-range electrostatics onaccelerators/coprocessors supporting CUDA or OpenCL in LAMMPShave been published in detail [18,19,10]. LAMMPS supports ac-celeration for short-range force calculation [18] with optionalacceleration for neighbor list builds and/or (P3M) long-rangeelectrostatics [19]. Neighbor list builds are performed on the accel-erator by first constructing a cell list that is utilized to build a Ver-let list using a radix sort to assert deterministic results. The van derWaals and short-range electrostatic forces are computed in a sepa-rate kernel. For each particle, the force-accumulation is performedby one or multiple threads. A default number of threads is chosenbased on the hardware and the potential model being used for cal-culation. The short-range calculation can be performed in single,mixed, or double precision. For mixed precision, all accumulationis performed in double precision and forces, torques, energies, andvirials are stored in double precision. For long-range electrostat-ics, acceleration for P3M is supported for charge assignment to themesh and force interpolation. The parallel FFT is performed on thehost (see below). The P3M calculation can be performed in singleor double precision.
All of the statistics computations, thermostats, barostats, timeintegration, bond/angle/dihedral/improper calculations, and anyother simulationmodifications are performed on the host. In orderto achieve efficient acceleration, these calculations must be par-allelized within each node on the host [18]. This is performed byusing multiple MPI processes, each sharing one or more acceler-ators on a compute node. This approach has several advantages.Those relevant to the work here include full compatibility with allof the other routines in LAMMPS that run on the CPU, the abil-ity to overlap data transfers and computation from different MPIprocesses sharing the accelerator, concurrent calculation of non-bonded forces on the accelerator and bonded forces on the host,and concurrent execution of long-range and short-range forces us-ing separate partitions of MPI processes [10]. The downside of theapproach is the requirement to determine an optimal number ofMPI processes to share the accelerator. This will not necessarily beall cores available on the node for smaller problem sizes due to theoverhead for sharing an accelerator on current hardware.
2.2. Accelerator model
For thiswork,we consider accelerators and coprocessors that fita model suited for OpenCL and CUDA. Because OpenCL and CUDAuse different terminology, we have listed equivalent (in the con-text of this paper) terms in Table 1. Here, we will use OpenCLterminology. The host consists of CPU cores and associated ad-dressable memory. The device is an accelerator consisting of 1 ormore compute units that typically correspond to processors ormul-tiprocessors in the hardware (note that for OpenCL this devicemight be the CPU). Each compute unit has multiple processing ele-ments that typically correspond to cores in the processor. The de-vice has global memory that may or may not be addressable by theCPU, but is shared among all compute units. Additionally, the de-vice has local memory for each compute unit that is shared by theprocessing elements on the compute unit. Each processing elementon the device executes instructions from awork-item (this conceptis similar to a thread running on a CPU core). We assume that thecompute unit might require SIMD instructions in hardware; there-fore, branches that could result in divergence of the execution pathfor different work-items are a concern. In this paper, the problemis referred to as work-item divergence. We also assume that globalmemory latencies can be orders of magnitude higher when com-pared to local memory access.
We assume that access latencies for coalesced memory will bemuch smaller. Coalescedmemory access refers to sequentialmem-ory access for data that is correctly aligned in memory. This willhappen, for example, when data needed by individual processingelements on a compute unit can be ‘‘coalesced’’ into a larger se-quential memory access given an appropriate byte alignment forthe data. Consider a case where each processing element needs toaccess one element in the first row of a matrix with arbitrary size.If the matrix is row-major in memory, the accelerator can poten-tially use coalescedmemory access; if the matrix is column-major,it cannot. The penalties for incorrect alignment or access of non-contiguous memory needed by processing elements will vary de-pending on the hardware.
A kernel is a routine compiled for execution on the device.The work for a kernel is decomposed into a specified number ofwork-groups each with a specified number of work-items. Eachwork-group executes on only one compute unit. The number ofwork-items in a work-group can exceed the number of physicalprocessing elements on the compute unit, allowing more work-items to share local memory and the potential to hide memory ac-cess latencies. The number of registers available per work-item islimited. A device is associated with one or more command queues.A command queue stores a set of kernel calls and/or host–devicememory transfers that can be executed asynchronously with hostcode.
Table 1Equivalent OpenCL and CUDA terminology.
OpenCL CUDA
Compute unit MultiprocessorProcessing element CoreLocal memory Shared memoryWork-item ThreadWork-group Thread blockCommand queue Stream
2.3. Geryon library
For our LAMMPS implementation, we have used the Geryonlibrary that provides a succinct API allowing a single code tocompile with both CUDA and OpenCL [18]. Currently, OpenCLlibraries or beta libraries are available for all major vendors forCPUs, GPUs, accelerators, and coprocessors. The Geryon library isavailable under the Free-BSD license from http://users.nccs.gov/∼wb8/geryon/index.htm.
2.4. Accelerating 3-body interactions
As the name implies, 3-body contributions, U3, to a potentialenergymodel are evaluated using triplets of atoms instead of pairs,
U3 =
i
j=i
k>j
φ(pi, pj, pk) rij < rc, rik < rc
0 otherwise(1)
for atom position p. and interatomic separation r... rc is set to en-force a spherical cutoff to allow for implementations with O(N)time complexity as opposed to O(N3). Pairwise potentials typi-cally only require half of the atoms within the cutoff to be storedin a neighbor list such that each pair of atoms is only evaluatedonce during the force loop. Evaluation of Eq. (1) however, requiresneighbor lists storing all atoms within the cutoff. The computa-tional time for simulations employing 3-body interactions is typi-cally almost entirely dominated by the force calculation loop. Fig. 1shows the profile for simulation of a 32,000 atom water box withthe 3-body Stillinger–Weber potential. In this case the force cal-culation is 90% of the entire simulation on a single node and 95%of this time is spent on the 3-body interaction. For some 3-bodypotentials it is substantially higher. This type of profile is an idealcase for porting to hybrid machines because of the high upperbound to performance gains from running a single routine (orsmall set of routines) on the accelerator. The observed performancegains, of course, will depend on how well the routines run on theaccelerator—how much fine-grain parallelism can be exposed fordoing as much computation with as little global memory access aspossible.
Although we do use a spatial decomposition for some routineson the device such as kernels for neighbor lists [18], the needfor efficient non-uniform memory access coupled with limitedper-core memory typically favors shared-memory atom- or force-decomposition for parallelism of the force computation loop. In anatom-decomposition, each work-item iterates through the forceloop for a single atom. In a force-decomposition, the terms inthe force loop are split between multiple work-items. In eithercase, the force is computed as the gradient of the potential energywith respect to atom position. For the naïve approach (NA), thismeans updating the force on three different atoms for each termcalculated for the summation in Eq. (1) in order to minimize thecomputation:
for (i=0; i<n; i++)num_nbors=get_neighbor_count(i);for (jj=0; jj<num_nbors; jj++) {
Fig. 1. Time for simulation of a 32000 molecule water box with mW on a singleCPU using 16 cores. The profile shows time for calculation of forces, energies,and virials (Force), neighbor-list builds (Neigh), MPI communications (Comm),screen/file (Output), and time integration, statistics and other calculations (Other).
The NA has several issues that arise when parallelizing forshared memory: (1) parallelization of the jj and kk loops underthe kk > jj condition results in more computation for work-itemswith lower rank, (2) multiple work-items can potentially updatethe same force location in memory and therefore atomic opera-tions are required, (3) because molecular dynamics codes typicallyemploymethods to sort atoms inmemory based on location (to im-prove data locality), the number ofmemory collisions encounteredduring atomic operations will increase resulting in much highermemory access times, and (4) the number of global memory up-dates required is high. Issue (1) can be addressed with transfor-mations of the 3D matrix of indices for force computations tobalance the amount of work between threads [28]. Issue (3) canbe addressed with a simple reindexing of the i atom based on thework-item rank [19]. Issues (2) and (4) aremore difficult, however.The force updates in the listing above can bemodified to use regis-ters such that there is only a single global memory access for eachiteration of each of the i, jj, and kk loops. In this case, the numberof forces that must be updated in global memory is n + n ·
b2n+bn2
for n atoms with bn neighbors per atom. The global memory accessproblem is made worse by the requirement to use atomic opera-tions to store the j and k atom forces. This also has the undesirableeffect of introducing randomness into the code.
Although the NA minimizes computation, memory access is amuch more common bottleneck for force calculations on accelera-tors andmany-core chips. Therefore, approaches that reducemem-ory access in trade for increased computational requirements canimprove performance. For pairwise potentials, this is commonlyaddressed by doubling the number of force computations to elim-inate the possibility of memory collisions. For 3-body interactions,a similar approach can be used; however, it requires more sub-stantial modifications. We refer to this approach as the redundant
computation approach (RCA). For the RCA, the force computationis parallelized such that the force on each atom in a given tripletcan be computed by different work-items, each performing someredundant computations:
In this case, an additional inner loop is added and the force com-putation is divided into two separate kernels. The first inner loop issimilar, and handles the case where the j and k atom in the tripletare bothwithin a distance less than rα , the sumof the potential cut-off and the neighbor list skin [18]. For the single triplet illustratedin Fig. 2 this loop updates the force when i equals atom 2. The loopis changed such that only the force for atom i is updated. Therefore,the force kernel can potentially be simplified to only include termsrelevant to atom i (this depends on themodel used).When i is atom1 or 3 in Fig. 2, the force must also be updated for the triplet, butin this case, the j and k atoms are not necessarily within the cut-off. This can be handled by either increasing the skin such that itis greater than the cutoff or by looping over neighbors of atom jrather than i in the inner loop. The latter approach of looping overneighbors of neighbors (used in the listing), will almost always bemore efficient. Again, a separate kernel with reduced computationcan potentially be used to only calculate terms for the triplet nec-essary for the i atom.
When implementing the RCA formultiple acceleratorswith dis-tributed memory, additional issues arise. For a spatial decompo-sition in MD, each process stores ghost atoms at the borders ofthe subdomain that can be within a distance rα of the local atomssimulated on the process (Fig. 2). For the NA, atoms within theborders have forces with contributions calculated on two differ-ent processes (because only a single neighbor list is used for eachtriplet). Therefore, communication is required at every timestepto accumulate forces for the ghost atoms. For the RCA, however,neighbors of neighbors are used in the force computation. There-fore, neighbor lists are required not only for local atoms, but alsofor ghost atoms when using the RCA. In order to avoid includingthe force contribution for a triplet more than once, the neighborlists for ghosts must either distinguish local and ghost neighborsor the length of the border must be doubled to allow the forces tobe calculated entirely on a single MPI process. The latter increasesthe amount of communications for the ghost-atom data exchange,but eliminates the need for communication in force accumulation.This is the approach that we use here.
Depending on the study, calculation of energy and virial termsmight be necessary on some or all timesteps. If only global energiesand pressures are needed, they can be calculated entirely in the
Fig. 2. Illustration of a triplet of atoms across the border of two MPI processes.Filled (red) circles denote atoms local to a process; circles without fill are ghostatoms. For the NA, a single neighbor list is used for every triplet and interprocesscommunication is required for force accumulation. For the RCA, the 3-bodyinteraction loop is calculated over neighbors of neighbors. This requires calculationof neighbor lists for ghost atoms and for the length of the border to be doubled to2 · rα . (For interpretation of the references to colour in this figure legend, the readeris referred to the web version of this article.)
threebody_center kernel in amanner similar to the NA. If per-atomenergies or stresses are required, the calculation must occur inboth 3-body kernels. In this case, the kernels will probably haveto perform the full 3-body calculation that is not simplified for theredundant computation. The RCA requires up to 3× the numberof calculations when compared to the NA, requires calculation ofneighbor lists for ghost atoms, and an increase in the numberof ghost atoms with a doubling of the length of the borders. Tooffset this increase, however, the number of force updates in globalmemory is reduced from n + n ·
b2n+bn2 to n and the requirement
for atomic operations is eliminated. Additionally, for the RCAdescribed here, the requirement for interprocess communicationfor force accumulation is eliminated.
For the RCA, the nested force computation loop must be paral-lelized across the work-items. For an atom decomposition, this isdone by parallelizing the i loop to assign an atom to eachwork-item, with the work-item performing the entire force ac-cumulation for that atom. This approach can lead to significantperformance gains, however, it is not ideal for GPU accelerators.The first issue is work-item divergence where cores are effectivelyidlewhen j atoms are notwithin the cutoff. The second issue is thatthis limits the number of work-items that can be used for a givennumber of atoms. Accelerators andmany-core chips require a largenumber of work-items for efficient performance when comparedto traditional CPUs and for many it is desirable to have many morework-items than cores in order to hide latencies. This is an impor-tant concern for HPC implementations where it is often desirableto scale up to large node counts, reducing the number of atoms perMPI process to decrease the time to solution. Although for largeatom counts a difference is not observed, we have shown that forcedecompositions can provide substantial improvements in parallelefficiency for pairwise models [19].
Force decompositions also divide the inner loops among work-items. This increases the number of work-items available to keepcores busy and also amortizes the access latencies. The tradeoff isan increase in the amount of computation required for additionalreductions and in some cases, the requirement for additionalwork-item synchronizations [19]. We tested many different combina-tions for parallelizing the jj and kk loops between an arbitrarynumber of work-items. Work-item divergence occurring at thecutoff check in the jj loop is a significant concern, because this di-vergence occurs for the duration of the inner kk loop. Therefore, wetested parallelization where the jj loop was split across a numberof work-items equal to the number of cores that are restricted toperform the same instruction. This eliminates divergence at the jj
Fig. 3. Illustration showing the atom and neighbor indices for the static regularforce decomposition.
loop cutoff check, but because this number is high for the Nvidiahardware used (32), the overhead for the reduction was too largeat typical neighbor list sizes. In our tests, we observed that theoptimal number of work-items for parallelization of both the jjand kk loops was determined by the neighbor list size. Based onthese results,we implemented a static regular force decomposition(SRFD) where a single parameter is used to determine the numberof work-items for the parallelization; the parameter is constant forall atoms throughout the duration of the simulation. Although thisworks well for Nvidia hardware, the advantages of the approachmight be vendor dependent. We have left this for a future study.
In the SRFD, a single work-item parameter, wn is specified. Theoptimal choice for wn depends on the neighbor list size and thehardware, however, defaults are chosen at runtime in LAMMPSbased on the typical neighbor list size and the detected hardware.The force accumulation for each atom is performed by w2
n work-items with wn work-items performing the innermost (kk) loopfor each neighbor, jj (Fig. 3). The neighbors are stored in memorysuch that wn neighbors for an atom are contiguous in memory andsuch that the neighbors for i and i + 1 are contiguous. In the casewhere wn = 1, the SRFD is equivalent to an atom decomposition.The condition that kk > jj for the innermost loop in the RCAlisting will lead to an imbalance in the amount of computationassigned to each work-item. This can potentially be addressed bylooping from 0 to jnum − 1 and only computing the force whenmod(jj + kk, 2) = 0 when jj > kk or mod(jj + kk, 1) = 0 whenjj < kk [27]. This approach can balance the amount of computationfor each work-item if implemented in a manner that does notexacerbate work-item divergence. The SRFD approach decreasesthe impact from work-item divergence in our tests and allows forsubstantially better performance at smaller atom counts.
2.5. Accelerating Stillinger–Weber
For this paper, we have chosen to evaluate acceleration of3-body interactions using the RCA with SRFD parallelism forthe Stillinger–Weber potential. The Stillinger–Weber model wasfirst introduced in 1985 to probe the behaviors of four-bondedmonoatomic systems such as carbon, silicon and germanium [15].More recently, it has been parameterized to represent watermolecules, which although multi-atomic, share many fundamen-tal properties with elemental silicon and germanium. The tunableparameters of mWwater have been optimized to yield tetrahedralordering in between that of carbon and silicon; this lead to compa-rable or better accuracy than the most popular point-charge mod-els at orders-of-magnitude faster wall-clock time [20].
where i indexes the center atom, θjik the angle between the atoms,and rij and rik give the interatomic separations. The other lettersdenote empirical parameters for the element types based on themodel. The Stillinger–Weber potential consists of a 2-body term,φ2, and a 3-body term, φ3. In order to provide a balanced compu-tational workload to work-items for the Stillinger–Weber model,three kernels are used. This includes a kernel for the 2-body term,implemented similar to other accelerated pairwise potentials inLAMMPS [18], and two kernels for the 3-body terms and describedabove for the RCA. Because there are no data dependencies forthese kernels, they can be calculated independently and concur-rently. The code can be compiled for concurrent force calculation(CFC) in which case the 2-body and threebody_center kernels arelaunched in a separate command queue from the threebody_endkernel in order to allow for concurrent execution. This is poten-tially advantageous for small atom counts in order to increase thenumber of work-items in flight at any given time. Because this re-quires an additional synchronization and reduction, CFC is optionaland for testing purposes.
A defaultwn parameter for the SRFD is chosen at runtime basedon the hardware or the user can optionally specify this parame-ter. The wn parameter determines the parallelization of the RCAas described above and also the number of work-items that per-form the force accumulation for the 2-body term as described pre-viously [19]. Neighbor list calculation is performed on the device asdescribed previously [18]with the exception that neighbor lists arealso calculated for ghost atoms. As discussed above, interprocesscommunication for ghost-atomdata exchange is removed from theaccelerated version in trade for a border length for ghost-atomdataexchange that is double that in the CPU-only version.
2.6. Benchmarks
We have benchmarked the strong and weak scaling perfor-mance of bulk water in a cubic box with periodic boundary condi-tions, comparing the performance of mWwater against that of oneof the least expensive point charge models, SPC/E [29]. We havealso benchmarked the performance of a production run comprisinga water droplet with one million mW molecules placed on a tun-able contact-angle surface,where thewalls and ceiling of the simu-lation box are reflecting (Fig. 4) [30]. For the droplet, wemodel onemillion molecules, which is orders of magnitude larger than typi-cal simulations, to minimize any finite-size effects that could leadto the erroneous interpretation of simulation results. In particu-lar, given our desire to probe spontaneous nucleation behaviors indroplets, our system size must be at least several times larger thanthe critical nucleus size. Moreover, the surface-area-to-bulk ratiomust be sufficiently small that surface effects not unduly dominate.
All of the simulations were conducted using the canonical en-semble where the relaxation time of the Nose–Hoover thermostatwas 1 ps. For mW, the timestep was 10 fs and for SPC/E, it was1 fs. A neighbor list skin of 1 Å was used. For mW, neighbor listbuilds were forced to be at least 2 timesteps apart, with check-ing every 2 timesteps for atom movement requiring a new build.For SPCE/E, the builds were forced to be at least 6 timesteps apart.The LAMMPS ‘‘grid numa’’ option was used to optimize MPI pro-cess mapping to reduce off-node communications [10]. A warmup run of 20 timesteps was used followed by a 400 timestep runused for the timings presented here. Single precision FFTs wereused for long-range electrostatics to reduce MPI communications.All benchmarks using the GPUs on Titan were performed withmixed precision as opposed to full double precision for the CPU-only runs [18].
2.7. Titan XK7 supercomputer
For the benchmark simulations performed here, we used theTitan supercomputer at Oak Ridge National Laboratory. Titan is a
Fig. 4. Close-up view of million molecule mW water droplet on tunable contact-angle surface. Lines indicate bonds among themWwatermolecules.White denotesaverage particle mobility, red denotes higher-than-average particle mobility andblue denotes lower-than-average particle mobility. Rendered with VMD [31]. (Forinterpretation of the references to colour in this figure legend, the reader is referredto the web version of this article.)
Cray XK7 computer with 18688 compute nodes, 512 XIO nodes,and a Gemini interconnect. Each node holds a single 16-core AMDOpteron 6274 running at 2.2 GHz with 32 GB ECC DDR3 SDRAM.The Opteron is connected to a Tesla K20X via PCI-e 2.0. TheK20X has 2688 compute cores running at 732 MHz with 6 GB ofGDDR5 SDRAM. The Gemini interconnect is connected in a 3-Dtorus topology and has 1–2µs latency for point-to-point messagesand 20 GB/s of injection bandwidth per node. At the time of thebenchmarks, Titan was running version 4.1u2 of the Cray LinuxEnvironment compiled with version 304.47.13 of the Nvidia CUDAdriver. The 2013 April 24th version of LAMMPS was used withthe modifications described below. The code was compiled usingversion 4.1.40 of the Cray GNU programming environment withgcc version 4.7.2 and nvcc version 5.0. The Nvidia proxy server wasused for runswithmore than oneMPI process sharing the device inorder to allow for context sharing and concurrent execution/datatransfer from different processes. Runs using less than 9 MPIprocesses per nodewere launchedwith the core affinity set so thatonly 1 core per AMD Bulldozer module was used.
3. Results
Fig. 1 shows the timing results for simulation of a 32,000molecule water boxwith LAMMPSwhen run on the CPU of a singleXK7 node. The simulation uses the Stillinger–Weber potentialwith the mW parameterization. In this case, 16 cores are usedand the entire simulation time is 11.4 s for 400 timesteps. Theforce calculation is 89.6% of the simulation time with 6.2% forneighbor list builds, 3.65% for MPI communications, and 0.55% fortime integration and other statistics calculations. Most of the forcecalculation time, 95%, is for calculation of 3-body interactions.
Fig. 5. Time for simulation of a 32000moleculewater boxwithmWona single XK7nodewith acceleration. PPN is the number ofMPI processes sharing the accelerator.wn = 1 is an atom decomposition and wn = 2 is a force decomposition with4 threads assigned to each atom. The profile shows time for calculation of forces,energies, and virials (Force), neighbor-list builds (Neigh), MPI communications ormemory copies (Comm), screen/file output (Output), time integration, statistics andother calculations (Other), host–device data transfer (Data Transfer), and time onthe host for casting and packing data for transfer (Cast/Pack). ‘‘Force + Neigh +
Data’’ combines ‘‘Force’’, ‘‘Neigh’’, ‘‘Data Transfer’’, and ‘‘Cast/Pack’’ when individualtimings are not available. Here, acceleration is for ‘‘Force’’ and ‘‘Neigh’’ calculations.
The results with acceleration using the RCA are shown in Fig. 5.For a single MPI process using an atom decomposition (wn =
1), the simulation time is reduced to 3.66 s with 1.47 s for theforce computation and 0.89 s for the neighbor list build. Most ofthe time for force computation, 96.8%, is used for calculation of3-body interactions. Although we are performing the neighbor listbuild on the accelerator, the time required is 26% slower thanneighbor list calculation on 16 cores using the CPU. In part this isdue to the increase in the number of atoms requiring neighbor listbuilds since lists are also required for ghost atoms. Performing theneighbor build on the accelerator reduces substantially the amountof host–device data transfer required, however, since the neighborlist storage typically dominates the memory usage for moleculardynamics. With neighbor list builds on the GPU, the host–devicedata transfer represents 4.1% of the total simulation time.
Using a force decomposition with wn = 2 reduces the time forforce calculation by 20%. Although there is additional code requiredfor storage of neighbors to allow for contiguous memory accesswith the SRFD, the neighbor list build time is not impacted. Thechoice of wn = 2 will depend on the device, the cutoff, and thenumber of particles on the accelerator. Devices with more coresand devices that benefit from oversubscribing the cores will favorlarger values of wn. Models with larger cutoffs see much moresubstantial benefits with higher values of wn. For the model anddevices used here, wn = 2 performed best at larger moleculecounts with a 12% reduction in force time with 256000 moleculesand a 40.6% with 8000molecules. For 4000 and 2000molecules onthe device, wn = 4 performed best with up to a 78.6% reduction inforce calculation time.
Running with a single MPI process using the device has theadvantage that there is no overhead from scheduling or handlingdriver requests frommultiple processes. Additionally, in the singlenode case in Fig. 5, MPI communications are replaced with mem-ory copies. (Because LAMMPS is intended as a parallel code, atomsacross periodic boundaries are treated as ghost atoms, even whenusing a single MPI task. Therefore, the packing of atom data at bor-ders for communication still occurs, although memory copies areused in place of the MPI calls). When using a single process how-ever, routines on the CPU, such as time integration, thermostats,
barostats, and other statistics, are performed in serial. In addi-tion to parallelizing these routines, runningmultipleMPI processessharing the accelerator has the advantage of allowing pipelining forhost–device data transfers and kernel execution. That is, the devicecan overlap host–device communications from one process withforce calculations from another. Due to these benefits, we see asignificant performance improvementwhen sharing the device be-tweenmultipleMPI processes. For the single node 32000moleculewater box simulation in Fig. 5, there is a 27.5% reduction in over-all simulation time when using 8 MPI processes. In this case, usingthe accelerators on the XK7 results in a speedup of 4.68. 17.8% ofthe simulation time is spent in various CPU calculations with an-other 9% required for interprocess communication. We note thatthere is a slight decrease in the time for MPI communications ver-sus the single process memory copies and data packing. Becausethe amount of data per MPI process for exchange of ghost-atomforces scales sublinearly for a fixed-size simulation, the commu-nication and data-packing time can be reduced when using multi-ple processes running on a single node (this, of course, will dependon the hardware and, in the case of multiple nodes, can result inslower times despite the reduction in data).
We also tested LAMMPS compiled to allow concurrent forcecalculation (CFC) in which case the device can choose to runmultiple force calculation kernels concurrently on the device.Although we did see some performance improvement for smallermolecule counts with a single MPI process, in most cases therewas a noticeable performance degradation from the additionalsynchronization and reduction overhead. The approach was neverfaster when using multiple MPI processes sharing the device (inwhich case kernels from different processes can run concurrently)and therefore we have not enabled this option in LAMMPS.
Results from parallel simulations using multiple GPUs areshown in Fig. 6. In the strong scaling tests, a fixed simulation sizeof 256000 molecules is benchmarked using between 1 and 128nodes. The strong scaling benchmarks test the ability to reducethe time to solution for a given simulation. For the mW modelon a single node, the simulation rate is 5.44 times faster whenusing acceleration. On 128 nodes, this speedup is reduced to 1.48with a simulation rate slightly over 0.5 µs per day. This reductionin relative performance on the GPU device is expected at lowermolecule counts; more work is required on each node in orderto effectively utilize the thousands of cores on each accelerator.In the weak scaling tests, the number of molecules per node isheld constant at 32000; this benchmarks the ability to run largersimulations on more nodes. In this case, the simulations withacceleration are 4.71 times faster on a single node and 3.52 timesfaster on 1024 nodes resulting in a parallel efficiency of 74.7%.
The mW model is intended to allow faster simulations ofwater with coarse-grain simulations that reduce the number ofparticles and allow for larger timesteps. Additionally, long-rangeelectrostatic terms are not included. In order to maintain accuracy,a 3-body potential is used. Therefore, we also compared perfor-mance to simulation with the SPC/E water model. For the strongscaling, GPU acceleration for the SPC/E model improves perfor-mance on a single node by 4.24× and by 1.89× on 128 nodes.Although the short-range force calculation in SPC/E is a smallerfraction of the simulation time, GPU acceleration still provides sig-nificant improvements because of efficient overlap of short-range,bond, and long-range forces with concurrent CPU and acceleratorcalculations [19]. Additionally, the number of particles is higher inthe SPC/E case, with 3 times the number atoms for the same num-ber of water molecules. For weak scaling, however, the parallel ef-ficiency is impacted by the effectively all-to-all communicationsfor the FFT-based Poisson solve. The speedup on 1 node is 4.18versus 1.86 on 1024 nodes. Although there are methods to reducethe impact of the long-range electrostatics solve (such as multiple
Fig. 6. Simulation rates for a fixed-size 256k molecule water box in parallel (left) and a scaled-size water box with 32k molecules per node (right). Parallel simulation rateswith ideal efficiency would have a slope equal to that of the green line in each plot. (For interpretation of the references to colour in this figure legend, the reader is referredto the web version of this article.)
Table 2Summary of best speedups versus running on a single XK7 CPU for CPU-only andaccelerated runs. Simulation is 400 timesteps for a 1 million molecule droplet. Thespeedups are calculated based on the single node loop time of 440.3 s.
Test case 1 node 64 nodesCores Speedup Cores Speedup
timestepping), the mW model offers speedups of over two ordersof magnitude due to the coarse-grain model employed and the ab-sence of long-range electrostatics.
Specifically, the simulation loop time on a single XK7 nodewith256k molecules is 4.8 times faster when using the mW modelinstead of SPC/E; on 128 nodes it is 3.8 times faster. However,because the coarse-grain mW model allows a timestep to be usedthat is 10 times larger, the actual simulation rates are 48 and 38times faster respectively. Because themWmodel does not considerlong-range electrostatics, the gains from use of the mW modelbecome more pronounced with larger simulations on more nodes.In the weak scaling tests, the simulation rate on a single node is40.5 times faster; on 1024 nodes it is 127.8 times faster.
For the final evaluation of mWperformance, we compared sim-ulation of awater droplet on a substratewith LAMMPS scripts usedfor production simulations. The simulations consist of a 1 millionmolecule water droplet interacting with a 9-3 Lennard-Jones wallfor the substrate. The results are summarized in Table 2. On a sin-gle XK7 node, we are able to achieve simulation rates that are 6.6times faster when using acceleration with the RCA. On 64 nodes,the simulations are 5.1 times faster. The relative performance inthis case is slightly better than for a bulk water box with a simi-lar molecule count. This is due to the fact that a perfectly balancedspatial decomposition to divide the molecules between processescannot be achieved in LAMMPS (which requires rectangular subdo-mains for dynamic load balancing) and also because the commu-nications topology for the simulations is different. For the latter,the simulation box is factored such that each process only has fourneighbors and there is no division ofwork in the dimension normalto the substrate.
4. Discussion
Potential energy models with 3-body interactions are essen-tial to many studies using molecular dynamics. The developmentof algorithms to run these simulations on hybrid machines withcoprocessors or accelerators is critical to achieving performancegains on current and future HPC systems. Because of the modelcomplexities and additional data dependencies, implementation of
3-body interactions requiresmore substantialmodifications to tra-ditional MD codes. We have shown, however, that a conceptuallysimple approach can be used to reduce dramatically global mem-ory access for these models while eliminating data dependenciesto allow for a deterministic code. Although this approach signifi-cantly increases the number of floating point operations requiredto perform force calculations, we have shown that substantial per-formance gains can still be realized because of the inherent par-allelism in the approach and the fact that the 3-body interactiontypically dominates the simulation time. This approach can be usedfor efficient acceleration of many important potentials includingMEAM, Tersoff, REBO, AIREBO, Stillinger–Weber, Bond-Order Po-tentials, and others.
Although porting existingmodels for hybridmachines is impor-tant, if the current trend in computing continues, it will be neces-sary to also consider the development of newmodels that allow forbetter data locality with computations that can better exploit mas-sive concurrency [32]. That is, because clock speeds are no longerincreasing and data movement is becoming the dominant bottle-neck, we cannot expect to continue to get increased performanceon future machines that addmore cores without reconsidering themodels that are used. Therefore, research into novel approachesfor simulation that trade increased computation for greater accu-racy and simulation rates are critical. For themWmodel used here,for example, researchers can achieve simulation rates that are or-ders of magnitude higher despite the additional complexity in themodel.
Our work for the acceleration of Stillinger–Weber can have animmediate impact onMD employing themWmodel. There is greatinterest in understanding the microscopic mechanism of dropletsfreezing on surfaces, and yet probing this behavior experimentallyis extremely challenging. Molecular dynamics is a tool that allowssuch molecular-level interactions to be probed, but the require-ment for large system sizes and in particular very long simula-tion times renders modeling cost-prohibitive in most simulationsof rare, activated processes. By accelerating the simulation time,the barrier to adoption is greatly reduced, opening up the possi-bility of using molecular dynamics as a valuable tool that comple-ments experimental findings.
Acknowledgments
This research was conducted in part under the auspices ofthe Office of Advanced Scientific Computing Research, Office ofScience, US Department of Energy under Contract No. DE-AC05-00OR22725 with UT-Battelle, LLC. This research was also con-ducted in part under the auspices of the GE Global Research HighPerformance Computing program. This research used resources of
the Leadership Computing Facility at Oak Ridge National Labora-tory, which is supported by the Office of Science of the US De-partment of Energy. Accordingly the US Government retains anon-exclusive, royalty-free license to publish or reproduce thepublished form of this contribution, or allow others to do so, forUS Government purposes. All of the code described in this paper isavailable in the open-source LAMMPS software package, availableat http://lammps.sandia.gov/ or by contacting the authors.
References
[1] B. Bland, High Performance Computing, Networking, Storage and Analysis(SCC), 2012 SC Companion, IEEE, 2012, pp. 2189–2211. http://dx.doi.org/10.1109/SC.Companion.2012.356.
[3] Green500, The Green 500. http://www.green500.org (accessed 06.05.13).[4] J.A. Baker, J.D. Hirst, Molecular Informatics 30 (2011) 498–504.[5] M. Harvey, G. De Fabritiis, Wiley Interdisciplinary Reviews: Computational
Communications 183 (2012) 449–459.[20] V. Molinero, E.B. Moore, The Journal of Physical Chemistry B 113 (2008)
4008–4016.[21] E.B. Moore, V. Molinero, Nature 479 (2011) 506–508.[22] M. Matsumoto, S. Saito, I. Ohmine, Nature 416 (2002) 409–413.[23] M. Yamada, S. Mossa, H.E. Stanley, F. Sciortino, Physical Review Letters 88
(2002) 195701.[24] D.W. Oxtoby, Journal of Physics: Condensed Matter 4 (1992) 7627.[25] A. Alizadeh, M. Yamada, R. Li, W. Shang, S. Otta, S. Zhong, L. Ge, A. Dhinojwala,
K.R. Conway, V. Bahadur, et al., Langmuir 28 (2012) 3180–3186.[26] A. Alizadeh, V. Bahadur, A. Kulkarni, M. Yamada, J.A. Ruud, MRS Bulletin 38
(2013) 407–411.[27] S. Plimpton, Journal of Computational Physics 117 (1995) 1–19.[28] J. Sumanth, D.R. Swanson, H. Jiang, Proceedings of the 21st Annual
International Conference on Supercomputing, ACM, 2007, pp. 105–115.http://dx.doi.org/10.1145/1274971.1274988.
[29] H. Berendsen, J. Grigera, T. Straatsma, The Journal of Physical Chemistry 91(1987) 6269–6271.
[30] M. Yamada, A. Alizadeh, B.J. Moore, 2013 submitted for publication.[31] W. Humphrey, A. Dalke, K. Schulten, Journal of Molecular Graphics 14 (1996)
33–38.[32] T.D. Nguyen, J.-M.Y. Carrillo, A.V. Dobrynin, W.M. Brown, Journal of Chemical
