BeatBox — HPC Simulation Environment for Biophysicallyand Anatomically Realistic Cardiac Electrophysiology

Mario AntoniolettiEPCC

Edinburgh, UKm.antonioletti@epcc.ac.uk

Vadim N. Biktashev∗

University of ExeterHarrison Building, North Park

RoadExeter EX4 4QF, UK

v.n.biktashev@exeter.ac.uk

Adrian JacksonEPCC

Edinburgh, UKa.jackson@epcc.ac.uk

Sanjay R. KharcheHarrison Building, North Park

RoadExeter EX4 4QF, UK

s.r.kharche@exeter.ac.uk

Tomas StaryUniversity of Exeter

Harrison Building, North ParkRoad

Exeter EX4 4QF, UKt.stary@exeter.ac.uk

Irina V. Biktasheva†

University of LiverpoolAshton building, Ashton street

Liverpool L69 3BX, UKivb@liverpool.ac.uk

ABSTRACTBeatBox combines flexible script language user interface withrobust solvers to set up cardiac electrophysiology in-silicoexperiments without re-compilation. It is free C softwarerun on Unix platforms sequentially or in MPI, for mul-tiscale modelling from single cell to whole heart simula-tions with run-time local/global measurements, cardiac re-entry tip/filament tracing, etc. Extensible solvers, cell andanatomy repositories provide open framework for new de-velopments in the field.

KeywordsCardiac simulations; realistic anatomy; reaction-diffusionsystem; finite difference solver; irregular domain; MPI par-allelisation.

1. BACKGROUNDCardiovascular disease (CVD) is the main cause of death

in Europe, accounting for 47% of all deaths [30]. Cardiacarrhythmias, where the electrical activity of the heart re-sponsible for its pumping action is disturbed, are amongthe most serious CVDs. Despite over a century of study,the circumstances from which such fatal cardiac arrhyth-mias arise are still poorly understood. Although several ad-vancements have been made in linking genetic mutations to

∗Also with EPSRC Centre for Predictive Modelling inHealthcare, University of Exeter, Exeter, EX4 4QJ, UK†Corresponding author

As Submitted To:EuroMPI 2016 September 25–28, 2016, Edinburgh, UK

ACM ISBN .

DOI:

arrhythmogenic CVD [13, 32, 41], these do not explain theresultant mechanisms by which arrhythmia and fibrillationemerge and sustain at the whole heart level, for the positionof the heart in torso makes in vivo measurement awkwardand invasive, prohibitively so for study in humans. Thus, forsome genetic cardiac diseases, the first presenting symptomis death with understandably limited opportunity to makeeven superficial examinations in vivo. The most modern ex-perimental methods do not provide sufficient temporal andspacial resolution to trace down the multi-scale fine detailsof fibrillation development in samples of cardiac tissue, notto mention the heart in vivo.

Combination of mathematical modelling [8, 4, 5, 9, 11] andthe latest realistic computer simulations of electrical activityin the heart have much advanced our understanding of heartfibrillation and sudden cardiac death[10, 6, 22, 23], and theimpact of in-silico modelling, or indeed in-silico ”testing”,is expected to increase significantly as we approach the ul-timate goal of the whole-heart modelling. With the vastamount of quantitative experimental data on cardiac my-ocytes action potential and the underlying transmembraneionic currents ready for inclusion into the in-silico modeling,and the recent advance in high-resolution DT-MRI provisionof detail anatomy models, the biophysically and anatomi-cally realistic computer simulations allow unimpeded accessto the whole heart with greater spatial and temporal reso-lutions than in a wet experiment, and allow to synthesisesuch elusive phenomena for closer study, hence improvingprospects of their treatment and prevention.

The biophysically and anatomically realistic simulation ofcardiac action potential propagation through the heart iscomputationally expensive due to the huge number of equa-tions per cell and the vast spacial and temporal scales re-quired. Complexity of realistic cardiac simulations spansmulti-physical scales to include greater detail at cellularlevel, tissue heterogeneity, complex geometry and anisotropyof the heart. Due to huge number of strongly nonlinear equa-tions to be solved on the vast temporal and spatial scalesdetermined by the high-resolution DT-MRI anatomy mod-

arX

iv:1

605.

0601

5v1

[cs

.CE

] 1

9 M

ay 2

016

els, its timely running relies on use of parallel processors -High Performance Computing (HPC).

To address the intrinsically modular cardiac electrophys-iology in silico modelling, we developed modular softwarepackage BeatBox [27], with a built-in simulation script in-terpreter, extendable repositories of cell and tissue/anatomymodels, capable to run both sequentially and in parallel ondistributed (MPI) memory architecture, fig.1. The Beat-box cardiac simulation envoronment allows setup of compli-cated numerical experiments without re-coding at low-level,so that cell excitation, tissue and anatomy models, stimula-tion protocols may be included into a script, and BeatBoxsimulation run either sequentially or in parallel without re-compilation. Importantly, the BeatBox modular paradigmprovides an open framework for new developments in thefield, for the open source BeatBox solvers, and cell and tis-sue/anatomy repositories are extended via robust and flex-ible interfaces. This paper gives an overview of the cur-rent state of BeatBox cardiac electrophysiology simulator to-gether with description of the main computational methodsand MPI parallelisation approaches.

2. CARDIAC TISSUE MODELSComputer simulation of cardiac muscle requires a mathe-

matical model, describing the relevant biophysical and elec-trophysiological processes. The bidomain model considersintracellular and extracellular spaces in the syncytium ofcardiac myocites. Those two domains are separated fromeach other by cellular membranes, the conductivity throughwhich is controlled by ionic channels. This situation is de-scribed by a system of partial (PDE) and ordinary (ODE)differential equations of the form:

Cm∂V

∂t= −Iion(V ,g) +

1

χ∇ · σi∇Φi,

∇ · (σi + σe)∇Φi = ∇ · σe∇V − Iext

∂g

∂t= f(g, V , ~r), (1)

where V is the transmembrane voltage, Φi is the intracellu-lar electrostatic potential (so Φe = Φi−V is the extracellularpotential), σi and σe are the anisotropic conductance ten-sors of the intra- and extracellular domains respectively, Cmis the specific capacitance of the membrane and χ is theaverage surface to volume ratio of the cells. The transmem-brane ionic currents Iion are controlled by gating variablesand ionic concentrations, represented by the vector g. Thekinetic rates are expressed in terms of the vector-functionf . The term Iext designates the external elecric current, sayfrom experimental or defibrillation electrodes. In the system(1), the first equation is parabolic, the second is elliptic andthe third effectively is a system of ODEs at every point ofthe tissue charaterised by its location ~r. If the intracellularconductances are proportional, i.e. σe = νσi for a scalar ν,then Φi, Φe and V are proportional to each other, and thesystem (1) simplifies to a monodomain model:

Cm∂V

∂t= −Iion(V ,g) +

1

χ∇ · σeff∇V − Ieff(~r, t),

∂g

∂t= f(g, V , ~r), (2)

where σeff = ν1+ν

σi, Ieff = 1χ(1+ν)

Iext. System (2) belongs to

the class of reaction-diffusion systems, used for modelling of

a large variety of natural and artificial nonlinear dissipativesystems [31].

Computationally, the bidomain description is dramati-cally more challenging than the monodomain, as the el-liptic equation has to be solved at every time step. Prac-tice shows that unless an external electric field is involved,the bidomain models give results that differ only slightlyfrom corresponding appropriately chosen monodomain mod-els, which, together with the fact that experimental data onthe intra- and extracellular conductivity tensors are scarse,means that in practice the monodomain simulations are usedmore widely.

The complexity of cardiac electrophysiology simulations isfurther increased as these expand to span multiple physicalscales to include greater detail at the cellular level, such ascell metabolism, and greater integration with the surround-ing biological systems, such as vascular fluid dynamics. Inthis context, it is not surprising that the timely completionof simulations relies on modern high performance computinghardware.

Use of HPC facilities, although essential, is severely lim-ited by specialized software development skills required, so aseparation of the low-level coding from the processes of for-mulating and solving research problems is highly desirable.The BeatBox project seeks to overcome these difficulties byproviding a computational environment that could serve as aunifying paradigm for all in silico cardiac electrophysiologyresearch, and for research in similar phenomena involvingreaction-diffusion systems outside the cardiology domain.

3. LOGICAL STRUCTURE AND USER IN-TERFACE

Figure 1: BeatBox formalism paradigm [27].

The fundamental paradigm used by BeatBox is to rep-resent a simulation as a ring of “devices”, i.e. individualmodules that perform specific computational, input/outputor control tasks. Module of each type can be used morethan once in the ring, thus providing more than one deviceinstance. This ring of devices, fig. 2, is constructed at start-up, based on the instructions given in an input script. Thescript describes the sequence of devices used in a particu-lar simulation and their parameters, using a domain-specfic

Figure 2: “Ring of devices” set up by sample.bbs

script (see fig. 11 in the Appendix).

scripting language with a flexible syntax that includes thingslike a built-in interpreter of arithmetic expression, recursivecalls to other scripts, etc. This allows complicated nu-merical experiments to be set-up without low-levelre-coding.

With some simplification, BeatBox computable data are oftwo kinds: the main bulk of the data is in a four-dimensionalcomputational grid, of which three dimensions correspond tothe spatial dimensions and the fourth dimension enumerateslayers, which are allocated, e.g. for the components of thereaction-diffusion system, for the output data produced bysome of the devices (e.g. those computing the values of thediffusion term), or as a scratch memory area for devices thatrequire this. When working with a complicated geometry,only a subset of the 3D grid corresponding to the tissuepoints is involved. Every device typically works only withsome layers of the grid. Apart from the computational grid,there are global variables, the values of which may be used bysome devices, and modified by others. An important featureis that any device instance in the ring is associated with aglobal variable that serves as the flag indicating whether thisdevice instance should be executed at this particular step inthe computation.

A simple example of a BeatBox script, used for simula-tion of the feedback-controlled resonant drift algorithm asdescribed in [7], is provided in the Appendix, with the cor-responding ring of the devices illustrated in fig. 2. The scriptinvolves the emulation of an electrode registering electricalactivity of the cardiac muscle at a point, and of an exter-nal device which switches on a time-delay line when a signalfrom the registering electrode satisfies a certain condition,and issues a low-voltage defibrillating shock upon the ex-piration of the time delay. We stress here that the imple-mentation of this protocol in the majority of other cardiacsimulation software would require modifications of the codeat a low level.

4. AN OVERVIEW OF COMPUTATIONALALGORITHMS

We believe that the main features of BeatBox are the flex-ibility of its user interface, and the fact that any new com-putational features can always be added as a new device.At the same time, it is clear that its utility at present de-pends on specific computational capabilities. In the nextsections, we describe the components that are most likely

to be required for a typical cardiac dynamics simulations,in the state they are currently available in at the time ofwriting this paper.

4.1 Splitting the problem into parts

4.1.1 Computation of intermediate expressions“Divide and conquer” is a popular and successful strategy

for evolution-type problems. The idea is to split the right-hand sides of complex evolution equations to simpler com-ponents, implement solvers corresponding to each of thesecomponents, and then coordinate the work of the solvers sotogether they solve the whole problem. The modular struc-ture of a BeatBox job makes this approach particularly easyto implement. One way of doing so is by computing differentparts of the right-hand side by different devices, and thenallow the time-stepping device to use the results of thesecomputations. We illustrate this by using a simple example,with reference to the sample script illustrated by fig. 2 andlisted in in the Appendix. Ignoring the effect of an externalelectric field for now, the mathematical problem solved bythe script is:

∂u

∂t=

1

ε

(u− u2/3− v

)+∇D∇u,

∂v

∂t= ε (u+ β − γv) . (3)

The script implements a forward Euler timestep (see Sec-tion 4.2.1 below) for this equation, using two devices:

diff v0=[u] v1=[i] Dpar=D Dtrans=D/4 hx=hx;euler v0=[u] v1=[v] ht=ht ode=fhncubpar={eps=eps bet=@[b] gam=gam Iu=@[i]};

In this fragment, the macro [u] expands to 0, that isthe very first layer of the grid allocated to the u field, [v]expands 1, standing for the second layer of the grid, allo-cated to the v field, and [i] expands to 2, which is thethird layer of the grid, for the value of the diffusion term,∇D∇u = χ−1∇σeff∇u. So, the function of diff is compu-tation of

(Iu)n = ∇D∇unwhere un stands for the u field at the current time step n(regarding the approximation of the second derivatives, seebelow, Section 4.3); the result of this computation, denotedhere as (Iu)n, is placed into layer 2 of the grid.

The euler device, with the parameter ode=fhncub, per-forms a forward Euler step for the cubic FitzHugh-NagumoODE system,

∂u

∂t=

1

ε

(u− u2/3− v

)+ Iu,

∂v

∂t= ε (u+ β − γv) + Iv,

in which parameters ε and γ are given by the values ofthe global variables eps and gam defined previously in thescript (in the included parameter file <fhn.par>), the valueof parameter β is taken from layer [b] which expands to 3(parameter β is spatially dependent in this simulation, andlayer 3 was pre-filled with values by the same k_func devicethat computed the initial conditions), the value of parame-ter Iu is taken from layer [i] which contains the values ofthe anisotropic diffusion term (Iu)n, computed for this time

step by the preceding diff device, and the value of param-eter Iv = 0 by default. Overall, with un(x, y) and vn(x, y)designating the fields u and v at the n-th time step, the pairof devices computes

un+1 = un + k

[1

ε

(un − 1

3un

2 − vn)

+∇D∇un],

vn+1 = vn + kε (un + β(x, y)− γvn) ,

where k is the time step, represented by the global variableht in the BeatBox script.

4.1.2 Operator splittingOperator splitting is another popular“divide and conquer”

strategy. In this approach, the right-hand sides still are splitinto simpler parts, but now an evolution sub-step is done foreach part in turn, as if this part was the whole right-handside. For example, computation of kinetics and diffusionin the right-hand side of (3) can be split into the kineticspart and diffusion part, and then one device performs thediffusion substep, and another device performs the kineticssubstep. So, the BeatBox script fragment from Section 4.1.1can be modified as

diffstep v0=[u] v1=[i] Dpar=D Dtrans=D/4 hx=hx ht=ht;euler v0=[u] v1=[v] ht=ht ode=fhncubpar={eps=eps bet=@[b] gam=gam};

where the device diffstep computes the diffusion term, anddoes a forward Euler step with it, as if this was the onlyterm in the equation. Combined with the fact that nowin the euler device the parameter Iu is not specified so itdefaults to zero, the given fragment of the script implementsthe following computation scheme:

un+1/2 = un + k[∇D∇un

],

un+1 = un+1/2 + k

[1

ε

(un+1/2 −

1

3u2n+1/2 − vn

)],

vn+1 = vn + kε(un+1/2 + β(x, y)− γvn

).

Once again, this is just a simple example illustrating how theBeatBox paradigm naturally fits the idea of operator split-ting. This of course applies first of all to the simplest (Lie)splitting; more sophisticated, higher-order operator splittingschemes could be implemented at the BeatBox script level,or on the device, i.e. a C-source code level.

4.2 Kinetics solvers

4.2.1 Explicit solversBoth the monodomain “reaction-diffusion” models of the

form (2) or the more complicated bidomain models (1) haveequations with time derivatives. Solving those equations inBeatBox is done as if they were ordinary differential equa-tions,

dV

dt= − 1

Cm(Iion(V ,g)− Ieff(~r, t))

dg

dt= f(g, V , ~r), (4)

(depending on ~r as a parameter) either with the value ofthe diffusion term, computed by the corresponding diffu-sion device, appearing in the voltage equation, or withinthe operator-splitting paradigm, i.e. performing time sub-steps as if the model was restricted to the ODEs representing

the reaction terms, leaving the space-dependent part of themodel to be computed at alternative sub-steps.

The simplest and arguably most popular in practice solverfor ODEs is the first-order explicit (time-forward) schemeknown as the forward Euler scheme, which for a systemof ODEs (4) means:

V n+1 = V n − k

Cm(Iion(V n,gn)− Ieff(~r, tn))

gn+1 = gn + kf(gn, V n, ~r), (5)

where tn is the n-th value in the time grid, k = tn+1 − tnis the time step, and V n = V (~r, tn), gn = g(~r, tn). Thisscheme is implemented in BeatBox in the euler device.

The Euler scheme’s well known disadvantages is its lowaccuracy due to only first-order approximation of the ODE,and, as any explicit scheme, only conditional stability (seee.g. [36]). The first disadvantage does not usually play a cru-cial role in cardiac dynamics studies as the proven accuracyof cardiac dynamics models themselves is not particularlyhigh; there is, however, rk4 device implementing Runge-Kutta fourth-order scheme for cases when accuracy isessential, and other standard explicit solvers may be easilyimplemented in a similar way. The stability considerationis more significant as for stiff models it severely limits themaximal allowable time step k, hence making simulationscostly.

4.2.2 Exponential solversThe standard solution to the stability problem is, of

course, using implicit or semi-implicit schemes. The latterpossibility is much more popular as fully implicit approachesfor nonlinear equations are numerically challenging. Amongthe semi-implicit approaches available in cardiac dynamics,the exponential scheme for ionic gates, known as the Rush-Larsen technique [37], is very popular. The idea is basedon the observation that in the models of ionic excitability,since the seminal work by Hodgkin and Huxley [20], an im-portant role is played by equations of the form:

dy

dt= α(V )(1− y)− β(V )y, (6)

where the dynamic variable y, called the gating variable,possibly in conjunction with other gating variables, deter-mines the permittivity of certain ionic currents. A conve-nient (even if not biophysically precise) interpretation is thata channel is open if all of the gates controlling that channelare open, and the variable y is the probability for that gateto be open. Hence α and β are transition probabilities perunit of time, of a closed gate to open, or for an open gateto close, respectively. In equation (6) the transition proba-bilities depend on the current value of the transmembranevoltage V , as in the Hodgkin-Huxley model; in more mod-ern models gating variables of some channels may dependon other dynamical variables, say the concentration of cal-cium ions. The importance of the gating variables is thatequations of the type (6) are often the stiffest in the wholecardiac excitation model. The Rush-Larsen scheme in itssimplest form can be obtained by assuming that V does notchange much during a time step, t ∈ [tn, tn+1], and replac-ing V (t) with the constant value V n = V (tn) turns (6) intoa linear equation with constant coefficients, the solution of

which can be written in a closed form, which gives:

yn+1 = A(V n) +B(V n)yn (7)

where

A(V ) = exp (−(α(V ) + β(V ))k) ,

B(V ) =α(V )

α(V ) + β(V )[1−A(V )] . (8)

As far as equation (6) is concerned, this scheme is uncondi-tionally stable, and gives an exact answer if V (t) = const,i.e. its first-order accuracy depends exclusively on the speedof change of the transmembrane voltage V . This schemeis implemented in the BeatBox device rushlarsen. Nat-urally, this device requires a more detailed description ofthe excitable model than euler: the gating variables y andtheir transition rates A, B need to be explicitly identifiedfor rushlarsen whereas euler only requires a definition ofthe functions computing the right-hand sides of the dynamicequations, i.e. Iion and g.

Some modern cardiac excitation models use a Markovchain description of the ionic channels. This description isbased on the assumption that an ionic channel can be in afinite number of discrete states, and transitions between thestates can happen with certain probability per unit of time,which may depend on control variables, such as transmem-brane voltage V or calcium ion concentration c. The timeevolution of the vector u of the probabilities of the channelto be in each particular state is described by the system oflinear ODE, known in particular as Kolmogorov (forward)equations, or the master equation

du

dt= M(V , c)u, (9)

where M is the matrix of transition rates. The extensionof the Rush-Larsen idea to this system was done in [40].Assuming again that the control variables do not changemuch within a time step and replacing them with a constant,V (t) = V n and c(t) = cn for t ∈ [tn, tn+1], the system (9)is a system of homogeneous linear equations with constantcoefficients and its exact solutions can be explicitly written.Assuming that M is diagonalizable, the resulting computa-tional scheme can be written as:

un+1 = T(V n, cn)un, (10)

where

T(V , c) = S(V , c) exp (Λ(V , c)k) S(V , c)−1 (11)

and S and Λ are respectively the matrix of eigenvectorsand the diagonal matrix of eigenvalues of M. This ma-trix Rush-Larsen scheme is also implemented in the de-vice rushlarsen mentioned earlier.

Finding eigenvalues and eigenvectors for the diagonalisa-tion and computing exponentials are relatively time consum-ing operations. For that reason the rushlarsen device doesa tabulation. That is, for the case when the coefficients A,B depend on V and matrices T depend only on one controlvariable, e.g. V (are “univariate”), their values are precom-puted for a sufficiently fine grid of the control variable atthe start time.

If matrix M depends on multiple control variables, e.g.both V and c (are “multivariate”), it can sometimes be pre-sented as a sum of univariate matrices. Then rushlarsen

uses Lie operator splitting and integrates each of the subsys-tems associated with each of the univariate matrices usingthe tabulated“matrix Rush-Larsen”separately. For some ki-netics models, M can be presented as a sum of one or moreunivariate matrices and a remainder, which is multivariatebut uniformly small. In that case the subsystem associatedwith the small remainder is done using the forward Eulermethod. Finally, if any such decomposition is not possible,“matrix Rush-Larsen” step still can be done, just withouttabulation, but by doing the diagonalization “on the fly”,i.e. at run time rather than start time. Although such com-putation is relatively costly, the benefit of larger time stepmay still outweigh the expense. The possibility of tabulatingmultivariate function theoretically exists but is not consid-ered in BeatBox due to resource implications.

The diagonalization is done using appropriate routinesfrom GSL [19]; the relevant subset of GSL is included inBeatBox distribution for portability and the users conve-nience.

4.3 Monodomain diffusion and boundaryconditions

We focus here on the device diff which is the main de-vice implementing the diffusion term in the monodomaindiffusion, which mathematically can be described as:

Lu =

3∑j,k=1

∂

∂xj

(Djk(~r)

∂u

∂xj

)(12)

with the naturally associated non-flux boundary conditions,

3∑j,k=1

nkDjk(~r)∂u

∂xj= 0 (13)

where ~n = (nk) is the normal to the boundary Γ of thedomain D, i.e. excitable tissue. Currently this is imple-mented for the transversely istoropic case, i.e. when thediffusion tensor D = (Cmχ)−1 σeff = (Djk) has only twodifferent eigenvalues: the bigger, simple eigenvalue D‖ cor-responding to the direction along the tissue fibers, and thesmaller, double eigenvalue D⊥, corresponding to the direc-tions across the fibres, as this is the most popular case inmodelling anisotropic cardiac tissue (the modification for thegeneral orthotropic case is straightforward though). In thiscase,

Djk = D⊥δjk +(D‖ −D⊥

)f jfk, (14)

where ~f = (fk) is the unit vector of the fiber direction.The simple finite-difference approximation of (12,13) in diff

device is along the lines described, e.g. in [14]. In detail, wehave

(Lu)p =∑

q∈{0,±1}3Wp

qup+q, (15)

where p ∈ Z3 is the 3D index of a grid node with positionvector ~rp, up = u(~rp) is the value of the field u at thegrid node p, (Lu)p is the value of the diffusion operatorapproximation at that point, q ∈ {0,±1}3 is its increment,and the weights Wp

q are defined by the following expressions:

Wpq = W

pq + W

p

q, (16)

Wpq =

ψp+q

h2

Dpjj , q = ±qj ,

12Dpjk, q = ±(qj + qk), j 6= k,

− 12Dpjk, q = ±(qj − qk), j 6= k,

0, q = ±q1 ± q2 ± q3,

, (17)

Wp

q =ψp+qψp−q

4h2

±3∑j=1

(D

p+qj

jk −Dp−qj

jk

), q = ±qk,

0, otherwise,

(18)

Wp(0,0,0) = −

∑q6=(0,0,0)

Wpq, (19)

where j, k ∈ (1, 2, 3), ψp is the grid indicator function of thedomain D, that is, ψp = 1 if ~rp ∈ D and ψp = 0 otherwise,Dpjk = Djk (~rp), q1 = (1, 0, 0), q2 = (0, 1, 0), q3 = (0, 0, 1).

10-4

10-3

10-2

10-1

10-2

10-1

err

or

h

‖ε‖L∞‖ε‖L2

∝ h∝ h2

Figure 3: Numerical convergence of the solution ofthe problem (20)–(24). Slope lines are with slopes1, 2 and best fits with slopes 1.564 for L∞, and 1.719for L2.

The above discretization is probably the simplest possi-ble approach; there are alternatives available, see for exam-ple [18], however these require extra information about Dbeyond the grid function ψp. We assess the approximationproperties of the simple scheme described above by solvingthe following initial-boundary value problem for the diffu-sion equation:

∂u

∂t=∂2u

∂x2+∂2u

∂y2, (x, y) ∈ D, t ∈ [0, T ]; (20)

u = J0

(γ√

(x− x0)2 + (y − y0)2), (x, y) ∈ D, t = 0;

(21)

∂u

∂~n= 0, (x, y) ∈ Γ, t ∈ R+; (22)

D ={

(x, y)|(x− x0)2 + (y − y0)2 ≤ 1}, (23)

Γ = ∂D ={

(x, y)|(x− x0)2 + (y − y0)2 = 1}. (24)

Here J0(·) is Bessel function of the first order of index 0,and γ ≈ 3.8317 . . . is the first positive root of its derivative,J′0(γ) = 0. This problem has an exact solution:

u = J0

(γ√

(x− x0)2 + (y − y0)2)e−γ

2t. (25)

Figure 3 illustrates the numerical convergence of the solu-tion of the problem (20)–(24) by BeatBox using diff de-vice to the exact solution provided by (25), for T = 0.2.The timestepping is by using a forward Euler scheme with

a time step of k = h2/80. The error, i.e. the differenceε(~r, t) = u](~r, t) − u(~r, t) between the exact solution u(~r, t)and its approximation obtained numerically, u](~r, t), is char-acterized by two norms,

‖ε‖L∞ = maxt∈[0,T ]

max~r∈D|ε(~r, t)| , (26)

‖ε‖L2 =

1

Tµ(D)

T∫0

∫D

|ε(~r, t)|2 d2~r dt

1/2

. (27)

where µ(D) stands for the area of D, and all the integrals arecalculated by the trapezoidal rule. Each h has four corre-sponding points on each graph, corresponding to four sim-ulations, with different position of the centre of the circle(x0, y0) with respect to the grid (~rp). We can see that theconvergence is worse than h2, but better than h1. The L2

norm of the error converges faster than L∞ norm, which isan indication that the main source of error is localized —this is, of course, to be expected, as the boundary conditions,in a sense, approximate the curvilinear boundary Γ with oneconsisting of pieces of straight lines parallel to the x and yaxes, thus typically making an error O (h). We stress thatin cases where the realistic tissue geometry is available as aset of points with the same resolution as the computationalgrid, the knowledge of any curvilinear boundary is in anycase unavailable, so any loss of accuracy associated with it,or, equivalently, any notional gain of accuracy that would beassociated with using a curvilinear boundary instead, wouldbe purely theoretical.

4.4 Bidomain diffusion

10-4

10-3

10-1

100

err

or

h

‖ε‖L∞

‖ε‖L2

∝ h2

Figure 4: Numerical convergence of the solution ofthe problem (29)–(35). Slope lines: slope 2 (black)and best fits with slopes 2.009 for L∞, and 1.9889 forL2.

Computations using the bidomain tissue description (1)differ primarily by the presence of the equation:

∇ · (σi + σe)∇Φi = ∇ · σe∇V − Iext, (28)

which is elliptic with respect to both Φi and V . We haveimplemented a solver for elliptic equations in the elliptic

device. This uses Full Multigrid iterations with vertex-centered restriction/prolongation operators with bi/tri-linear interpolation, and a (multicoloured) Gauss-Seidel ora Jacobi smoother.

The linear system to which the solver applies naturally oc-curs through discretization of the diffusion operator in thesame way as described in the previous subsection. The solu-tion to the problem (28) with non-flux boundary conditions

is not unique, so the solver will require an extra ”pinning”condition, to select a unique solution out of infinitely many:the solution at a given point should have a given value. Forsolving the bidomain model (1) using operator splitting, theelliptic device can be used to solve the elliptic equationwith respect to Φi, leaving the parabolic diffusion equationfor timestepping V using the diff device and timesteppingV and g according to the reaction kinetics via an ODEsolver, such as euler device.

We illustrate this computation scheme on an exampleof a bidomain problem with an exact solution. We con-sider a bidomain system (1) with a one-component “cellmodel”, dim g = 0, corresponding to the Zeldovich-Frank-Kamenetsky [42] also known as Nagumo equation [28] andSchlogl model [38]:

∂V

∂t= V (V − α)(1− V ) +

(Di‖∂

2x +Di

⊥∂2y

)Φi, (29)[

(De‖ +Di

‖)∂2x + (De

⊥ +Di⊥)∂2

y

]Φi =

(De‖∂

2x +De

⊥∂2y

)V .

(30)

If posed on the whole plane, (x, y) ∈ R2, this system has afamily of exact solutions in the form of plane waves,

V ∗ ={

1 + exp[(x cos θ + y sin θ − s− ct) /

√2D∗

]}−1

,

(31)

Φ∗i = KV ∗, (32)

where the angle of the wave propagation, θ, and its initialphase, s, are arbitrary constants, and the other parametersof the solution are defined by c =

√2D∗

(12− α

), D∗ =

Di∗D

e∗/(D

i∗ + De

∗), K = De∗/(D

i∗ + De

∗), Di∗ = Di

‖ cos2 θ +

Di⊥ sin2 θ, De

∗ = De‖ cos2 θ +De

⊥ sin2 θ.For testing BeatBox as a bidomain solver, we consider the

problem for the system (29,30) in a square domain of sizeL, D = [0, L]2, for a time interval t ∈ [0, T ], with the initialand (non-homogeneous Dirichlet) boundary conditions setin terms of (31,32) as

V (x, y, 0) = V ∗, Φi(x, y, 0) = Φ∗i , (x, y) ∈ D; (33)

V (x, y, t) = V ∗, Φi(x, y, t) = Φ∗i , (x, y) ∈ Γ, t ∈ (0, T ),(34)

Γ = ∂D = {0, L} × [0, L] ∪ [0, L]× {0, L}. (35)

To implement solution of this problem in a BeatBox script,we split the equations into fragments, determning the sub-steps in the calculations:

diff(1) : S1 = ∇De∇V ,

elliptic : ∇(De + Di

)∇Φi = S1,

diff(2) : S2 = ∇Di∇Φi,

euler : V t = V (V − α)(1− V ) + S2.

The resulting fragment of BeatBox looks like this:

def str domain x0=xil x1=xir y0=yil y1=yir;// source term in the elliptic equationdiff [domain] v0=[u] v1=[s]Dpar=Dex Dtrans=Dey hx=hx;

// solving the elliptic equationelliptic [domain] v0=[s] v1=[p]Dpar=Dex+Dix Dtrans=Dey+Diy hx=hx

tolerance=tol delta=0.5 upper_level=3vcycles=20 preiter=1 postiter=2 maxiter=1e6;

// source term in the parabolic equationdiff [domain] v0=[p] v1=[s]Dpar=Dix Dtrans=Diy hx=hx;

// timestep of the parabolic equationeuler [domain] v0=[u] v1=[u]ode=zfk ht=ht par={alpha=alpha Iu=@[s]};

In this fragment, the first diff device computes the right-hand side of the elliptic equation, S1 = ∇ · De∇V and de-posits the result S1 into the layer [s]; then elliptic solvesthe elliptic equation for Φi using the provided fine-tuningalgorithm parameters and puts the result into the layer [p].

The second diff device computes∇·Di∇Φi and puts the re-sult S2 into the layer [s] (which is therefore“recycled”), andthe euler device does the time step of the cell model. Theinterior of the domain D is mapped to the subgrid [domain]

with the grid x-coordinate from xil to xir and y-coordinatefrom yil to yir. The non-homogeneous, non-stationaryDirichlet boundary conditions (34) were implemented by ak_func device, computing the boundary values for V andΦi for the grid nodes surrounding this subgrid [domain],i.e. those with grid coordinates xil-1,xir+1,yil-1,yir+1(this part of the script is not shown).

The accuracy of the computational scheme is illustratedin fig. 4. We take L = 10, T = 40, α = 0.13, Di

‖ = 2,

Di⊥ = 0.2, De

‖ = 8, De⊥ = 2 and s = −5. The time step

k = 3h2/(16De

‖)

is varied together with the space step with

the coefficient 3/(16De

‖)

chosen from the considerations ofnumerical stability [14], resulting in quadratic convergenceof the algorithm, as should be expected.

5. COMPLEX GEOMETRIES AND DO-MAIN PARTITIONING

Sharing work between processes in the MPI version ofBeatBox is done by splitting the computational grid intosubdomains, such that computation in each subdomain isdone by a process. The method of domain decomposition isillustrated in fig. 5, for a 2D domain. Each of the x, y andz dimensions of the grid is separated by a certain number ofequal subintervals (approximately equal, when the grid sizeis not divisible by the number of subintervals). The numberof subintervals in different dimensions do not have do be thesame. In the example shown in fig. 5, the x and y dimensionsare split to have 3 subintervals each; the z dimension is notsplit. The grid nodes, in which computations are done, arerepresented in fig. 5 by solid circles (“bullets”).

The continuity of computations across subdomains neces-sary for devices involving the diffusion operator is achievedby using message passing with exchange buffers. Thedepth of the exchange buffer in each direction is one gridpoint. This imposes a limitation on the stencils that can beused by diffusion-like devices, such as a 9-point stencil for2D and up to a 27-point stencil for 3D. In fig. 5, the hollowcircles represent the fictitious grid nodes which are imagesof corresponding nodes from neighbouring subdomains, andthe dashed lines designate the whole buffers, including thenodes to be sent and nodes to be received. The buffer ex-change should be effected twice (forwards and backwards)for each dimension, i.e. four exchanges in 2D simulationsand six exchanges in 3D simulations. If the buffer exchangesare done in the correct order, then this will ensure correct

⇔ ⇔

⇔ ⇔

⇔ ⇔⇔

⇔

⇔⇔

⇔⇔

Figure 5: Schematic of the domain partitioning inMPI implementations of BeatBox. Solid circles repre-sent nodes on which actual computations are done,empty circles are the “halo” points, the rectanglesdenote the exchange buffers and the solid black linerepresents the boundary of the computational do-main (excitable tissue).

exchange of node values in the diagonal directions as well(magic corners, see [26] for details).

In bidomain computations, the buffer exchanges can bedone either before each iteration or more seldom; in thelatter case the result is different from the sequential run,but inasmuch as both MPI and sequential results are closeto the actual solution, the difference between the two shouldbe negligible. Similar consideration applies to the use ofthe Gauss-Seidel smoother, which of course will also givedifferent results if applied in each subdomain separately.

When working with complex (non-cuboidal) domains, theBeatBox approach is to inscribe the domain into the small-est cuboid, and then proceed as before, with the differencethat computations are only done at those grid nodes thatbelong to that domain, and the one outside domain remainidle. This is also illustrated schematically in fig. 5, wherethe boundary of the domain is drawn by a closed bold solidblack line. This creates a challenge to the performance: withhigh-degree parallelization and complicated geometry of thedomain, the load imbalance between processes can becomesignificant; in particular, a large number of partitions willcontain no points of the domain (in fig. 5, there is one suchpartition, in the bottom left corner). This problem is wellknown and there are efficient tools for solving them for struc-tured as well as unstructured grids, see e.g. [24, 21]. In thecurrent version of BeatBox, however, only the crudest opti-mization method is used: the partitions that are completelyidle are not allocated to processes, which considerably limitsthe expected slow-down because of the uneven load (roughlyspeaking, at worst twice on average).

6. INPUT/OUTPUT AND VISUALIZATIONFinally, we briefly mention some input/output options

currently available in BeatBox. Each of these options is im-plemented in the corresponding device. This includes:

• Full precision binary input and output of a specifiedsubset grid (devices load and dump respectively);

• Discretized “fixed-point” (1 byte per value) output ofthree selected layers of a subset of the grid (deviceppmout);

• Plain text outputs of a defined subset of the grid orlist of expressions involving global variables (devicesrecord and k_print).

Some computational devices also have i/o options. Forinstance, device k_func, which performs computations byformulas specified in the BeatBox script, can read data fromof a plain text file; such data are interpreted as a tabulatedunivariate vector-function and is often used to create initialconditions by phase-distribution method. Another exampleis device singz, which finds phase singularities in z-cross-sections of the grid. In addition to assigning the coordinatesof the signularity points to global variables, it can also out-put those to a plain text file and/or visualize. Many deviceshave an optional debug parameter for printing plain textmessages about details of their work.

Regarding run-time visualization, the sequential versionof BeatBox has a number of devices for 1D and 2D visual-ization via X11 protocol if available (devices k_draw, k_plot,k_paint). 3D output typically requires much more tuning tobe effective. Theoretically, one possibility is to do the tuningin the interactive mode while the computations are stopped,as it is done e.g. in EZSCROLL, see [2]; this, however, wouldgo against one of the principles of BeatBox, that all detailsof the run are specified in the input script, so any simulationis reproducible. Instead, currently the visualization is doneby post-processing of the output data; most often the dataproduced by ppmout.

7. PARALLEL SCALING PERFORMANCEFigure 6 illustrates the computation time taken by the

MPI version of BeatBox on ARCHER (UK national super-computing facility, http://www.archer.ac.uk/) as a functionof the number of processes, for three series of test jobs, pre-senting different challenges from the parallelization view-point. In all series, the jobs simulated the monodomainmodel (2) with Courtemanche et al. 1998 model of humanatrial cells [15], with dim g = 23, and anisotropic diffusion(19-point stencil), but with different geometries. Figure 6(a)is for a cubic grid of 300× 300× 300 points. Figure 6(b) isfor the rabbit ventricles geometry, described in [17], which isinscribed in a cuboid grid 119×109×131, containing 470324tissue points, which is about 27.7% out of the total numberof points in the grid. Figure 6(c) is for the human atrialgeometry [39], inscribed into a cuboid grid 237× 271× 300,containing 1602787 tissue points, about 8.3% of the total.The human atrium geometry tests were using the crude op-timization of partitioning, i.e. the subdomains that do notcontain tissue points are not allocated to processes; the rab-bit ventricle jobs are done without such optimization. Inall job series, the simulation was with an initial conditionof a scroll wave with a filament along the y-axis, using the

(a)

10-1

100

Run

tim

e, sec/t

ime

ste

p

with ppm

without ppm

ideal

(b) 10-3

10-2

Run

tim

e,

sec/tim

e s

tep

with ppm

without ppm

ideal

(c)

10-2

10-1

102

103

Runtim

e,

sec/t

ime s

tep

Number of cores

with ppm

without ppm

ideal

Figure 6: Log-log plots: the wall clock time per onetime step in the simulation job, vs the number ofcores. (a) Full box; (b) Rabbit ventricle geometry,(c) Human atrium geometry.

procedure described e.g. in [22]. The output was via ppmout

device, which outputs up to three selected layers of the grid,discretized to the 0..255 scale (3D extension of the ppm for-mat of Netpbm, [35]); the corresponding curves in the graphsare marked as “without ppm” and “with ppm” respectively.Such outputs were done once in every 200 timesteps for thefull box and human atrium geometries, and once in every1000 timesteps for the rabbit atrium geometry. The fullbox and human atrium jobs also did plain text output ofthe activity at a single point once in every 20 time steps.To test the expenses associated with sophisticated control,the full box and human atrium jobs implemented feedback-controlled stimulation, similar to that implemented in thesample script described in the Appendix. To exclude theeffects of the time taken by the start-up operations, we com-puted the time per step by running jobs identical in all re-spect except the number of time steps, and then consideringthe difference. ARCHER has 24 cores per node, and thenumbers of processes in the test jobs are power-of-two mul-tiples of 24. The “ideal” lines are drawn based on the resultof the “without-ppm” job on 24 processors.

We observe that the effect of feed-back control and plaintext outputs on the parallel performance is relatively small,and the main slow-down at high parallelization happens due

to uneven load of the processes. The parallel scaling is, asexpected, best for the full box: without ppmout it is close toideal for up to 1536 processes, while the curves for the com-plicated geometries deviate from ideal noticeably. Humanatrium geometry is “much thinner” than the rabbit ventriclegeometry, and the deviation from ideal is more pronouncedin fig. 6(c) than in fig. 6(b). However, the detrimental effectof the uneven load is limited: notice that the “without ppm”curve in fig. 6(c) is almost parallel to the ideal in the in-terval of 192–1536 cores, and the slow down is only slightlymore than by a factor of two. Another significant factor isthe bulk output. In the human atrium geometry, such out-puts were 10 times more frequent, and their effects is morepronounced overall and starts increasing at smaller paral-lelizations. Notice that the relative effect of the bulk outputis much less in the full box: an obvious explanation is thatthe ppmout format always outputs the full enclosing gridwhile computations are only done in the tissue nodes, hencethe output/computation ratio is about 12 times bigger forhuman atrium than in the full box.

We conclude that the parallel performance is in accor-dance with expectations and satisfactory. The biggest chal-lenge is to simulations with complex and “thin” geometries,and a significant improvement may be achieved by optimiz-ing bulk outputs.

8. EXAMPLES OF USE IN RECENT ANDCURRENT RESEARCH

Figure 7: Generation of a scroll wave out of miscro-scopic re-entries in excitable medium with random,space- and time-dependent distribution of parame-ters, modelling movement of ischaemic border zoneduring reperfusion; Beeler-Reuter [3] kinetics [6].

BeatBox has been used to produce simulation results pre-sented in dozens of publications. In this section, we mentiona handful of recent and representative studies, illustratingthe key features of this software.

Figure 7 illustrates the possibility of implementing compli-cated simulation set-ups. This figure is an example from [6]

which modelled arrhythmogenic mechanisms of the bound-ary layer between ischaemic and normal cardiac tissue, mov-ing due to reperfusion. The model assumed that a certain“excitability” parameter was varying in space and time dueto two factors: firstly, space-only random distribution due toproperties of individual cells; secondly, deterministic smoothtransition between low excitability of the ischaemic tissueand high excitability of the recovered tissue, changing intime due to reperfusion. On top of that, the isotropic dif-fusion coefficient was also varied along a similar transitionbetween low diffusivity of the ischaemic tissue and high dif-fusivity of the recovered tissue, of a profile different from,moving synchronously with of the excitability parameter.

Figure 8: Drift of scroll wave along a thickness step,FitzHugh-Nagumo kinetics[11].

Figures 8, 9, 10 illustrate simulations in non-cuboid do-mains. Figure 8 shows a surface view to a simulation in anartificially defined domain, used to quantitatively test pre-dictions of an asymptotic theory about the drift of a scrollwave in a thin layer due to sharp variations of thickness.This simulation uses two-component FitzHugh-Nagumo ki-netics. Shown is the surface view at a selected moment oftime, colour coding represents states of the activator vari-able (red colour component) and inhibitor variable (greencolour component) on dark-blue background; the white lineshows the trajectory of the tip of the spiral wave seen at theupper surface of the domain for a period of time precedingthe moment the presented surface view.

(a) (b)

Figure 9: Drift of scroll wave in a realistic humanatrium geometry, Courtemanche et al. [15] kinet-ics, (a) spontaneous, caused purely by the anatomyfeatures [22]; (b) resonant, caused by feedback-controlled electrical stimulation [23].

Figure 9(a) illustrates the corresponding geometry-induced drift in anatomically realistic model of human

atrium. Shown are a number of trajectories of tips of spiralwaves appearing on the surface of the atrium nearest to theviewer; the yellow background indicates prominent anatom-ical features (the pectinate muscles and the terminal crest).To make the visualization clearer, the trajectories are rep-resented by lines connecting tip positions separated by ex-actly one period of rotation (“stroboscopic view”); shown areseveral trajectories starting at different initial positions andmade within equal time intervals.

Figure 10: A scroll wave in human foetal heartanatomy, FitzHugh-Nagumo kinetics [34, 12].

Figure 10 shows a volume view of a scroll wave in a humanfoetal heart geometry [34, 12]. Shown is the surface u =const, where u is the activator variable (FitzHugh-Nagumomodel was used in this case for illustrative purposes), pink,the boundary of the computational domain, i.e. surface ofthe ventricular geometry, is shown in blue, both surfaces aresemi-transparent. The yellow line traversing the ventricularwall is the filament of the scroll, defined as the intersectionof the same u = const surface shown in the picture with av = const surface.

Finally, fig. 9(b) illustrates further the capability ofBeatBox for complictaed simulation protocols. This paneldisplays results of simulations in the same model as thoseshown in fig. 9(a), but now the initial position for thescroll wave is chosen far from any sharp features so theanatomy-induced drift is not pronounced, and instead, thescroll wave is subject to low-voltage pulses of external elec-tric field, Ieff = Ieff(t). The delivery of the pulses is con-trolled by a protocol similar to that illustrated by the samplescript fig. 11, namely:

• There is a “registration electrode”, at a point at thesurface of the atrium that is the most distant from theviewer, position of which is indicated by the purplecone.

• The signal from the registration electrode is monitoredfor the moment of arrival of an excitation wave, de-fined as the moment when the transmembrane voltagecrosses a certain threshold value upwards.

• From the moment of the front arrival to the registra-tion electrode, a certain waiting interval (delay) is ob-served.

• On expiration of the delay interval, a pulse of Ieff(t) ofa certain duration and certain amplitude is applied.

In fig. 9(b) we see three different trajectories starting at thesame point: they differ in the value of the delay intervalbetween registration of the front arrival and delivery of theelectric pulse.

The visualization in all cases was done by post-processingof the simulation data. For fig. 7, we used Iris Explorer [33].Both panels of fig. 9 were generated with ParaView [1].Figures 8 and 10 were produced by an in-house visual-izer, based on the graphical part of Barkley and Dowle’sEZSCROLL [16, 2], which is in turn based on the MarchingCubes algorithm [25, 29].

9. CONCLUSIONSThe leading idea underlying BeatBox development is ro-

bustness, portability, flexibility and user-friendliness in thefirst place, connected with efficiency as an important butsecondary consideration. In the present form, BeatBox canbe exploited in sequential and parallel (MPI) modes, withrun-time and/or post-processing visualization, on any unix-like platform from laptops to supercomputing facilities. Themodular structure of BeatBox effectively decouples the userinterface, which at present is a scripting language used toconstruct the ring of devices, from the implementation ofthe computationally intensive stages in individual devices.The current computational capabilities have emerged fromthe needs of the BeatBox developers in their research so far,and can be expanded in accordance with the needs of widerusership without changing the backbone ideology.

As far as MPI features are concerned, the straightforwardapproach to parallelization via domain decomposition, yieldsacceptable results for small to medium scale, becoming in-efficient for number of threads beyond about 1000 for theexamples tried. The limiting factors seem to be the unevenload of the parallel threads for “thin” complex geometries ofthe computational domains, and output, which determinespossible direction of further development. The uneven loadcan be addressed by a more careful fine-tuning of domaindecomposition to specifics of particular domain geometry,which to some extent may be achieved without violating themain principles of the domain decomposition, by allowinguneven partitioning along the coordinate axes. The slowdown in cases of extensive output is a problem which is notspecific for BeatBox; however, some improvement in somecases may be achived by making any input-output opera-tions exclusive to one or more designated threads specializ-ing on this and relieved from computational load as such.

As the current BeatBox solvers use finite difference, regu-lar grid ideology, incorporation of DT-MRI regular cartesiangrid anatomy models into BeatBox simulations is straightforward, fig. 9 and 10, without a meshalizer step requiredfor finite element/finite volume solvers. However, architec-turally there is no principal problems in extending BeatBox

functionality to the finite element approximation as long asregular mesh of finite elements is used that can be mappedto a rectangular array. Extension to irregular meshes wouldrequire more substantive changes, however the main idea ofthe ring of devices may be useful there as well.

10. ACKNOWLEDGMENTS

The early stages of development of BeatBox and its pre-decessors benefited from contributions from A.V. Karpovand R. McFarlane. The recent development of BeatBox wassupported by EPSRC grant EP/I029664 (UK). VNB alsogratefully acknowledges the current financial support of theEPSRC via grant EP/N014391/1 (UK).

11. REFERENCES[1] U. Ayachit. The ParaView Guide: A Parallel

Visualization Application Paperback. Kitware, USA,2015.

[2] D. Barkley and M. Dowle. EZ-SCROLL: A code forsimulating scroll waves. http://www.warwick.ac.uk/˜masax/Software/ez software.html, 2007.

[3] G. W. Beeler and H. Reuter. Reconstruction of actionpotential of ventricular myocardial fibers. J. Physiol.— London, 268(1):177–210, 1977.

[4] V. N. Biktashev. A three-dimensional autowaveturbulence. Int. J. of Bifurcation and Chaos,8(4):677–684, 1998.

[5] V. N. Biktashev. Dissipation of the excitationwavefronts. Phys. Rev. Lett., 89(16):168102, 2002.

[6] V. N. Biktashev, I. V. Biktasheva, and N. A.Sarvazyan. Evolution of spiral and scroll waves ofexcitation in a mathematical model of ischaemicborder zone. PLoS ONE, 6(9):e24388, 2011.

[7] V. N. Biktashev and A. V. Holden. Design principlesof a low-voltage cardiac defibrillator based on theeffect of feed-back resonant drift. J. Theor. Biol.,169(2):101–113, 1994.

[8] V. N. Biktashev, A. V. Holden, and H. Zhang. Tensionof organizing filaments of scroll waves. Phil. Trans.Roy. Soc. Lond. ser. A, 347:611–630, 1994.

[9] I. V. Biktasheva and V. N. Biktashev. Wave-particledualism of spiral waves dynamics. Phys. Rev. E,67:026221, 2003.

[10] I. V. Biktasheva, V. N. Biktashev, W. Dawes,A. Holden, R. Saumarez, and A.M.Savill. Dissipationof the excitation front as a mechanism ofself-terminating arrhythmias. Int. J. of Bifurcationand Chaos, 13(12):3645–3655, 2003.

[11] I. V. Biktasheva, H. Dierckx, and V. N. Biktashev.Drift of scroll waves in thin layers caused by thicknessfeatures: asymptotic theory and numericalsimulations. Phys. Rev. Lett., 114(6):068302, 2015.

[12] F. Cai, E. Pervolaraki, A. V. Holden, and I. V.Biktasheva. paper in preparation, 2016.

[13] C. E. Clancy and Y. Rudy. Linking a genetic defect toits cellular phenotype in a cardiac arrhythmia. Nature,400:566–569, 1999.

[14] R. H. Clayton and A. V. Panfilov. A guide tomodelling cardiac electrical activity in anatomicallydetailed ventricles. Prog. Biophys. Mol. Biol.,96:19–43, 2008.

[15] M. Courtemanche, R. J. Ramirez, and S. Nattel. Ionicmechanisms underlying human atrial action potentialproperties: insights from a mathematical model. Am.J. Physiol., 275:H301–H321, 1998.

[16] M. Dowle, R. M. Mantel, and D. Barkley. Fastsimulations of waves in three-dimensional excitablemedia. International Journal of Bifurcation andChaos, 7(11):2529–2545, 1997.

[17] F. H. Fenton, E. M. Cherry, H. M. Hastings, and S. J.Evans. Multiple mechanisms of spiral wave breakup ina model of cardiac electrical activity. CHAOS,12:852–892, 2002.

[18] F. H. Fenton, E. M. Cherry, A. Karma, and W.-J.Rappel. Modeling wave propagation in realistic heartgeometries using the phase-field method. Chaos,15:013502, 2005.

[19] M. Galassi et al. GNU scientific library.http://www.gnu.org/software/gsl/manual/gsl-ref.pdf,2015.

[20] A. L. Hodgkin and A. F. Huxley. A quantitativedescription of membrane current and its application toconduction and excitation in nerve. J. Physiol. Lond.,117:500–544, 1952.

[21] G. Karypis. METIS: A software package forpartitioning unstructured graphs, partitioning meshes,and computing fill-reducing orderings of sparsematrices. http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf, 2013.

[22] S. R. Kharche, I. V. Biktasheva, G. Seeman,H. Zhang, and V. N. Biktashev. A computersimulation study of anatomy induced drift of spiralwaves in the human atrium. BioMed ResearchInternational, 2015:731386, 2015.

[23] S. R. Kharche, I. V. Biktasheva, G. Seemann,H. Zhang, J. Zhao, and V. N. Biktashev.Computational modelling of low voltage resonant driftof scroll waves in the realistic human atria. LectureNotes in Computer Science, 9126:421–429, 2015.

[24] D. LaSalle, M. M. A. Patwary, N. Satish,N. Sundaram, P. Dubey, and G. Karypis. Improvinggraph partitioning for modern graphs andarchitectures. In 5th Workshop on Irregularapplications: Architectures and Algorithms,Supercomputing, 2015.

[25] W. Lorensen and H. Cline. Marching cubes: A highresolution 3D surface construction algorithm. Comput.Graph., 21(4):163–169, 1987.

[26] R. McFarlane. High-Performance Computing forComputational Biology of the Heart. PhD thesis,University of Liverpool, 2010.

[27] R. McFarlane and I. V. Biktasheva. Beatbox — acomputer simulation environment for computationalbiology of the heart. In VISIONS OF COMPUTERSCIENCE: BCS International Academic Conference,pages 99–109. The British Computer Society, London,UK, 2008.

[28] H. P. McKean, Jr. Nagumo’s equation. Advances inMathematics, 4(3):209–223, 1970.

[29] C. Montani, R. Scateni, and R. Scopigno. A modifiedlook-up table for implicit disambiguation of marchingcubes. The Visual Computer, 10(6):353–355, 1994.

[30] M. Nichols, N. Townsend, P. Scarborough, M. Rayner,J. Leal, R. Luengo-Fernandez, and A. Gray. Europeancardiovascular disease statistics, 2012 edition.http://www.escardio.org/static file/Escardio/Press-media/press-releases/2013/EU-cardiovascular-disease-statistics-2012.pdf,2012.

[31] G. Nicolis and A. De Wit. Reaction-diffusion systems.Scholarpedia, 2(9):1475, 2007.

[32] D. Noble. Unraveling the genetics and mechanisms ofcardiac arrhythmia. Proceedings of the NationalAcademy of Sciences, 99(9):5755–5756, 2002.

[33] Numerical Algorithms Group. Iris Explorertm.http://ftp.nag.co.uk/Welcome IEC.asp, 2016.

[34] E. Pervolaraki, R. A. Anderson, A. P. Benson,B. Hays-Gill, A. V. Holden, B. J. R. Moore, M. N.Paley, and H. G. Zhang. Antenatal architecture andactivity of the human heart. INTERFACE FOCUS,3(2):20120065, 2013.

[35] J. Poskanzer, B. Henderson, et al.http://netpbm.sourceforge.net/.

[36] L. Ridgeway Scott. Numerical Analysis. PrincetonUniversity Press, Princeton and Oxford, 2011.

[37] S. Rush and H. Larsen. A practical algorithm forsolving dynamic membrane equations. IEEE Trans.BME, 25(4):389–392, 1978.

[38] F. Schlogl. Chemical reaction models fornon-equilibrium phase transitions. Z. Phys. A,253(2):147–161, 1972.

[39] G. Seemann, C. Hoper, F. B. Sachse, O. Dossel, A. V.Holden, and H. Zhang. Heterogeneousthree-dimensional anatomical and electrophysiologicalmodel of human atria. Phil. Trans. Roy. Soc. A,364:1465–1481, 2006.

[40] T. Stary and V. N. Biktashev. Exponential integratorsfor a Markov chain model of the fast sodium channelof cardiomyocytes. IEEE Trans. BME,62(4):1070–1076, 2015.

[41] M. Veldkamp, P. Viswanathan, and C. Bezzina. Twodistinct congenital arrhythmias evoked by amultidysfunctional Na+ channel. Circulation Research,86:e91–e97, 2000.

[42] Y. B. Zel’dovich and D. A. Frank-Kamenetsky.Towards the theory of uniformly propagating flames.Doklady AN SSSR, 19:693–697, 1938.

APPENDIXA sample BeatBox scriptThe complete listing of a simple but non-trivial example ofa BeatBox script, which is illustrated by fig. 2, is providedin fig. 11. The main features of the syntax may be seen fromthe script itself which is intended to be self-explanatory, butnevertheless:

• Comments in the script can be in C style, within/*...*/ or in C++ style, between // and the end ofline.

• Expression <fhn.par> means inclusion of another textfile with name fhn.par, as part of the script, similar to#include <fhn.par> in C.

• The script is a sequence of sentences, each concludingwith a semicolon, ‘;’.

• Sentences starting with keyword def declare globalvariables, with optional initial values.

• Global variables of type str are string macros, expan-sion of a string macro declared as def str foo bar;

is done using syntax [foo] which will produce bar inplace of expansion.

• The script finishes with a sentence end;

<fhn.par> // model pars are read in from file fhn.pardef str b 3; // spatially dependent parameter in 4th layerdef real grad [1]; // its gradient is 1st command-line parameter

// Integer and real stimulation parametersdef int xr 100; def int yr 100; def int zr 100; // reg electrode position, in space stepsdef int dr 5; // reg elecrode size, in space stepsdef real Amp 3.0; // pulse amplitudedef real Dur 0.1; // pulse durationdef real Del 6.0; // pulse delaydef real Tstart 100.0; // when to switch on the feedback loop

state geometry=ffr.bbg anisotropy=1 // the file contains tissue geometry and fibresvmax=4; // 2 dyn vars + diffusive current + parameter

def real T;def real begin;def real out;def real end; // real vars control works of some devicesk_func name=timing nowhere=1 pgm={ // this function operates only global variablesT = t*ht; // t is integer time counter; T is real timebegin = eq(t,0); // 1.0 at the very beginning, otherwise 0.0out = eq(mod(t,100),0); // 1.0 every 100 timesteps, otherwise 0.0end = ge(T,100.0)}; // 1.0 after 100 ms, otherwise 0.0

// This function operates at every point but only in the very beginningk_func when=begin pgm={u[u]=ifle0(x-25,1.7,-1.7); u[v]=ifle0(y-25,0.7,-0.7) // Cross-field initial conditionsu[b]=bet+grad*(z-0.5*zmax)}; // vertical gradient of parameter

// The feedback loopdef real signal;def real front; def real Tfront;reduce operation=max result=signal v0=[u] v1=[u] // signal=max of voltage field within given volumex0=xr xr1=xr+dr-1 y0=yr yr1=yr+dr-1 z0=zr zr1=zr+dr-1; // the values are arithmeitc expressions

k_poincare nowhere=1 sign=1 // remember T when signal crossed value umid upwardpgm={ front=signal-umid; Tfront=T };

k_func nowhere=1 // force lasts Dur ms starting Del ms after crossingpgm={ force=ht*Amp*ge(T,Tstart)*ge(T,Tfront+Del)*le(T,Tfront+Del+Dur) };

// The computationdiff v0=[u] v1=[i] Dpar=D Dtrans=D/4 hx=hx; // anisotropic diffusionk_func when=force pgm={u[i]=u[i]+force}; // this applies everywhere, only when force is nonzeroeuler v0=[u] v1=[v] ht=ht ode=fhncub // cubic FitzHugh-Nagumo kinetics

par={eps=eps bet=@[b] gam=gam Iu=@[i]}; // varied beta and current as calculated before

// Outputppmout when=out file="[0]/%04d.ppm" mode="w" // every 100 timesteps:r=[u] r0=umin r1=umax // value-discretizedg=[v] g0=vmin g1=vmax // output for subsequentb=[i] b0=0 b1=255; // visualization

k_print when=always file=stdout list={T; force; signal}; // to monitor work of the feedback looprecord when=end file=[0].rec when=end v0=0 v1=1; // ascii dump of all field values in the end of run

stop when=end;end;

Figure 11: A sample BeatBox script.

• Other sentences describe instances of devices compris-ing the ring; the first keyword in each sentence is thedevice type; other words describe the parameters de-termining the specifics of the work of this particularinstance.

The particular devices used in the script have the followingfunction:

• k_func computes, depending on the current value of theloop counter t, the “flag” global variables that controlwhich of the other devices will or will not work at thecurrent loop of the ring.

• k_func, another instance of the same device as before,but now it computes not the global variables, but thevalues of the field variables at every point of the grid,according to a given formula. This device works onlyonce, at the very first loop, and its function is to pro-duce initial conditions for the computations.

• reduce is a device that computes the value for a globalvariable based on current state of one or more of thefields represented in the computational grid. In the cur-rent example, this device emulates the work of a reg-istration electrode, which measures the maximal valueof the “transmembrane voltage” in a particular smallvolume in the grid. This measurement will be used asa signal in a feedback loop that is used to control theelectrical excitation using a putative low-voltage defib-rillation protocol.

• k_poincare is a device that implements the idea of aPoincare cross-section from the dynamical systems the-ory. In the present case, this device checks whether atthe current loop, the signal, measured by the previousreduce device, has crossed a given threshold value inthe required direction. If that has happened, then acertain flag (a global variable) is “raised” and the time(the value of the counter t) when this happened is re-membered in another global variable.

• k_func is yet another instance of this versatile device.This instance again works with global variables: it com-putes, using a given formula, the value of the variablethat defines the defibrillating electric field, dependingon the time that has passed since the event registeredby the k_poincare device.

• diff is the first of the devices which does “the actualcomputations” in the sense that it changes the fieldsrepresented in the computational grid. As could beguessed from its name, it computes the diffusion term,i.e. a value of the Laplacian of the field representedby one layer of the computational grid, and records theresult in the other layer of the grid.

• k_func — this instance of the device works is “local”,i.e. it works on the computational grid: computes theaction of the defibrillating electrical field (computed bythe previous instance of this device) onto the excitablecells.

• euler is a computational device that performs the timestep for the dynamic fields represented in the compu-tational grid, with account of the given kinetic model,the values of the diffusion term, and the action of thedefibrillating pulse.

• ppmout is one of the output devices. It only works once-in-so-many-steps (as explicitly defined in the script)and produces an output to a disk file in ppm format,where each byte represents a value of one element ofthe grid, from up to three selected layers of the grid,discretized to the 0..255 scale. This format is in fact animage format, could be converted to other popular andless space-consuming formats either by postprocessingor on-the-fly (not done in the current simple script).

• k_print is a more straightforward output device: eachtime it is called (in the current example, at the everystep), it adds to a disk file a plain-text record of thevalues of the global variables involved in the feed-backloop controlling the defibrillation stimuli.

• record is the last output device: it also outputs intoa plain text, but not an explicitly listed set of globalvariable, but a defined subset of the values in the com-putational grid. In this example, the devices works onlyon the last step of the computations, and the outputfile can be used for initial conditions if continuation ofthe present simulations is required.

• stop is the device whose function is to interrupt thecomputations and terminate the program. Naturallythis device must be present in the ring unless it is in-tended that the program run is to be interrupted by theoperator. In the presented example, the device workssimply when a certain number of time steps has beendone; but it can be controlled by any global variable,which may be computed by a k_func device dependingon how the simulations progresses, according to arbi-trary criteria as defined by the user.