Mathematical modelling for health systems research: a ...

RESEARCH ARTICLE Open Access

Mathematical modelling for health systemsresearch: a systematic review of systemdynamics and agent-based modelsRachel Cassidy1* , Neha S. Singh1, Pierre-Raphaël Schiratti2,3, Agnes Semwanga4, Peter Binyaruka5,Nkenda Sachingongu6, Chitalu Miriam Chama-Chiliba7, Zaid Chalabi8, Josephine Borghi1 and Karl Blanchet1

Abstract

Background: Mathematical modelling has been a vital research tool for exploring complex systems, most recentlyto aid understanding of health system functioning and optimisation. System dynamics models (SDM) and agent-based models (ABM) are two popular complementary methods, used to simulate macro- and micro-level healthsystem behaviour. This systematic review aims to collate, compare and summarise the application of both methodsin this field and to identify common healthcare settings and problems that have been modelled using SDM andABM.

Methods: We searched MEDLINE, EMBASE, Cochrane Library, MathSciNet, ACM Digital Library, HMIC, Econlit andGlobal Health databases to identify literature for this review. We described papers meeting the inclusion criteriausing descriptive statistics and narrative synthesis, and made comparisons between the identified SDM and ABMliterature.

Results: We identified 28 papers using SDM methods and 11 papers using ABM methods, one of which usedhybrid SDM-ABM to simulate health system behaviour. The majority of SDM, ABM and hybrid modelling paperssimulated health systems based in high income countries. Emergency and acute care, and elderly care and long-term care services were the most frequently simulated health system settings, modelling the impact of healthpolicies and interventions such as those targeting stretched and under resourced healthcare services, patient lengthof stay in healthcare facilities and undesirable patient outcomes.

Conclusions: Future work should now turn to modelling health systems in low- and middle-income countries toaid our understanding of health system functioning in these settings and allow stakeholders and researchers toassess the impact of policies or interventions before implementation. Hybrid modelling of health systems is stillrelatively novel but with increasing software developments and a growing demand to account for both complexsystem feedback and heterogeneous behaviour exhibited by those who access or deliver healthcare, we expect aboost in their use to model health systems.

Keywords: System dynamics, Agent-based, Hybrid, Health systems, Systematic review, Modelling

© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

* Correspondence: [email protected] of Global Health and Development, London School of Hygieneand Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH, UKFull list of author information is available at the end of the article

Cassidy et al. BMC Health Services Research (2019) 19:845 https://doi.org/10.1186/s12913-019-4627-7

http://crossmark.crossref.org/dialog/?doi=10.1186/s12913-019-4627-7&domain=pdf

http://orcid.org/0000-0002-4824-0260

http://creativecommons.org/licenses/by/4.0/

http://creativecommons.org/publicdomain/zero/1.0/

mailto:[email protected]

IntroductionHealth systems are complex adaptive systems [1]. As such,they are characterised by extraordinary complexity inrelationships among highly heterogeneous groups ofstakeholders and the processes they create [2]. Systemsphenomena of massive interdependencies, self-organisingand emergent behaviour, non-linearity, time lags, feedbackloops, path dependence and tipping points make healthsystem behaviour difficult and sometimes impossible topredict or manage [3]. Conventional reductionist ap-proaches using epidemiological and implementation re-search methods are inadequate for tackling the problemshealth systems pose [4]. It is increasingly recognised thathealth systems and policy research need a special set ofapproaches, methods and tools that derive from systemsthinking perspectives [5]. Health systems encompass amany tiered system providing services to local, district andnational populations, from community health centres totertiary hospitals. Attempting to evaluate the performanceof such a multi-faceted organisation presents a dauntingtask. Mathematical modelling, capable of simulating thebehaviour of complex systems, is therefore a vital researchtool to aid our understanding of health system functioningand optimisation.

System dynamics model (SDM)System dynamics models (SDM) and agent-based models(ABM) are the two most popular mathematical modellingmethods for evaluating complex systems; while SDM areused to study macro-level system behaviour such as themovement of resources or quantities in a system overtime, ABM capture micro-level system behaviour, such ashuman decision-making and heterogeneous interactionsbetween humans.While use of SDM began in business management [6, 7]

it now has wide spread application from engineering toeconomics, from environmental science to waste and re-cycling research [8–13]. A SDM simulates the movementof entities in a system, using differential equations tomodel over time changes to system state variables. A stockand flow diagram can be used to provide a visual repre-sentation of a SDM, describing the relationships betweensystem variables using stocks, rates and influencing fac-tors. The diagram can be interpreted as mimicking theflow of water in and out of a bath tub [7]; the rates controlhow much ‘water’ (some quantifiable entity, resource) canleave or enter a ‘bath tub’ (a stock, system variable) whichchanges over time depending on what constraints or con-ditions (e.g. environmental or operational) are placed onthe system. Often before the formulation of a stock andflow diagram, a causal loop diagram is constructed whichcan be thought of as a ‘mental model’ of the system [14],representing key dynamic hypotheses.

Agent-based model (ABM)Unlike SDM, ABM is a ground-up representation of asystem, simulating the changing states of individual‘agents’ in a system rather than the broad entities oraggregate behaviour modelled in SDM. Aggregate systembehaviour can however be inferred from ABM. Use ofABM to model system behaviour has been trans-disciplinary, with application in economics to ecology,from social sciences to engineering [15–19]. There canbe multiple types of agent modelled, each assigned theirown characteristics and pattern of behaviour [20, 21].Agents can learn from their own experiences, make deci-sions and perform actions based on set rules (e.g. heuris-tics), informed by their interactions with other agents,their own assigned attributes or based on their interactionwith the modelled environment [22]. The interactionsbetween agents can result in three levels of communica-tion between agents; one-to-one communication betweenagents, one-to-many communication between agents andone-to-location communication where an agent can influ-ence other agents contained in a particular location [22].

Why use SDM and ABM to model health systems?ABM and SDM, with their ability to simulate micro- andmacro-level behaviour, are complementary instrumentsfor examining the mechanisms in complex systems andare being recognised as crucial tools for exploratory ana-lysis. Their use in mapping health systems, for example,has steadily risen over the last three decades. ABM iswell-suited to explore systems with dynamic patient orhealth worker activity, a limitation of other differentialequation or event-based simulation tools [23–25]. Unlikediscrete-event simulation (DES) for example, which sim-ulates a queue of events and agent attributes over time[26], the agents modelled in ABM are decision makersrather than passive individuals. Closer to the true systemmodelled, ABM can also incorporate ongoing learningfrom events whereby patients can be influenced by their in-teractions with other patients or health workers and bytheir own personal experience with the health system [21].SDM has also been identified as a useful tool for simulatingfeedback and activity across the care continuum [27–30]and is highly adept at capturing changes to the system overtime [31]. This is not possible with certain ‘snapshot intime’ modelling approaches such as DES [32]. SDM is bestimplemented where the aim of the simulation is to exam-ine aggregate flows, trends and sub-system behaviour asopposed to intricate individual flows of activity which aremore suited to ABM or DES [33].There are also models that can accommodate two or

more types of simulation, known as hybrid models. Hybridmodels produce results closer to true system behaviour bydrawing on the strengths of one or more modelling methodswhile reducing the limitations associated with using a single

Cassidy et al. BMC Health Services Research (2019) 19:845 Page 2 of 24

simulation type [27]. The activity captured in suchmodels emulates the individual variability of patientsand health professionals while retaining the complex,aggregate behaviour exhibited in health systems.Health scientists and policy makers alike have recog-

nised the potential of using SDM and ABM to model allaspects of health systems in support of decision makingfrom emergency department (ED) optimisation [34] topolicies that support prevention or health promotion[35]. Before implementing or evaluating costly healthpolicy interventions or health service re-structuring inthe real world, modelling provides a relatively risk-freeand low budget method of examining the likely impactof potential health system policy changes. They allowthe simulation of ‘what if’ scenarios to optimise an inter-vention [36]. They can help identify sensitive parametersin the system that can impede the success of initiativesand point to possible spill-over effects of these initiativesto other departments, health workers or patients. Per-haps most important of all, these modelling methodsallow researchers to produce simulations, results and agraphical-user interface in relation to alternative policyoptions that are communicable to stakeholders in thehealth system [37], those responsible for implementingsystem-wide initiatives and changes.

Study aim and objectivesGiven the increasing amount of literature in this field,the main aim of the study was to examine and describethe use of SDM and ABM to model health systems. Thespecific objectives were as follows: (1) Determine thegeographical, and healthcare settings in which thesemethods have been used (2) Identify the purpose of theresearch, particularly the health policies or interventionstested (3) Evaluate the limitations of these methods andstudy validation, and (4) Compare the use of SDM andABM in health system research.Although microsimulation, DES and Markov models have

been widely used in disease health modelling and healtheconomic evaluation, our aim in this study was to reviewthe literature on mathematical methods which are used tomodel complex dynamic systems, SDM and ABM. Thesemodels represent two tenants of modelling: macroscopic(top-level) and microscopic (individual-level) approaches.Although microsimulation and DES are individual-basedmodels like ABM, individuals in ABM are “active agents” i.e.decision-makers rather than “passive agents” which are thenorm in microsimulation and DES models. Unlike Markovmodels which are essentially one-dimensional, unidirectionaland linear, SDM are multi-dimensional, nonlinear with feed-back mechanisms. We have therefore focussed our reviewon SDM and ABM because they are better suited to charac-terise the complexity of health systems. This study reviewsthe literature on the use of SDM and ABM in modelling

health systems, and identifies and compares the key charac-teristics of both modelling approaches in unwrapping thecomplexity of health systems. In identifying and summaris-ing this literature, this review will shed light on the types ofhealth system research questions that these methods can beused to explore, and what they add to more traditionalmethods of health system research. By providing an overoverview of how these models can be used within healthsystem research, this paper is also expected to encouragewider use and uptake of these methods by health system re-searchers and policy makers.

MethodsThe review was conducted in compliance with the Pre-ferred Reporting Items for Systematic Reviews andMeta-Analysis (PRISMA) statement [38].

Search strategy and information sourcesThe literature on ABM and SDM of health systems hasnot been confined to a single research discipline, makingit necessary to widen the systematic review to capturepeer-reviewed articles found in mathematical, computing,medicine and health databases. Accordingly, we searchedMEDLINE, EMBASE, Cochrane Library, MathSciNet,ACM Digital Library, HMIC, Econlit and Global Healthdatabases for literature. The search of health system litera-ture was narrowed to identify articles that were concernedwith modelling facility-based healthcare, services andrelated healthcare financing agreements which had beenexcluded or were not the focus of previous reviews[34, 35, 39–41]. The search criteria used for MED-LINE was as follows, with full search terms for eachdatabase and search terms used to locate SDM andABM literature found in Additional file 1:

(health system* OR health care OR healthcare ORhealth service* OR health polic* OR health facil* ORprimary care OR secondary care OR tertiary care ORhospital*).ab,ti. AND (agent-based OR agentbased).ab,ti. AND (model*).ab,ti.

In addition, the reference list of papers retained in thefinal stage of the screening process, and systematic re-views identified in the search, were reviewed for relevantliterature.

Data extraction and synthesisThe screening process for the review is given in Fig. 1(adapted from [38]). All search results were uploaded toMendeley reference software where duplicate entrieswere removed. The remaining records were screenedusing their titles and abstracts, removing entries basedon eligibility criteria given in Table 1. Post-abstract re-view, the full text of remaining articles was screened.


Papers retained in final stage of screening were scruti-nised, with data imported to Excel based on the follow-ing categories; publication date, geographical andhealthcare setting modelled, purpose of research inaddition to any policies or interventions tested, rationalefor modelling method and software platform, validationand limitations of model. The results were synthesisedusing descriptive statistics and analysis of paper contentthat were used to answer the objectives.The studies were first described by three characteristics:

publication date, geographical setting, and what aspect ofthe health system was modelled and why. These charac-teristics were chosen for the following reasons. Publicationdate (Fig. 2) allows us to examine the quantity of SDMand ABM studies over time. Geographical settings (Fig. 2,top) allows us to see which health systems have been stud-ied, as health systems in LMIC are very different from

those in developed countries. Studies are classified asmodelling health systems in high, upper middle, lowermiddle and low income countries as classified by TheWorld Bank based on economy, July 2018 [42]. Finally, weexamined which aspects of the health system have beenmodelled and the types of research/policy questions thatthe models were designed to address, to shed light on therange of potential applications of these models, and alsopotential gaps in their application to date.The analysis of paper content was split into three

sections; SDM use in health system research (including hy-brid SDM-DES), ABM use in health system research (in-cluding hybrid ABM-DES) and hybrid SDM-ABM use inhealth system research. The quality of selected studies willnot be presented as our aim was to compare and summar-ise the application of SDM and ABM in modelling healthsystems rather than a quality appraisal of studies.

Fig. 1 a Flow-chart for systematic review of SDMs and b ABMs of health systems (Database research discipline is identified by colour;mathematical and computing (red), medicine (blue) and health (green) databases). Adapted from PRISMA [38]

Table 1 Eligibility criteria for review

Criteria Inclusion Exclusion

Type of study/model Studies that describe the development andpresentation of SDM or ABM or hybrid model.

Poster presentations, conference abstracts, reviewpapers (reference list reviewed), commentaries,debate papers, papers that describe the qualitativedata used to inform a later developed model, papersthat only present conceptual SDM or ABM model,papers that present exclusively a DES model or othermodelling method.

Setting Facility-based healthcare or related policies/financing arrangements

Papers that primarily describe a disease/transmissionmodel or delivery of non-facility-based healthcare

Publication date Up to May 2019

Language English Other languages


ResultsStudy selectionThe search initially yielded 535 citations for ABM and996 citations for SDM of facility-based healthcare andservices (see Fig. 1). Post-full text screening 11 ABM and28 SDM papers were retained for analysis, six of whichutilised hybrid modelling methods. Three of the hybridmodelling papers integrated SDM with DES [43–45], twointegrated ABM with DES [24, 46] and one integratedSDM with ABM [47]. A summary table of selected papersis given in Table 2.

Descriptive statisticsPublication dateThe first SDM paper to model health systems was pub-lished in 1998 [56] whilst the first publication [66] uti-lising ABM came almost a decade later (Fig. 2). Wefound an increasing trend in publications for bothmodelling approaches, with 90.9% (10/11) and 71.4%(20/28) of all ABM and SDM articles, respectively, hav-ing been published in the last decade. The first hybridmodelling article was published in 2010 [43], usingSDM and DES to model the impact of an interventionto aid access to social care services for elderly patientsin Hampshire, England.

Geographical settingThe proportion of papers that modelled health systems inhigh, upper middle, lower middle and low income coun-tries is presented in Fig. 2. Eighteen (18/28) papers thatemployed SDM simulated health systems in high incomecountries including England [33, 36, 43, 45, 50, 54, 56, 57]and Canada [28, 51, 62]. Four SDM papers simulatedupper middle income country health systems, including

Turkey [52, 59] and China [64], with a nominal number ofpapers (5/28) focussing on lower middle or low incomecountries (West Bank and Gaza [48, 55], Indonesia [37],Afghanistan [30] and Uganda [60]). Almost all ABMpapers (9/11) modelled a high income country health sys-tem, including the US [20, 23, 25] and Austria [65]. Two(2/11) ABM papers described an upper-middle incomebased health system (Brazil [22, 67]). All six articles thatimplemented a hybrid SDM or ABM simulated health sys-tems based in high income countries, including Germany[44] and Poland [47].

Healthcare setting and purpose of researchThe healthcare settings modelled in the SDM, ABM andhybrid simulation papers are presented in Fig. 3. Health-care settings modelled using SDM included systems thatwere concerned with delivering emergency or acute care(11/28) [28, 31, 36, 45, 47, 50, 56–58, 61, 62], elderly orlong-term care services (LTC)(12/28) [28, 31, 36, 43–45,49–51, 54, 61, 62] and hospital waste management (4/28)[37, 48, 52, 55]. Twenty of the SDM papers selected in thisreview assessed the impact of health policy or interven-tions on the modelled system. Common policy targets in-cluded finding robust methods to relieve stretchedhealthcare services, ward occupancy and patient length ofstay [28, 31, 36, 43, 49, 50, 54, 58, 62], reducing the timeto patient admission [33, 53, 61], targeting undesirablepatient health outcomes [47, 58, 60, 63], optimising per-formance-based incentive health system policies [30, 59]and reducing the total cost of care [33, 54, 61]. Theremaining eight papers explored factors leading toundesirable emergency care system behaviour [56,57], simulating hospital waste management systemsand predicting future waste generation [37, 48, 55],

Fig. 2 Number of articles in the final review by year of publication and economic classification


Table 2 Summary of studies included at full paper review (SDM) and studies included at full paper review (ABM)

Paper/Year/Ref Purpose Sector of health system modelled Key results Software platform

System dynamics models (SDMs)

Al-Khatib (2016) [48] Assess the impact ofkey factors on thehospital wastemanagement systemand compare thefuture total wasteoutput betweenprivate, charitableand governmenthospitals.

• Model simulates hospitalwaste management inNablus, Palestine.

• Focus on three differenttypes of hospital (private,charitable and governmenthospitals).

• The amount of wastegenerated heavilydependent on the numberof beds.

• Waste treatment wasdependent on stafftraining and theenforcement of legislation.

• iThink.

Alonge (2017) [30] Explore effectiveimplementation structurefor improving healthsystem performancethrough pay-for-performance(P4P) initiative.

• The model is a genericrepresentation of the payfor performance initiative inprimary health facilities inAfghanistan.

• P4P initiative would likelyhave a beneficial impacton the volume and qualityof health services if correctlyimplemented.

• May prove ineffective if theimpact of gaming is notmitigated or if the methodfor distributing financialrewards are inadequate.

• MATLAB andSimulink.

Ansah (2014) [49] Assess the impact ofdifferent long-term care(LTC) capacity policieson uptake of acute care,demand for and utilisationof LTC services.

• Generic representation ofLTC utilisation and resourcesfor care and is not based orset in a particular health facility.

• Proactive adjustment ofLTC capacity stemmedthe number of acutecare visits but requireda modest increase in staff.

• Movement of health staff(through delayed trainingor from LTC to the acutecare sector) will impedethe success of this policy.

• Does not state.

Brailsford (2004) [50] To determine howemergency and ondemand care is currentlyconfigured and whatpolicies could alleviatepressure on the healthsystem.

• Entire healthcare systemthat provides emergencycentres etc) in Nottingham,England.

• Significant impact onelective hospital admissionsas emergency cases arecurrently prioritised.

• Redirecting certain elderlypatients to appropriateservices relieved pressureon emergency services.

• STELLA.

Brailsford (2010)a [43] Investigate how localauthorities such asHampshire CountyCouncil (HCC) canimprove access toservices and supportfor older people, inparticular assess thelong-term impact ofa new contact centrefor patients.

• HCC system for long-termcare, including a call centrethat older patients canaccess to receive advice orbe directed to appropriatecare.

• The number of patientswho contact the callcentre on a second occasion(having failed to makecontact the first time) wherethe health status of the patienthas now deteriorated, felldrastically after the introductionof two additional call handlers.

• SDM is Vensim,DES model isSimul8.

Cepoiu-Martin (2018) [51] To examine patienttransition from hometo supportive living (SL)or long term care (LTC)in persons with dementiaand discern policy impacton the deficit of nursesand health care assistants.

• The Alberta ContinuingCare System comprisingof home living, SL or LTCservices.

• Introducing benchmarks forhiring nurses and healthcare assistants in SL andLTC facilities will resultinitially in a greater deficitof staff but will stabilise theratio of health professionalsto patients in the long term.

• Does not state.

Chaerul (2008) [37] To determine keyfactors that impact themanagement of hospitalwaste and predict futurewaste output.

• The model describeshospital wastemanagement in theCity of Jakarta, Indonesia.

• Hospital waste disposalis impacted by thereluctance of a denselypopulated cityto allowfurther waste to bedumped in landfill sites.

• The simulation indicated thatexisting and new landfill siteswill be at full capacity by 2011and 2020, respectively.

• STELLA.


Table 2 Summary of studies included at full paper review (SDM) and studies included at full paper review (ABM) (Continued)


Ciplak (2012) [52] To predict futurehealthcare wasteproduction andoptimise themanagement ofhealthcare waste.

• Healthcare wastegeneration fromhealthcare facilities, thesingle healthcare wastetreatment facility andalternative waste treatmentfacilities in Istanbul, Turkey.

• Employing stringent wasteseparation strategies wouldrelieve the pressure on alreadyat capacity waste treatmentfacility in Istanbul.

• Up to 77% of healthcare wastecould be diverted to alternativetreatment technologies thatdo not require treatment atthe incineration facility.

• Vensim.

De Andrade (2014) [53] To examine the reasons fordelayed ST-segment elevationmyocardial infarction (STEMI)treatment and exploreinterventions that can speedup wait time in primary carefacilities.

• A primary care hospitaland a PercutaneousCoronary InterventionCentre (PCI) in Brazil.

• It was observed that 50%reduction in waiting timefor patients is possibleunder a combination ofinterventions targetingECG transmission and PCIcentre team feedbacktime and patient transferwaiting time.

• Vensim.

Desai (2008) [54] To forecast demand for olderpeople’s services and explorethe future impact of challengesthat accompany an ageingpopulation.

• Adult Services Departmentof Hampshire County Councilincluding 13 different types ofcare package that can be offeredby the funding and assessmentbody.

• Providing care packagesonly to critical patientsreduced the overall numberof patients receiving acutecare.

• Savings can be made byincreasing the number ofunqualified care workerswhich can be fed back intocare funding.

• STELLA.

Djanatliev (2012)b [47] Presenting the functionalityof the Prospective HealthTechnology Assessment(ProHTA) tool, which cansimulate the impact ofoptimised technologyprospectively beforephysical development.

• Mobile Stroke Unit (MSU) casestudy was simulated for Berlin,includes a generic hospitalwith emergency services wherepatients are taken by the MSU.

• In the simulationimplementing MSU,18.2% of patients receivedthrombolysis treatmentcompared with 10.6% inthe simulation without MSU.

• Fewer patients were alsofound to have developedsevere disability in thesimulation with MSU asa consequence of fasterimplemented treatment,reducing the long term costsfor rehabilitation and care.

• AnyLogic.

Eleyan (2013) [55] To predict general andmedical waste generationfor a complex hospitalwaste management system.

• Model simulates hospitalwaste management inthree hospitals based inJenin, Palestine.

• Increases in the amountof hospital waste areconsistent with bedoccupancy. Over the next20 years, the total amountof waste generated willrise as will the total costof treating hazardous waste.

• iThink.

Esensoy (2018) [28] Transformation of stroke careto implement best practice.

• The model describes sixsectors of Ontario healthcare system and thepatient flow between them.

• When stroke best practicepolicy has been implemented(compared to the base casescenario), there is a reductionin length of stay across allsectors.

• A reduction in bed utilisation wasalso observed with a 10 and 11.1%reduction in acute care and rehabsectors, respectively.

• Vensim.

Ghaffarzad. (2013) [32] To explore physician decisionmaking behind scheduledcaesarean delivery (CD),unplanned CD and vaginaldelivery (VD) and examinefactors that influenceprocedure variation.

• The model does not reflect aparticular hospital but isparameterised using patientinformation from hospitaldischarge databases in Florida.

• The biggest impact on physiciandelivery decision is from thedelayed effect of colleague pastexperience.

• Turning off all learning experiencesreduces physician delivery variationfor scheduled CD delivery from 6.5to 4.7%.

• Vensim.




Lane (1998) [56] Explore the factors that leadto delays in Accident andEmergency Departments (A&E)and to elective admissions.

• A&E department at majorinner-London teachinghospital coded in the studyas ‘St Dane’s’.

• Reduction in bed numbersincreases emergency admissionwaiting times and delays andcancellations to elective surgeryadmissions.

• Increases in demand push thesystem to breaking point, withpatients waiting hours to beadmitted and health workersat full capacity.

• Does not state.

Lane (2000) [57] The model depicts theperformance of Accidentand Emergency (A&E)at acute hospitals,investigating thesensitivity of waitingtimes to hospital bednumbers.

• A&E department at Inner-London teaching hospitalcoded in the study as ‘StDane’s’.

• Reducing bed capacityincreased the % of electivecancellations, negating theimpact on other performancemeasures.

• Deterioration of services isnot attributed to lack ofbed capacity but insufficientprovision of A&E doctors whoreach 100% utilisation.

• iThink.

Lattimer (2004) [36] To evaluate ‘front door’services of local emergencyand urgent care facilitiesand test proposals forsystem change.

• Entire healthcare systemthat provides emergencyor on demand care (GP,NHS Direct, Walk in centresetc) in Nottingham.

• Reducing emergencyadmissions from GP by 4%showed successive reductionin occupancy levels in A&E.

• Interventions to loweradmissions of patients over60 resulted in a 1% reductionper annum in bed occupancyover 5 years.

• STELLA.

Mahmoudia. (2017) [58] To explore the intended andunintended consequencesof Intensive Care Unit (ICU)resource and bed managementpolicies on patient mortality,emergency departments (ED)and general wards.

• Generic model of ICU, EDand general hospital wards.

• Whilst general ward admissioncontrol is not as effective atreducing ICU and ED occupancyrates, it outperforms other policieswith regards to reducing patientmortality, arguably the moreimportant ICU managementperformance measure.

• Does not state.

Meker (2015) [59] To describe performance-based payment systems(PBPS) in second-step publichospitals and the impact onprocess measures in hospitals.

• Second-step publichospitals in Turkey.

• With reduced performancepayments, physicians moveto the private sector decreasingstaff levels, reducing time spentwith patients leading to a dramaticdecrease of correct diagnosis andtreatment.

• Does not state.

Mielczarek (2016)a [44] To estimate the futuredemand for healthcare frompatients with cardiac disease.

• Future demand for cardiacdisease care in WroclawRegion, Poland.

• Older population (over 60) willgenerate increasing demands forcare, specifically the growth ofcardiac patients was observedas more intense in men thanwomen (increases of 34.4 and30.15% respectively).

• Does not state.

Rashwan (2015) [31] To explore the flow ofelderly patients throughthe Irish healthcare systemand anticipate the growingdemand for services overthe next 5 years.

• Generic emergency carefacility in Ireland and sixpossible discharge locations.

• Under increasing demand, acombination of all three policieswas necessary to significantlyreduce elderly frail patients’ lengthof stay in acute hospitals andreduce delayed discharge numbers.

• Does not state.

Semwanga (2016) [60] To capture the dynamics ofthe Ugandan health systemand evaluate what impactinterventions might have onneonatal care.

• Does not focus on onetype of health facility butincorporates different servicesand levels of care offered tothis group.

• Integrating community healtheducation, free delivery kits andmotorcycle coupons has thebiggest impact on reducingneonatal death.

• Interventions targetingsocioeconomic status had agreater impact on reducingneonatal mortality than thosetargeting service delivery.

• STELLA.




Taylor (2005) [33] To examine the impact ofshifting cardiac catheterization(CC) services from tertiary tosecondary level for low riskinvestigations and explorehow improvements could bemade to services.

• The CC service pathwaysat two English districtgeneral hospitals, referredto using the pseudonyms‘Veinbridge Hospital’ and‘Ribsley Hospital’.

• Significant and stableimprovements in service(reducing waiting listtime and overall costsof service) were achievedwith the implementationof strict referral guidelinesfor patients.

• STELLA.

Walker (2003) [61] To model patient flow fromfeeder hospitals to a sub acuteextended care hospital to showthe impact of local rules usedby the medical registrar(medical admitting officer).

• A single extended carefacility in Victoria (Australia)and patient flow fromfeeder hospitals.

• Using the local rule, the costof care exceeds the budgetby 6%. Without the local rule,costs were 3% under budget.

• The unprioritized list maintainswaiting lists at a level thateffectively short-circuits thefeeder hospital second localrule of moving high acuitypatients on to the wait listof the sub-acute hospital.

• iThink.

Wong (2010) [62] To evaluate if smoothingthe number of dischargesover the week relievesthe pressure on emergencydepartments (ED).

• Model describes a generalinternal medicine (GIM)program at a single tertiarycare teaching hospital inToronto, Canada.

• Both scenarios for ‘smoothedaverage case’ were similar,resulting in reduction ofGIM in ED by 27% and GIMin ED length of stay by 31%.

• For ‘every day is a weekday case’, larger reductionsobserved.

• Vensim.

Worni (2012) [63] To estimate what impacta policy to denyreimbursement of totalknee arthroplasty (TKA)patient fees will have onvenous thromboembolism(VTE) rates and anyunintentional consequences.

• The model simulates allpatients (9.7 million) in theUS who have symptomaticosteoarthritis, over 65 andhave Medicare insurance.

• Model output indicatesnew policy will result in3-fold decrease in VTErates. Fraction of those(in simulation with newpolicy) with bleedingcomplications is 6-foldhigher and 6-fold morepatients ineligible for TKAper year.

• Vensim.

Yu (2015) [64] To explore the drivingfactors for a high proportionof patients in China notseeking medical care (alsoknown as potential medicaldemand) and examinepossible interventions.

• Three main sub-systems;medical demand of patients,outpatients in hospitals andoutpatients in communityhealth systems (CHS). It doesnot describe a specific hospitalor CHS.

• An increase in the numberof CHS and decrease in thenumber of hospitals wasfound to induce the biggestdecrease in the number ofpatients not seeking care.

• Varying the price ofoutpatient care in hospitalsand CHS had minimalimpact on increasing thenumber of patients whoseek care.

• Vensim.

Zulkepli (2012)a [45] Present a case study usinghybrid modelling (SDM-DES),explore patient flow in anintegrated care system (IC)and the impact of patientadmission on healthprofessional stress level.

• Three main sub-systems;patient flow through criticalcare facility, patient flowthrough intermediate careassessment and motivationand stress levels of healthprofessionals.

• Due to high demand ofintermediate care servicesbut limited spaces bedblocking may occur, withan increase in patientadmissions leading toan increase to healthprofessional stress level.

• SDM is Vensim,DES model isSimul8.

Agent-based models (ABMs)

Alibrahim (2018) [23] To explore the effect ofpatient choice on thehealthcare market,specifically providers thatform accountable careorganisations (ACO).

• A generalised simulation ofpatient (Medicare beneficiary,over 65 years old who has orcan develop congestive heartfailure) choice of medicalprovider (hospital or primarycare physician facility) in theUnited States.

• Where providers wereallowed to opt out of ACOnetwork, they wereable to optimise theirown profits by notimplementing a diseasemanagement programme -this led to a reduction inthe overall quality of care,driving patients to attend

• AnyLogic.




alterative care facilitiesreducing the utilisationof that facility.

Djanatliev (2012)b [47] Presenting the functionalityof the Prospective HealthTechnology Assessment(ProHTA) tool, which cansimulate the impact ofoptimised technologyprospectively beforephysical development.

• Mobile Stroke Unit (MSU)case study was simulatedfor Berlin, includes a generichospital with emergencyservices where patients aretaken by the MSU.

• In the simulationimplementing MSU,18.2% of patients receivedthrombolysis treatmentcompared with 10.6% inthe simulation without MSU.

• Fewer patients were alsofound to have developedsevere disability in thesimulation with MSU as aconsequence of fasterimplemented treatment,reducing the long term costsfor rehabilitation and care.

• AnyLogic.

Einzinger (2013) [65] To create a tool capable ofcomparing reimbursementschemes in outpatient care.

• Compared differentreimbursement schemes forAustrian outpatient healthsector simulating the vastmajority of health insuredpersons in Austria.

• Creation of a tool that canbe used to compare healthcare reimbursement schemesin Austria.

• AnyLogic.

Hutzsch. (2008) [66] To determine which mix ofpatients should be admittedto specialised hospitals tooptimise resource utility andto consider the impact ofunplanned patient arrivals onthis process.

• Cardiothoracic surgery(CTS) and intensive careunit (ICU) at CatharinaHospital Eindhoven (CHE)in the Netherlands. CTSand ICU are broken downinto their respective unitssuch as the high care unitof CTS etc.

• An additional ward bedon the CTS ward decreasedthe frequency of sendingpre- and post- operativeadmissions to other wards bya factor of 3 with minimal cost.

• The brute force optimiserindicated that the number ofIC high care beds should beincreased and number of ICbeds decreased to gainoptimum throughput ofpatients in simulation.

• Java.

Huynh (2012) [20] To assess the impact ofredesigning medicationadministration process (MAP)workflow for registered nursesto improve medicationadministration safety.

• A local (anonymous) medicalcentre where nurses areadministering medication topatients.

• Implementing a protocolfor the order of MAP tasksto be performed improvedthe amount of time spentperforming tasks.

• When registered nursesperformed tasks in the mostfrequently observed order(in the pilot study) thisimproved MAP task times.

• Netlogo.

Kittipitta. (2016)c [24] To examine patient flow inan outpatient clinic of anorthopedic department andexplore interventions thatcan improve clinical servicesto reduce patient waiting times.

• Orthopedic department atunidentified communityhospital.

• Average waiting time foroutpatient appointmentsfell by 32.03% under thenew management policy.

• AnyLogic.

Liu (2014) [21] To develop a tool that canbe used as a decisionsupport system for managersof emergency departments(ED) to assess risk, allocationof resources and identifyweakness in emergencycare service.

• ED at Hospital of Sabadell(University tertiary levelhospital in Barcelona, Spain).The Department is split intosections A (critical patients)and B (least critical patients).

• A tool that can be usedsimulate the behaviourof agents in ED.

• Netlogo.

Liu (2016) [25] To explore how accountablecare organisations (ACO) canimpact payers, healthcareproviders and patients undera shared savings paymentmodel for congestive heartfailure (CHF) and achieveoptimal outcomes.

• A generalised simulation ofpatients (Medicare beneficiary,over 65 years old who has orcan develop congestive heartfailure) seeking care (hospital orprimary care physician facility)in Unites States.

• Quality orientatedproviders yielded higherfinancial returns to thepayer agent (which werethen shared betweenproviders) than those thatwere profit-orientated.

• AnyLogic.


estimating future demand for cardiac care [44], ex-ploring the impact of patient admission on healthprofessionals stress level in an integrated care system[45], and variation in physician decision-making [32].ABM papers modelled systems focussed on delivering

emergency or acute care (4/11) [21, 22, 47, 67] and

accountable care organisations (ACO) or health insur-ance reimbursement schemes (3/11) [23, 25, 65]. Nine ofthe ABM papers assessed the impact of health policy orinterventions on the modelled system. Common policytargets included decreasing the time agents spent per-forming tasks, waiting for a service or residing in parts



Viana (2018)c [46] To examine and improvepatient flow through apregnancy outpatientclinic in light of theuncertainty in demandfor services from overduepatients.

• Overdue pregnancyoutpatient clinic,pregnancy clinic andpostnatal clinic atAkershus UniversityHospital, Norway.

• As expected increasingthe number of midwivesin the clinic reducesresource utilisation butcombined with an increasein demand led to an increasein doctor utilisation.

• Midwives act as a buffer(or bottleneck) to patientsseeing doctors.

• AnyLogic.

Yousefi (2017) [67] To apply group decision-making techniques foremergency department(ED) resource allocationand determine whetherthis approach improvesperformance indicators.

• A generic ED informedfrom the literature.

• Group-decision makingbetween agents in theED resulted in on averagea 12.7% decrease in totalwaiting time and 14.4%decrease in the numberof patients who leftwithout being seen.

• Netlogo.

Yousefi (2018) [22] To examine the behaviourof patients who leavepublic hospital emergencydepartments (ED) withoutbeing seen and the impactof preventative policies.

• ED at Hospital RisoletaTolentino Neves, a tertiaryhospital in Minas Gerais,Brazil.

• After applying preventativepolicies, average 42.14%reduction in the numberof patients leaving withoutbeing seen in the ED andaverage 6.05% reduction inpatient length of stay in EDwas observed, with mosteffective policy to fast-trackless critical patients after triage.

• NetLogo .

Note: aArticles implemented SDM-DES hybrid modellingbArticles implemented SDM-ABM hybrid modellingcArticles implemented ABM-DES hybrid modelling

Fig. 3 The health system sector locations modelled in the SDM, ABM and hybrid modelling literature. Long-term care (LTC); Accountable careorganisation (ACO); Maternal, newborn and child health (MNCH)


of the system [20, 22, 24, 67], reducing undesirable pa-tient outcomes [23, 25, 47, 67], reducing the number ofpatients who left a health facility without being seen by aphysician [22, 67] and optimising resource utility (bedsand healthcare staff) [46, 66, 67]. The remaining twopapers described simulation tools capable of comparinghealth insurance reimbursement schemes [65] and asses-sing risk, allocation of resources and identifying weak-nesses in emergency care services [21].Papers that utilised hybrid simulation, combining the

strengths of two modelling approaches to capture de-tailed individual variability, agent-decision making andpatient flow, modelled systems focussed on deliveringelderly care or LTC services [43–45] and emergency oracute care [45, 47]. Four of the hybrid simulation papersassessed the impact of policy or intervention on themodelled system. Policy targets included improving ac-cess to social support and care services [43], reducingundesirable patient outcomes [47], decreasing patientwaiting time to be seen by a physician [24] and improv-ing patient flow through the system by optimising re-source allocation [46]. The remaining two papers usedhybrid simulation to estimate the future demand forhealth care from patients with cardiac disease [44] andmodel patient flow through an integrated care system toestimate impact of patient admission on health care pro-fessionals wellbeing [45].

SDM use in health systems research (including hybridSDM-DES)Rationale for using modelGaining a holistic system perspective to facilitate theinvestigation of delays and bottlenecks in health facilityprocesses, exploring counter-intuitive behaviour andmonitoring inter-connected processes between sub-systems was cited frequently as reasons for using SDMto model health systems [28, 36, 37, 48, 56]. SDM wasalso described as a useful tool for predicting futurehealth system behaviour and demand for care services,essential for health resource and capacity planning [48,60]. Configuration of the model was not limited by dataavailability [28, 52, 64] and could integrate data fromvarious sources when required [51].SDM was described as a tool for health policy explor-

ation and optimising system interventions [33, 36, 51,54, 58, 64], useful for establishing clinical and financialramifications on multiple groups (such as patients andhealth care providers) [63], identifying policy resistanceor unintended system consequences [59, 61] and quanti-fying the impact of change to the health system beforereal world implementation [62]. The modelling platformalso provided health professionals, stakeholders and de-cision makers with an accessible visual learning

environment that enabled engagement with experts ne-cessary for model conception and validation [48, 50, 55,57]. The model interface could be utilised by decisionmakers to develop and test alternative policies in a ‘real-world’ framework that strengthened their understandingof system-wide policy impact [31, 49, 58, 61].SDM-DES hybrid models enabled retention of deter-

ministic and stochastic system variability and preserva-tion of unique and valuable features of both methods[44], capable of describing the flow of entities through asystem and rapid insight without the need for large datacollection [43], while simulating individual variabilityand detailed interactions that influence system behaviour[43]. SDM-DES offered dual model functionality [44]vital for simulating human-centric activity [45], reducingthe practical limitations that come with using eitherSDM or DES to model health systems such as attempt-ing to use SDM to model elements which have non-aggregated values (e.g. patient arrival time) [45] which isbetter suited for DES.

Healthcare settingSixteen papers that utilised SDM modelled systems thatwere concerned with the delivery of emergency or acutecare, or elderly care or LTC services.Ten of the reviewed papers primarily modelled sectors of

the health system that delivered emergency or acute care1,2.Brailsford et al. [50], Lane et al. [56], Lane et al. [57] andLattimer et al. [36] simulated the delivery of emergencycare in English cities, specifically in Nottingham andLondon. Brailsford et al. [50] and Lattimer et al. [36] cre-ated models that replicated the entire emergency caresystem for the city of Nottingham, from primary care (i.e.General Practice surgeries) to secondary care (i.e. hospitaladmissions wards), to aid understanding of how emergencycare was delivered and how the system would need to adaptto increasing demand. Lane et al. [56] and Lane et al. [57]modelled the behaviour of an ED in an inner-Londonteaching hospital, exploring the knock on effects of ED per-formance to hospital ward occupancy and elective admis-sions. Esensoy et al. [28] and Wong et al. [62] bothmodelled emergency care in Canada, Esensoy et al. [28] fo-cussing on six sectors of the Ontario health system thatcared for stroke patients while Wong et al. [62] simulatedthe impact of delayed transfer of General Internal Medicinepatients on ED occupancy. Rashwan et al. [31], Walker

2The single SDM-ABM paper that modelled the delivery of emergencyor acute care is discussed in section ‘SDM-ABM use in health systemresearch’.

1One of the elderly or LTC services papers also modelled emergencyor acute care but it was not the primary focus and is therefore notdiscussed here.


et al. [61] and Mahmoudian-Dehkordi et al. [58] modelledpatient flow through a generic emergency care facility withsix possible discharge locations in Ireland, a sub-acute ex-tended care hospital with patient flow from feeder facilitiesin Australia and an intensive care unit, ED and generalwards in a generic facility.Five of the SDM papers primarily simulated the behav-

iour of LTC facilities or care services for elderly patients3.Ansah et al. [49] modelled the demand and supply of gen-eral LTC services in Singapore with specific focus on theneed for LTC and acute health care professionals. Desaiet al. [54] developed a SDM that investigated future de-mand of care services for older people in Hampshire, Eng-land which simulated patient flow through adult socialcare services offering 13 different care packages. In model-ling complex care service demand, Cepoiu-Martin et al.[51] explored patient flow within the Alberta continuingcare system in Canada which offered supportive living andLTC services for patients with dementia. Brailsford et al.[43] used a hybrid SDM-DES model to investigate howlocal authorities could improve access to services and sup-port for older people, in particular the long term impactof a new contact centre for patients. The SDM replicatedthe whole system for long term care, simulating the futuredemography and demand for care services and the nestedDES model simulated the operational issues and staffingof the call centre in anticipation of growing demand forservices. Zulkepli et al. [45] also used SDM-DES to modelthe behaviour of an integrated care system in the UK,modelling patient flow (DES) and intangible variables(SDM) related to health professionals such as motivationand stress levels.

Policy impact evaluation/testingTwenty papers that utilised SDM tested the impact ofpolicy or interventions on key health system perform-ance or service indicators. The intended target of thesepolicies ranged from relieving strained and underresourced healthcare services, decreasing healthcarecosts to reducing patient mortality rates.Ansah et al. [49], Brailsford et al. [50] and Desai et al.

[54] aimed to reduce occupancy in acute or emergencycare departments through policies that targeted elderlyutilisation of these services. While demand for LTC ser-vices is expected to exponentially increase in Singapore,focus has been placed on expanding the acute care sec-tor. Ansah et al. [49] simulated various LTC service ex-pansion policies (static ‘current’ policy, slow adjustment,quick adjustment, proactive adjustment) and identifiedthat proactive expansion of LTC services stemmed the

number of acute care visits by elderly patients over timeand required only a modest increase in the number ofhealth professionals when compared with other policies.In Brailsford et al. [50] simulation of the entire emer-gency care system for Nottingham, England, policy test-ing indicated that while the emergency care system isoperating near full capacity, yearly total occupancy ofhospital beds could be reduced by re-directing emer-gency admissions from patients over 60 years of age(who make up around half of all admissions) to moreappropriate services, such as those offered by commu-nity care facilities. To explore challenges that accompanyproviding care for an ageing population subject tobudget restraints, Desai et al. [54] simulated the deliveryand demand for social care services in Hampshire over aprojected 5 year period. In offering care packages to onlycritical need clients and encouraging extra care servicesat home rather than offering residential care, the num-ber of patients accessing acute care services reducedover the observed period.Desai et al. [54], in addition to Taylor et al. [33] and

Walker et al. [61], also examined policies that could re-duce the total cost of care. Increasing the proportion ofhired unqualified care workers (over qualified careworkers who are employed at a higher cost rate) resultedin savings which could be fed back into care funding, al-though Desai et al. [54] remarked on the legal and prac-tical limitations to this policy. Taylor et al. [33]examined the impact of shifting cardiac catheterizationservices from tertiary to secondary level hospitals forlow risk investigations and explored how improvementscould be made to services. Significant and stable im-provements in service, including reduced waiting list andoverall cost of service, were achieved with the imple-mentation of strict (appropriate referral) guidelines foradmitting patients. Walker et al. [61] modelled patientflow from feeder hospitals to a single sub-acute extendedcare facility in Victoria, Australia, to assess the impact oflocal rules used by the medical registrar for admission.The local admission policy which prioritised admissionsfrom patients under the care of private doctors pushedthe total cost of care over the facility budget by 6%whereas employing no prioritisation rule reduced thetotal cost of care to 3% under budget.Semwanga et al. [60], Mahmoudian-Dehkordi et al. [58]

and Worni et al. [63] evaluated the impact of health policyon undesirable patient outcomes (mortality and post-treatment complication rates). Semwanga et al. [60] testedthe effectiveness of policies designed to promote maternaland neonatal care in Uganda, established from the litera-ture. Policies that enabled service uptake, such as commu-nity health education, free delivery kits and motorcyclecoupons were significant in reducing neonatal death overthe simulated period. Mahmoudian-Dehkordi et al. [58]

3Six of the emergency or acute care review papers and one of thecardiology care papers also modelled elderly or LTC services but it wasnot the primary focus and are therefore not discussed here.


explored the intended and unintended consequences ofintensive care unit resource and bed management policieson system performance indicators, including patient mor-tality. During a simulated crisis scenario, prioritising in-tensive care unit patient admission to general wards overemergency admissions was found to be the most effectivepolicy in reducing total hospital mortality. Worni et al.[63] estimated the impact of a policy to reduce venousthromboembolism rates post-total knee arthroplasty sur-gery and identified unintentional consequences of thestrategy. The policy prevented the reimbursement of pa-tient care fees in the event that a patient was not takingthe recommended prophylaxis medication and conse-quently develops venous thromboembolism. Simulationresults indicated a positive 3-fold decrease in venousthromboembolism rates but an unintended 6-fold increasein the number of patients who develop bleeding complica-tions as a result of compulsory prophylaxis treatment.

Validation (including sensitivity analysis)Statistically-based models are usually used in quantita-tive data rich environments where model parameters areestimated through maximum likelihood or least-squaresestimation methods. Bayesian methods can also be usedto compare alternative statistical model structures.SDMs and ABMs on the other hand are not fitted todata observations in the traditional statistical sense. Thedata are used to inform model development. Both quan-titative data and qualitative data (e.g. from interviews)can be used to inform the structure of the model andthe parameters of the model. Furthermore, model struc-ture and parameter values can also be elicited from ex-pert opinion. This means that the nature of validation ofABMs and SDMs requires more scrutiny than that ofother types of models.With increasing complexity of such models, and to

strengthen confidence in their use particularly for de-cision support, models are often subjected to sensitiv-ity analysis and validation tests. Twenty-two papersthat utilised SDM undertook model validation, themajority having performed behavioural validity tests(see Additional file 2 for details of validation methodsfor each model). Key model output such as bed occu-pancy [36, 50], department length of stay [62] andnumber of department discharges [31] were comparedwith real system performance data from hospitals [32,33, 36, 48, 50, 54, 58, 59, 61, 62], local councils [54],nationally reported figs [31, 64]. as well being reviewedby experts [57, 60] as realistic. Others performed morestructure orientated validity tests. Model conception[28, 60], development [30, 36, 50, 53, 54, 57, 62] andformulation [54, 56, 59] were validated by a variety of ex-perts including health professionals [47, 53, 54, 57, 59, 62],community groups [56] and leaders [60], steering

committees [36], hospital and care representatives [50, 56,59], patient groups [60] and healthcare policy makers [60].Further tests for structural validity included checkingmodel behaviour when subjected to extreme conditions orextreme values of parameters [30, 31, 52, 57, 59, 60, 64],model dimensional consistency [31, 52, 57, 59, 60], modelboundary adequacy [31] and mass balance [54] and inte-gration error checks [31, 52]. Sensitivity analysis was per-formed to assess how sensitive model output was tochanges in key parameters [49, 51, 57, 60, 64], to test theimpact of parameters that had been based on expertopinion on model output [28] and varying key systemparameters to test the robustness and effectiveness of pol-icies [28, 30, 52, 53, 58] (on the assumption of imperfectpolicy implementation [28]).

Limitations of researchMost of the model limitations reported were concernedwith missing parameters, feedback or inability to simulateall possible future health system innovations. Mielczareket al. [44], Cepoiu-Martin et al. [51], Ansah et al. [49] andRashwan et al. [31] did not take into account how futureimprovements in technology or service delivery may haveimpacted results, such as the possibility of new treatmentimproving patient health outcomes [51] and how this couldimpact the future utilisation of acute care services [49].Walker et al. [61] and Alonge et al. [30] described how themodels may not simulate all possible actions or interactionsthat occurred in the real system, such as all proactive ac-tions taken by hospital managers to achieve budget targets[61] or all unintended consequences of a policy on the sys-tem [30]. De Andrade et al. [53] and Rashwan et al. [31]discussed the reality of model boundaries, that SDMs can-not encapsulate all health sub-sector behaviour and spill-over effects. Although these have been listed here as limita-tions, not accounting for possible future improvements inhealthcare service or not simulating all possible actions inthe modelled system did not prevent authors from fulfillingstudy objectives. When developing a SDM, it is not possibleto account for all possible spill-over effects to other health-care departments and this should not be attempted; modelboundaries are set to only include variables and feedbackthat are pertinent to exploring the defined problem.Simplification of model parameters was another com-

mon limitation. Wong et al. [62] stated that this would re-sult in some model behaviour not holding in the realsystem, such as using weekly hospital admission and dis-charge averages in place of hourly rates due to the hospitalrecording aggregated data. This aggregation of model pa-rameters may not have reflected real system complexity;Eleyan et al. [55] did not differentiate between service leveland type of hospital when modelling health care wasteproduction (described as future work) and Worni et al.[63] refrained from stratifying post-surgery complications


by severity, potentially combining lethal and less harm-ful complications within the same stock (although thisdid not detract from the study conclusion that the rate ofcomplications would increase as a result of the tested policy).Data availability, lack of costing analysis and short time

horizons were also considered credible limitations. Modelsthat had been calibrated with real data were at risk ofusing datasets that contained measurement errors or in-complete datasets lacking information required to informmodel structure or feedback [32]. Routine facility data re-quired for model conception and formulation was unavail-able which restricted the replication of facility behaviourin the model [36] and restricted validation of model be-haviour [59], although it should be noted that this is onlyone method among many for SDM validation and the au-thor was able to use other sources of data for this purpose.Lack of costing or cost effectiveness analysis when testingpolicies [60], particularly policies that required significantinvestment or capacity expansion [58], limited discussionon their feasibility in the real system. Models that simu-lated events over short time scales did not evaluate longterm patient outcomes [33] or the long term effects of fa-cility policies on certain groups of patient [57].

ABM use in health system research (including hybridABM-DES)Rationale for using modelThe model’s ability to closely replicate human behaviourthat exists in the real system was frequently cited [20–22, 25, 66], providing a deeper understanding of multipleagent decision-making [23, 67], agent networks [25] andinteractions [21, 22]. The modelling method was de-scribed as providing a flexible framework capable of con-veying intricate system structures [20], wheresimulations captured agent capacity for learning andadaptive behaviour [20, 25] and could incorporate sto-chastic processes that mimicked agent transition be-tween states [25]. ABM took advantage of key individuallevel agent data [25] and integrated information fromvarious sources including demographic, epidemiologicaland health service data [65]. The visualisation of systemsand interface available with ABM software packages fa-cilitated stakeholder understanding of how tested pol-icies could impact financial and patient health outcomes[23], particularly those experts in the health industrywith minimal modelling experience [67].Integrating DES and ABM within a single model en-

sured an intelligent and flexible approach for simulatingcomplex systems, such as the outpatient clinic describedin Kittipittayakorn et al. [24]. The hybrid model cap-tured both orthopaedic patient flow and agent decision-making that enabled identification of health care bottle-necks and optimum resource allocation [24].

Healthcare settingSeven papers that utilised ABM modelled systems thatwere either concerned with delivering emergency or acutecare2, ACOs or health insurance reimbursement schemes.Liu et al. [21] and Yousefi et al. [22] modelled behav-

iour in EDs in Spanish and Brazilian tertiary hospitals.Liu et al. [21] simulated the behaviour of eleven keyagents in the ED including patients, admission staff,doctors, triage nurses and auxiliary staff. Patients wereadmitted to the ED and triaged before tests were re-quested and a diagnosis issued. Over time, agent stateschanged based on their interaction with other agentssuch as when a doctor decided upon a course of actionfor a patient (sending the patient home, to another ward,or continue with diagnosis and treatment). For furtherdetails of agent type and model rules for each paper, seeAdditional file 3.Yousefi et al. [22] modelled the activities of patients,

doctors, nurses and receptionists in a ED. Agents couldcommunicate with each other, to a group of other agentsor could send a message to an area of the ED whereother agents reside. They made decisions based on theseinteractions and the information available to them at thetime. The main focus of the simulation was on patientswho left the ED without being seen by a physician; pa-tients decided whether to leave the ED based on a ‘toler-ance’ time extracted from the literature, which changedbased on their interaction with other agents. In an add-itional paper, Yousefi et al. [67] simulated decision-making by patients, doctors, nurses and lab technicianswithin a generic ED informed from the literature. Groupdecision-making was employed, whereby facility staffcould interact with each other and reach a common so-lution for improving the efficacy of the department suchas re-allocating staff where needed. Yousefi et al. [67],Yousefi et al. [22] and Liu et al. [21] each used a finitestate machine (a computational model which describesan entity that can be in one of a finite number of states)to model interactions between agents and their states.Liu et al. [25] and Alibrahim et al. [23] modelled the

behaviour of patients, health providers and payers usingseries of conditional probabilities, where health providershad participated in an ACO in the United States. Liuet al. [25] presented a model where health providerswithin an ACO network worked together to reduce con-gestive heart failure patient healthcare costs and wereconsequently rewarded a portion of the savings from thepayer agent (hypothetically, the Centers for Medicareand Medicaid Services). Patients were Medicare benefi-ciaries over the age of 65 who developed diabetes, hyper-tension and/or congestive heart failure and sought carewithin the network of health providers formed of threehospitals and 15 primary care physician clinics. Alibra-him et al. [23] adapted Liu et al. [25] ACO network


model to allow patients to bypass their nearest medicalprovider in favour of an alternative provider. The deci-sion for a patient to bypass their nearest health centrewas influenced by patient characteristics, provider char-acteristics and the geographical distance between healthproviders. Providers were also given a choice on whetherto participate in an ACO network, where they wouldthen need to implement a comprehensive congestiveheart failure disease management programme.Einzinger et al. [65] created a tool that could be used

to compare different health insurance reimbursementschemes in the Austrian health sector. The ABM utilisedanonymous routine data from practically all personswith health insurance in Austria, pertaining to medicalservices accessed in the outpatient sector. In the simula-tion, patients developed a chronic medical issue (such ascoronary heart disease) that required medical care andled to the patient conducting a search of medical pro-viders through the health market. The patient thenaccessed care at their chosen provider where the reim-bursement system, notified of the event via a genericinterface, reimbursed the medical provider for patientscare.

Policy impact evaluation/testingNine papers tested the impact of policy on key healthsystem performance or service indicators. The intendedtarget of these policies ranged from decreasing patientlength of stay, to reducing the number of patients wholeave without being seen by a physician to reducing pa-tient mortality and hospitalisation rates.Huynh et al. [20], Yousefi et al. [22], Yousefi et al. [67]

and Kittipittayakorn et al. [24] tested policies to reducethe time agents spent performing tasks, waiting for a ser-vice or residing in parts of the system. Huynh et al. [20]modelled the medication administration workflow forregistered nurses at an anonymous medical centre in theUnited States and simulated changes to the workflow toimprove medication administration safety. Two policieswere tested; establishing a rigid order for tasks to be per-formed and for registered nurses to perform tasks in themost frequently observed order (observed in a real med-ical centre) to see if this improved the average amountof time spent on tasks. Yousefi et al. [67] modelled theeffects of group decision-making in ED compared withthe standard approach for resource allocation (where asingle supervisor allocates resources) to assess whichpolicy resulted in improved ED performance. Turning‘on’ group decision-making and starting the simulationwith a higher number of triage staff and receptionists re-sulted in the largest reduction of average patient lengthof stay and number of patients who left without beingseen. This last performance indicator was the subject ofan additional paper [22], with focus on patient-to-

patient interactions and how this impacted their decisionto leave the ED before being seen by a physician. Fourpolicies adapted from case studies were simulated toreduce the number of patients leaving the ED withoutbeing seen and average patient length of stay. The policyof fast-tracking patients who were not acutely unwellduring triage performed well as opposed to baseline,where acutely ill patients were always given priority.Kittipittayakorn et al. [24] used ABM-DES to identifyoptimal scheduling for appointments in an orthopaedicoutpatient clinic, with average patient waiting time fall-ing by 32% under the tested policy.Liu et al. [25], Alibrahim et al. [23] and Yousefi et al.

[67] tested the impact of health policy on undesirablepatient outcomes (patient mortality and hospitalisationrates). Liu et al. [25] modelled health care providers whooperated within an ACO network and outside of the net-work and compared patient outcomes. Providers whooperated within the ACO network worked together toreduce congestive heart failure patient healthcare costsand were then rewarded with a portion of the savings.As part of their membership, providers implementedevidence-based interventions for patients, includingcomprehensive discharge planning with post-dischargefollow-up; this intervention was identified in the litera-ture as key to reducing congestive heart failure patienthospitalisation and mortality, leading to a reduction inpatient care fees without compromising the quality ofcare. The ACO network performed well, with a 10% re-duction observed in hospitalisation compared with thestandard care network. In another study [23] six scenar-ios were simulated with combinations of patient bypasscapability (turned “on” or “off”) and provider participa-tion in the ACO network (no ACO present, optionalparticipation in ACO or compulsory participation inACO). Provider participation in the ACO, in agreementwith Liu et al. [25], led to reduced mortality and con-gestive heart failure patient hospitalisation, with patientbypass capability marginally increasing provider ACOparticipation. Yousefi et al. [67] also modelled the im-pact of group decision-making in ED on the number ofpatient deaths and number of wrong discharges i.e. pa-tients sent to the wrong sector for care after triage andare then discharged before receiving correct treatment.

Validation (including sensitivity analysis)Nine of the 11 papers that utilised ABM undertookmodel validation, consisting almost exclusively of behav-ioural validity tests. Model output, such as patient lengthof stay and mortality rates, was reviewed by health pro-fessionals [46, 66] and compared with data extractedfrom pilot studies [20], health facilities (historical) [22,24, 46, 65, 66], national health surveys [65] and relevantliterature [23, 25]. Papers presented the results of tests


to determine the equivalence of variance [20] and differ-ence in mean [20, 24] between model output and realdata. Structural validity tests included extreme conditiontesting [23, 46] and engaging health care experts to en-sure the accuracy of model framework [22, 47]. Sensitiv-ity analysis was performed to determine how variationsor uncertainty in key parameters (particularly where theyhad not been derived from historical or care data [65])affected model outcomes [23, 25].

Limitations of researchThe majority of model limitations reported were con-cerned the use or availability of real system or case data.Huynh et al. [20], Yousefi et al. [67] and Liu et al. [25]formulated their models using data that was obtainable,such as limited sample data extracted from a pilot study[20], national average trends [25] and data from previousstudies [67]. Yousefi et al. [22] case study dataset did notcontain key system feedback, such as the tolerance timeof patients waiting to be seen by a physician in the ED,although authors were able to extract this data from acomparable study identified in the literature.Missing model feedback or parameters, strict model

boundaries and simplification of system elements were alsoconsidered limitations. Huynh et al. [20], Hutzschenreuteret al. [66] and Einzinger et al. [65] did not model all therealistic complexities of their system, such as all possibleinterruptions to tasks that occur in patient care units [20],patient satisfaction of admission processes [66] (which willbe addressed in future work), how treatment influences thecourse of disease or that morbid patients are at higher riskof developing co-morbidity than healthier patients, whichwould affect the service needs and consumption needs ofthe patient [65]. To improve the accuracy of the model,Huynh et al. stated that further research is taking place toobtain real, clinical data (as opposed to clinical simulationlab results) to assess the impact of interruptions on work-flow. Liu et al.’s [21] model boundary did not include otherhospital units that may have been affected by ED behaviourand they identify this as future work, for example to includehospital wards that are affected by ED behaviour. Alibrahimet al. [23] and Einzinger et al. [65] made simplifications tothe health providers and networks that were modelled, suchas assuming equal geographical distances and identical careservices between health providers in observed networks[23], limiting the number of factors that influenced a pa-tients decision to bypass their nearest health provider [65]and not simulating changes to health provider behaviourbased on service utilisation or reimbursement scheme inplace [23]. Alibrahim et al. [23] noted that although themodel was constrained by such assumptions, the focus offuture work would be to improve the capability of themodel to accurately study the impact of patient choice oneconomic, health and health provider outcomes.

SDM-ABM use in health system researchA single paper used hybrid SDM-ABM to model healthsystem behaviour. Djanatliev et al. [47] developed a toolthat could be used to assess the impact of new healthtechnology on performance indicators such as patienthealth and projected cost of care. A modelling methodthat could reproduce detailed, high granularity system ele-ments in addition to abstract, aggregate health system var-iables was sought and a hybrid SDM-ABM was selected.The tool nested an agent-based human decision-makingmodule (regarding healthcare choices) within a system dy-namics environment, simulating macro-level behavioursuch as health care financing and population dynamics. Acase study was presented to show the potential impact ofMobile Stroke Units (MSU) on patient morbidity inBerlin, where stroke diagnosis and therapy could be initi-ated quickly as opposed to standard care. The modelstructure was deemed credible after evaluation by experts,including doctors and health economists.

Comparison of SDM and ABM papersThe similarities and differences among the SDM and ABMbody of literature are described in this section and shownin Table 3. A high proportion of papers across both model-ling methods simulated systems that were concerned withemergency or acute care. A high number of SDM papers(11/28) simulated patient flow and pathways throughemergency care [28, 31, 36, 45, 47, 50, 56–58, 61, 62] witha subset evaluating the impact of policies that relievedpressure on at capacity ED’s [28, 36, 50, 58, 62]. ABMpapers simulated micro-level behaviour associated withemergency care, such as health professional and patientbehaviour in EDs and what impact agent interactions haveon actions taken over time [21, 22, 47, 67]. ACOs andhealth insurance reimbursement schemes, a commonmodelled healthcare setting among the ABM papers [23,25, 65] was the focus of a single SDM paper [63] whilehealth care waste management, a popular healthcare set-ting for SDM application [37, 48, 52, 55] was entirelyabsent among the selected ABM literature. SDM and ABMwere both used to test the impact of policy on undesir-able patient outcomes, including patient mortality [23,25, 58, 60, 67] and hospitalisation rates [23, 25]. Inter-ventions for reducing patient waiting time for services[24, 33, 53, 61, 67] and patient length of stay [22, 31, 67]were also tested using these methods, while policy explor-ation to reduce the total cost of care was more frequentamong SDM studies [33, 54, 61].SDM and ABM software platforms provide accessible,

user-friendly visualisations of systems that enable engage-ment with health experts necessary for model validation[48, 50, 55, 57] and facilitate stakeholder understanding ofhow alternative policies can impact health system per-formance under a range conditions [31, 49, 58, 61]. The


Table 3 Comparison of content between SDM, ABM and hybrid models of health systems literature

SDM papers ABM papers Hybrid papers

Purpose of research Testing policies or interventions:• to relieve at-capacity healthcareservices, reduce ward occupancyand patient length of stay[28, 31, 36, 43, 49, 50, 54, 58, 62].

• to reduce time to patient admissionand treatment [33, 53, 61]

• to reduce delayed discharges [31]• to increase the uptake of healthcareservices and level of healthcareprovision [60]

• to target undesirable patient healthoutcomes (morbidity, mortality,post-treatment complications)[47, 58, 60, 63].

• to optimise performance-basedincentive policies against healthprofessional productivity, qualityof care and volume of services[30, 59].

• to reduce the total cost of care[33, 47, 58, 60, 61, 63].

• to reduce deficit of healthprofessionals [51]

• to reduce generation ofincineration-only health carewaste [52]

• to increase the number ofpatients who currently do notseek medical care [64]

Other:• explore factors leading toundesirable emergency caresystem behaviour [56, 57]

• simulating hospital wastemanagement systems andpredicting future wastegeneration [37, 48, 55].

• estimating future demand forcardiac care [44].

• exploring the impact of patientadmission on health professionalsstress level in an integrated caresystem (IC) [45].

• exploring variation in physiciandecision-making [32].

Testing policies or interventions:• to decrease the time agentsspent performing tasks, waitingfor a service or residing in partsof the system [20, 22, 24, 67].

• to reduce undesirable patientoutcomes (mortality andhospitalisation) [23, 25, 47, 67].

• to reduce the number of patientswho left a health facility withoutbeing seen by a physician [22, 67].

• to reduce number of patients whoare wrongly discharged [67]

• to optimise utility of resources(staff, beds) [46, 66, 67].

• on bypass rate of patientsaccessing care at alternativefacilities [23]

• to reduce total cost of care [25]Other:• Create tools capable of comparinghealth insurance reimbursementschemes [65].

• Assessing risk, allocation ofresources and identifyingweaknesses in emergency careservices [21].

Testing policies or interventions:SDM-DES• to improve access to socialsupport and care services [43].

ABM-DES• to decrease patient waiting timeto be seen by a physician [24].

• to improve patient flow and lengthof stay through the system byoptimising resource allocation [46].

SDM-ABM• to reduce undesirable patientoutcomes (morbidity) [47].

Other:SDM-DES• Estimate the future demand forhealth care from patients withcardiac disease [44].

• Model patient flow throughan integrated care system toestimate impact of patientadmission on health careprofessional’s wellbeing [45].

Healthcare settingmodelled

• Cardiology care [33, 53]• Elderly care or LTC services[28, 31, 36, 49–51, 54, 61, 62]

• Emergency or acute care[28, 31, 36, 50, 56–58, 61, 62]

• Hospital waste management[37, 48, 52, 55]

• ACO or health insurance schemes [63]• MNCH [32, 60]• Orthopaedic care [63]

• Cardiology care [66]• Emergency or acutecare [21, 22, 67]

• ACO or health insuranceschemes [23, 25, 65]

SDM-DES• Cardiology care [44]• Elderly care or LTC services [43–45]• Emergency or acute care [45]ABM-DES• MNCH [46]• Orthopaedic care [24]SDM-ABM• Emergency or acute care [47]

Rationale for usingmodel

• Gain holistic perspective of systemto investigate delays and bottlenecksin health facility processes, exploringcounter-intuitive behaviour andmonitoring interconnected processesbetween sub-systems over time[28, 30, 31, 36, 37, 48, 56, 58].

• Useful tool for predicting future healthsystem behaviour and demand for careservices, essential for health resourceand capacity planning [48, 60].

• Ability to closely replicate humanbehaviour that exists in the realsystem [20–22, 25, 66].

• Provides deeper understandingof multiple agent decision-making[23, 67], agent networks [25] andinteractions [21, 22].

• Provides flexible frameworkcapable of conveying intricatesystem structures [20], wheresimulations captured agent

SDM-DES• Enabled retention of deterministicand stochastic system variabilityand preservation of uniqueand valuable features of bothmethods [44].

• Capable of simulating flow ofentities through system andprovides rapid insight withoutneed for large data collection [43].

• Can simulate individual variability


Table 3 Comparison of content between SDM, ABM and hybrid models of health systems literature (Continued)


• Configuration of model was notlimited by data availability [28, 52, 64] and could integratedata from various sources whenrequired [51].

• Used as a tool for health policyexploration and optimisinghealth system interventions[33, 36, 51, 54, 57, 58, 64].

• Useful for establishing clinical andfinancial ramifications on multiplegroups (such as patients and healthcare providers) [63].

• Identifying and simulating feedback,policy resistance or unintended systemconsequences [59, 61].

• Quantifying the impact of change tothe health system before real worldimplementation [62].

• Visual learning environment enabledengagement with stakeholdersnecessary for model conception andvalidation [48, 50, 55, 57].

• Utilised by decision makers to developand test alternative policies in a‘real-world’ framework [31, 49, 58, 61].

• Suitable for quantitative analyses [53].• Fast running simulation [54].

capacity for learning andadaptive behaviour [20, 25].

• Could incorporate stochasticprocesses that mimicked agenttransition between states [25].

• Took advantage of key individuallevel agent data [25] and integratedinformation from various sources [65].

• Simulation allows patients to havemultiple medical problems at thesame time [65].

• Model can be made generalisableto other settings [65].

• Visualization of system facilitatedstakeholder understanding oftested policy impact [23],particularly those in the healthindustry with minimal modellingexperience [67].

and detailed interactions thatinfluence system behaviour [43].

• Offered dual model functionality[44] vital for simulating human-centric activity [45], reducing thepractical limitations that come withusing a single simulation methodto model health systems [45].

ABM-DES• Captured both patient flowthrough system and agentdecision-making that enabledidentification of health carebottlenecks and optimumresource allocation [24].

SDM-ABM• Could reproduce detailed, highgranularity system elements inaddition to abstract, aggregatehealth system variables [47].

Methods of validation Behavioural validity tests:• Model output reviewed byexperts [57, 60].

• Model output compared withhistorical data and relevant literature[31–33, 36, 48, 50, 54, 58, 59, 61, 62, 64].

Structural validity tests:• Model conception [28, 60],development [30, 36, 50, 53, 54, 57, 62]and formulation [54, 56, 59] validatedby experts.

• Extreme condition or value testing[30, 31, 52, 57, 59, 60, 64].

• Dimensional consistency checks[31, 52, 57, 59, 60].

• Model boundary accuracy checks [31].• Mass balance checks [54].• Integration error checks [31, 52].Sensitivity analysis• to assess how sensitive model outputwas to changes in key parameters[49, 51, 57, 60, 64].

• to test the impact of parameters thathad been based on expert opinionon model output [28].

• to test the robustness and effectivenessof policies [28, 30, 52, 53, 58, 63] (onthe assumption of imperfect policyimplementation [28]).

Behavioural validity tests:• Model output reviewed byexperts [46, 66].

• Model output compared withhistorical data and relevantliterature [20, 22–25, 46, 65, 66].

• F-test [20] and T-test [20, 24](equivalence of variance anddifference in mean tests).

Structural validity tests:• Extreme condition or valuetesting [23, 46].

• Model framework reviewed byexperts [22, 47].

Sensitivity analysis:• to determine how variations oruncertainty in key parameters(particularly where they had notbeen derived from historical orcare data [65]) affected modeloutcomes [23, 25].

Behavioural validity tests:ABM-DES• Model output reviewed byexperts [46].

• Model output compared withhistorical data [24, 46].

• T-test (difference in mean tests) [24].Structural validity tests:ABM-DES• Extreme condition or valuetesting [46].

SDM-ABM• Model framework reviewed byexperts [47].

Sensitivity analysis:SDM-DES• To assess how sensitive modeloutput was to changes in keyparameters [44].

Study limitations • Did not consider how futureimprovements in technologyor service delivery may impactresults [31, 44, 49, 51].

• May not have simulated all possibleactions or interactions that occurred inreal system [30, 61].

• Model cannot encapsulate all healthsub-sector behaviour and spill-overeffects [31, 53].

• Model parameterised with bestinformation available, sometimesmissing key data [20, 22, 25, 67].

• Did not model all real systemcomplexity, simplifications madeto agents and their attributes[20, 23, 65, 66].

• Did not consider all hospitalunits affected by possiblespill-over effects [21].

SDM-DES• Did not consider howfuture improvements intechnologymay impact results [44].

• Did not model all real systemcomplexity, stable number ofpatients with disease per agegroup [44].

• Lack of technology support led


ability to integrate information and data from varioussources was also cited as rationale for using SDM andABM [51]. Reasons for using SDM to model health sys-tems, as opposed to other methods, included gaining awhole-system perspective crucial for investigating undesir-able or counter-intuitive system behaviour across sub-systems [28, 36, 37, 48, 56] and identifying unintendedconsequences or policy resistance with tested health pol-icies [59, 61]. The ability to replicate human behaviour[20–22, 25, 66] and capacity for learning and adaptive be-haviour [20, 25] was frequently cited as rationale for usingABM to simulate health systems.Validation of SDMs and ABMs consisted mostly of behav-

ioural validity tests where model output was reviewed by ex-perts and compared to real system performance data or torelevant literature. Structural validity tests were uncommonamong ABM papers while expert consultation on model de-velopment [30, 36, 50, 53, 54, 57, 62, 63], extreme condition[30, 31, 52, 57, 59, 60, 64] and dimensional consistency tests[31, 52, 57, 59, 60] were frequently reported in the SDM lit-erature. The inability to simulate all actions or interactionsthat occur in the real system [20, 30, 61, 65, 66] and simplifi-cation of model parameters [23, 55, 62, 63, 65] were de-scribed as limitations in both SDM and ABM papers. Dataavailability for model conception and formulation [20, 22,25, 32, 36, 67] and the impact of model boundaries (restrict-ing exploration of interconnected sub-system behaviour [21,31, 53]) were also cited limitations common to both sets ofliterature. Lack of costing analysis [58, 60], short time

horizons [33, 57] and an inability to model future improve-ments in technology or service delivery [31, 44, 49, 51] wereadditionally cited among the SDM papers.

DiscussionStatement of principal findingsOur review has confirmed that there is a growingbody of research demonstrating the use of SDM andABM to model health care systems to inform policyin a range of settings. While the application of SDMhas been more widespread (with 28 papers identified)there are also a growing number of ABM being used(11), just over half of which used hybrid simulation.A single paper used hybrid SDM-ABM to modelhealth system behaviour. To our knowledge this isthe first review to identify and compare the applica-tion of both SDM and ABM to model health systems.The first ABM article identified in this review waspublished almost a decade after the first SDM paper;this reflects to a certain extent the increasing avail-ability of SDM and ABM dedicated software toolswith the developments in ABM software lagging be-hind their SDM modelling counterparts.Emergency and acute care, and elderly care and LTC

services were the most frequently simulated health sys-tem setting. Both sets of services are facing exponentialincreases in demand with constraints on resources, pre-senting complex issues ideal for evaluation throughsimulation. Models were used to explore the impact and

Table 3 Comparison of content between SDM, ABM and hybrid models of health systems literature (Continued)


• Simplification of real system inmodel [55, 62, 63].

• Lack of facility data required formodel conception, formulationand validation [32, 36, 59].

• Lack of costing or cost effectivenessanalysis [58, 60].

• Simulation was over a short timescale and did not evaluate longterm patient outcomes [33, 57].

• Assumptions made in modeldevelopment may not begeneralisableto other settings [36, 63].

• Discussion with stakeholdersthat contributed to modeldevelopment was not performedsystematically [51].

• Quantifying model uncertainty waslimited [64].

to simplifications in configurationof model (how information waspassed between two distinctmodels) [45].

ABM-DES• Need more case studies toexternally validate model [24].

Software platform • iThink or STELLA (same software)[33, 36, 37, 48, 50, 54, 55, 57, 60, 61].

• MATLAB and Simulink [30].• Vensim [28, 32, 52, 53, 62–64].• Did not state [31, 49, 51, 56, 58, 59].

• AnyLogic [23, 25, 65].• Java [66].• Netlogo [20–22, 67].

SDM-DES• Vensim and Simul8 [43, 45].• Does not state [44].ABM-DES• AnyLogic [24, 46]SDM-ABM• AnyLogic [47].


potential spill over effects of alternative policy options,prior to implementation, on patient outcomes, serviceuse and efficiency under various structural and financialconstraints.

Strengths and weaknesses of the studyTo ensure key papers were identified, eight databasesacross four research areas were screened for relevant lit-erature. Unlike other reviews in the field [39, 40], therewas no restriction placed on publication date. Theframework for this review was built to provide a generaloverview of the SDM and ABM of healthcare literature,capturing papers excluded in other published reviews asa result of strict inclusion criteria. These include reviewsthat have focussed specifically on compiling examples ofmodelled health policy application in the literature [35]or have searched for papers with a particular health sys-tem setting, such as those that solely simulate the behav-iour of emergency departments [34]. One particularlycomprehensive review of the literature had excluded pa-pers that simulated hospital systems, which we have ex-plicitly included as part of our search framework [39].The papers presented in this review, with selection re-

stricted by search criteria, provide a broad picture of thecurrent health system modelling landscape. The focus ofthis review was to identify models of facility-basedhealthcare, purposely excluding literature where the pri-mary focus is on modelling disease progression, diseasetransmission or physiological disorders which can befound in other reviews such as Chang et al. [39] andLong et al. [41]. The data sources or details of how datawas used to conceptualise and formulate models are notpresented in this paper; this could on its own be thefocus of another study and we hope to publish these re-sults as future work. This information would be usefulfor researchers who want to gain an understanding ofthe type and format of data used to model health sys-tems and best practice for developing and validatingsuch models.Literature that was not reported in English was ex-

cluded from the review which may have resulted in asmall proportion of relevant papers being missed. Papersthat described DES models, the other popular modellingmethod for simulating health system processes, were notincluded in this review (unless DES methods are pre-sented as part of a hybrid model integrated with SDM orABM) but have been compiled elsewhere [68–70]. Fi-nally, the quality of the papers was not assessed.

Implications for future researchA nominal number of SDM papers (9/28), an even lowerproportion of ABM papers (2/11) and none of the hybridmethods papers simulated health systems based in low-or middle-income countries (LMICs). The lower number

of counterpart models in LMICs can be attributed to alack of capacity in modelling methods and perhaps theperceived scarcity of suitable data; however, the richquantitative and qualitative primary data collected inthese countries for other types of evaluation could beused to develop such models. Building capacity for usingthese modelling methods in LMICs should be a priorityand generating knowledge of how and which secondarydata to use in these settings for this purpose. In this re-view, we observed that it is feasible to use SDM tomodel low-income country health systems, includingthose in Uganda [60] and Afghanistan [30]. The need toincrease the use of these methods within LMICs is para-mount; even in cases where there is an absence of suffi-cient data, models can be formulated for LMICs andused to inform on key data requirements through sensi-tivity analysis, considering the resource and healthcaredelivery constraints experienced by facilities in these set-tings. This research is vital for our understanding ofhealth system functioning in LMICs, and given thegreater resource constraints, to allow stakeholders andresearchers to assess the likely impact of policies or in-terventions before their costly implementation, and toshed light on optimised programme design.Health system professionals can learn greatly from using

modelling tools, such as ABM, SDM and hybrid models,developed originally in non-health disciplines to under-stand complex dynamic systems. Understanding the com-plexity of health systems therefore require collaborationbetween health scientists and scientists from other disci-plines such as engineering, mathematics and computerscience. Discussion and application of hybrid models isnot a new phenomenon in other fields but their utilisationin exploring health systems is still novel; the earliest articledocumenting their use in this review was published in2010 [43]. Five of the six hybrid modelling papers [43–47]were published as conference proceedings (the exceptionKittipittayakorn et al. [24]), demonstrating the need to in-clude conference articles in systematic reviews of the lit-erature in order to capture new and evolving applicationsof modelling for health systems research.The configuration and extent to which two distinct

types of models are combined has been described in theliterature [71–75]. The hybrid modelling papers selectedin this review follow what is described as ‘hierarchical’or ‘process environment’ model structures, the formerwhere two distinct models pass information to eachother and the latter where one model simulates systemprocesses within the environment of another model [72].Truly ‘integrated’ models, considered the ‘holy grail’ [43]of hybrid simulation, where elements of the system aresimulated by both methods of modelling with no cleardistinction, were not identified in this review and in thewider literature remain an elusive target. In a recent


review of hybrid modelling in operational research onlyfour papers were identified to have implemented trulyintegrated hybrid simulation and all used bespokesoftware, unrestricted by the current hybrid modellingenvironments [76].Of the six hybrid modelling papers, only Djanatliev

et al. [47] presented a model capable of both ABM andSDM simulation. The crucial macro- and micro- levelactivity captured in such models represent feedback inthe wider, complex system while retaining the variablebehaviour exhibited by those who access or deliver health-care. With increasing software innovation and growingdemand for multi-method modelling in not only inhealthcare research but in the wider research community,we need to increase their application to modelling healthsystems and progress towards the ‘holy grail’ of hybridmodelling.

ConclusionsWe identified 28 papers using SDM methods and 11 pa-pers using ABM methods to model health system behav-iour, six of which implemented hybrid model structureswith only a single paper using SDM-ABM. Emergencyand acute care, and elderly care and LTC services werethe most frequently simulated health system settings,modelling the impact of health policies and interventionstargeting at-capacity healthcare services, patient lengthof stay in healthcare facilities and undesirable patientoutcomes. A high proportion of articles modelled healthsystems in high income countries; future work shouldnow turn to modelling healthcare settings in LMIC tosupport policy makers and health system researchersalike. The utilisation of hybrid models in healthcare isstill relatively new but with an increasing demand todevelop models that can simulate the macro- and micro-level activity exhibited by health systems, we will see anincrease in their use in the future.

Supplementary informationSupplementary information accompanies this paper at https://doi.org/10.1186/s12913-019-4627-7.

Additional file 1. Search criteria used for each database.

Additional file 2. Descriptive table of validation methods used in SDMand ABM literature.

Additional file 3. Descriptive table of ABM model rules.

AbbreviationsACO: Accountable care organisation; ABM: Agent-based model;DES: Discrete-event simulation; ED: Emergency Department; LTC: Long-termcare; LMIC: Low- and middle-income countries; SDM: System dynamicsmodel

AcknowledgementsNot applicable.

Authors’ contributionsRC, KB, ZC specified the search criteria and selection of databases. RCscreened titles, evaluated full text articles and was responsible for full textextraction. RC is lead author, NSS, KB, ZC, PS, JB, AS, PB, CC, NS providedguidance on the draft and final manuscript. All authors read and approvedthe final manuscript.

FundingThe work described in this paper was funded by the Health SystemsResearch Initiative (HSRI). MRC Grant Reference Number: MR/R013454/1.

Availability of data and materialsData sharing is not applicable to this article as no datasets were generatedor analysed during the current study.

Ethics approval and consent to participateNot applicable.

Consent for publicationNot applicable.

Competing interestsThe authors declare that they have no competing interests.

Author details1Department of Global Health and Development, London School of Hygieneand Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH, UK.2Department of Mathematics, University College London, London, UK. 3SiaPartners UK, London, UK. 4Information Systems Department, College ofComputing and Information Sciences, Makerere University, P.O. Box 7062,Kampala, Uganda. 5Ifakara Health Institute, PO Box 78373, Dar es Salaam,Tanzania. 6Department of Gender Studies, School of Humanities and SocialSciences, University of Zambia, 10101 Lusaka, Zambia. 7Economic andBusiness Research Programme, University of Zambia, Institute of Economicand Social Research, P O Box 30900, 10101 Lusaka, Zambia. 8Department ofPublic Health, Environments and Society, London School of Hygiene andTropical, London, UK.

Received: 14 June 2019 Accepted: 11 October 2019

References1. Kitson A, Brook A, Harvey G, Jordan Z, Marshall R, O’Shea R, et al. Using

Complexity and Network Concepts to Inform Healthcare KnowledgeTranslation. Int J Heal Policy Manag. 2017;7:231–43. https://doi.org/10.15171/ijhpm.2017.79.

2. Paina L, Peters DH. Understanding pathways for scaling up health servicesthrough the lens of complex adaptive systems. Health Policy Plan. 2012;27:365–73. https://doi.org/10.1093/heapol/czr054.

3. Lipsitz LA. Understanding health care as a complex system: the foundationfor unintended consequences. JAMA. 2012;308:243–4. https://doi.org/10.1001/jama.2012.7551.

4. Adam T, de Savigny D. Systems thinking for strengthening health systemsin LMICs: need for a paradigm shift. Health Policy Plan. 2012;27:iv1–3.https://doi.org/10.1093/heapol/czs084.

5. de Savigny D, Blanchet K, Adam T. Applied systems thinking for health systemsresearch : a methodological handbook: McGraw-Hill Education; 2017.

6. Shepherd SP. A review of system dynamics models applied intransportation. Transp B Transp Dyn. 2014;2:83–105. https://doi.org/10.1080/21680566.2014.916236.

7. Sterman JD. Business dynamics: systems thinking and modeling for acomplex world: McGraw-Hill Companies Inc; 2000.

8. Kunc M, Mortenson MJ, Vidgen R. A computational literature review of thefield of system dynamics from 1974 to 2017. J Simul. 2018;12:115–27.https://doi.org/10.1080/17477778.2018.1468950.

9. System Dynamics for Engineering Students. Elsevier; 2018. doi:https://doi.org/10.1016/C2011-0-05346-2.

10. Radzicki MJ. System Dynamics and Its Contribution to Economics andEconomic Modeling. Encycl. Complex. Syst. Sci. New York: Springer NewYork; 2009. p. 8990–9000. https://doi.org/10.1007/978-0-387-30440-3_539.

11. Ford A. Global warming and system dynamics. Int Conf Syst Dyn Soc. 2007.


https://doi.org/10.1186/s12913-019-4627-7

https://doi.org/10.1186/s12913-019-4627-7

https://doi.org/10.15171/ijhpm.2017.79

https://doi.org/10.15171/ijhpm.2017.79

https://doi.org/10.1093/heapol/czr054

https://doi.org/10.1001/jama.2012.7551

https://doi.org/10.1001/jama.2012.7551

https://doi.org/10.1093/heapol/czs084

https://doi.org/10.1080/21680566.2014.916236

https://doi.org/10.1080/21680566.2014.916236

https://doi.org/10.1080/17477778.2018.1468950

https://doi.org/10.1016/C2011-0-05346-2

https://doi.org/10.1016/C2011-0-05346-2

https://doi.org/10.1007/978-0-387-30440-3_539

12. Fiddaman TS. Exploring policy options with a behavioral climate-economymodel. Syst Dyn Rev. 2002;18:243–67. https://doi.org/10.1002/sdr.241.

13. Popli K, Sudibya GL, Kim S. A review of solid waste management usingsystem dynamics modeling. J Environ Sci Int. 2017;26:1185–200. https://doi.org/10.5322/JESI.2017.26.10.1185.

14. Groesser SN, Schaffernicht M. Mental models of dynamic systems: takingstock and looking ahead. Syst Dyn Rev. 2012;28:46–68. https://doi.org/10.1002/sdr.476.

15. Epstein JM. Generative social science: studies in agent-based computationalmodeling: STU-Stud. Princeton University Press; 2006.

16. Wilensky U, Rand W. An Introduction to Agent-Based Modeling: Mit Press; 2015.17. Miller JH, Page SE. Complex Adaptive Systems: An Introduction to Computational

Models of Social Life: STU-Stud. Princeton University Press; 2007.18. Railsback SF, Grimm V. Agent-Based and Individual-Based Modeling:

Princeton University Press; 2011.19. Abar S, Theodoropoulos G, Lemarinier P, O’Hare G. Agent based Modelling

and simulation tools: a review of the state-of-art software. Comput Sci Rev.2017;24:13–33. https://doi.org/10.1016/j.cosrev.2017.03.001.

20. Huynh N, Snyder R, Vidal J, Tavakoli A, Cai B. Application of computersimulation modeling to medication administration process redesign. JHealthc Eng. 2012;3:649–62. https://doi.org/10.1260/2040-2295.3.4.649.

21. Liu Z, Cabrera E, Rexachs D, Luque E. A generalized agent-based model tosimulate emergency departments. Sixth Int Conf Adv Syst Simul. 2014:65–70.

22. Yousefi M, Yousefi M, Fogliatto FS, Ferreira RPM, Kim JH. Simulating thebehavior of patients who leave a public hospital emergency departmentwithout being seen by a physician: a cellular automaton and agent-basedframework. Brazilian J Med Biol Res. 2018;51:e6961. https://doi.org/10.1590/1414-431X20176961.

23. Alibrahim A, Wu S. An agent-based simulation model of patient choice ofhealth care providers in accountable care organizations. Health Care ManagSci. 2018;21:131–43. https://doi.org/10.1007/s10729-016-9383-1.

24. Kittipittayakorn C, Ying K-C. Using the integration of discrete event andagent-based simulation to enhance outpatient service quality in anorthopedic department. J Healthc Eng. 2016;2016:4189206. https://doi.org/10.1155/2016/4189206.

25. Liu P, Wu S. An agent-based simulation model to study accountable careorganizations. Health Care Manag Sci. 2016;19:89–101. https://doi.org/10.1007/s10729-014-9279-x.

26. Allen M, Spencer A, Gibson A, Matthews J, Allwood A, Prosser S, et al. Rightcot, right place, right time: improving the design and organisation ofneonatal care networks – a computer simulation study. Heal Serv Deliv Res.2015;3:9. https://doi.org/10.3310/hsdr03200.

27. Brailsford SC. Tutorial: Advances and challenges in healthcare simulationmodeling: 2007 Winter Simul. Conf., IEEE; 2007. p. 1436–48. https://doi.org/10.1109/WSC.2007.4419754.

28. Esensoy AV, Carter MW. High-Fidelity whole-system patient flow modelingto assess health care transformation policies. Eur J Oper Res. 2018;266:221–37. https://doi.org/10.1016/j.ejor.2017.09.019.

29. Homer JB, Hirsch GB. System dynamics modeling for public health:background and opportunities. Am J Public Health. 2006;96:452–8. https://doi.org/10.2105/AJPH.2005.062059.

30. Alonge O, Lin S, Igusa T, Peters DH. Improving health systems performancein low- and middle-income countries: a system dynamics model of the pay-for-performance initiative in Afghanistan. Health Policy Plan. 2017;32:1417–26. https://doi.org/10.1093/heapol/czx122.

31. Rashwan W, Abo-Hamad W, Arisha A. A system dynamics view of the acutebed blockage problem in the Irish healthcare system. Eur J Oper Res. 2015;247:276–93. https://doi.org/10.1016/j.ejor.2015.05.043.

32. Ghaffarzadegan N, Epstein AJ, Martin EG. Practice variation, bias, and experientiallearning in cesarean delivery: A data-based system dynamics approach. HealthServ Res. 2013;48:713–34. https://doi.org/10.1111/1475-6773.12040.

33. Taylor K, Dangerfield B. Modelling the feedback effects of reconfiguringhealth services. J Oper Res Soc. 2005;56:659–75. https://doi.org/10.1057/palgrave.jors.2601862.

34. Mohiuddin S, Busby J, Savovic J, Richards A, Northstone K, Hollingworth W,et al. Patient flow within UK emergency departments: a systematic reviewof the use of computer simulation modelling methods. BMJ Open. 2017;7:e015007. https://doi.org/10.1136/bmjopen-2016-015007.

35. Atkinson J-A, Wells R, Page A, Dominello A, Haines M, Wilson A. Applicationsof system dynamics modelling to support health policy. Public Heal ResPract. 2015;25. https://doi.org/10.17061/phrp2531531.

36. Lattimer V, Brailsford S, Turnbull J, Tarnaras P, Smith H, George S, et al. Reviewingemergency care systems I: insights from system dynamics modelling. Emerg MedJ. 2004;21:685–91. https://doi.org/10.1136/emj.2002.003673.

37. Chaerul M, Tanaka M, Shekdar AV. A system dynamics approach for hospitalwaste management. Waste Manag. 2008;28:442–9. https://doi.org/10.1016/j.wasman.2007.01.007.

38. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items forsystematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. https://doi.org/10.1136/bmj.b2535.

39. Chang AY, Ogbuoji O, Atun R, Verguet S. Dynamic modeling approaches tocharacterize the functioning of health systems: A systematic review of the literature.Soc Sci Med. 2017;194:160–7. https://doi.org/10.1016/j.socscimed.2017.09.005.

40. Rusoja E, Haynie D, Sievers J, Mustafee N, Nelson F, Reynolds M, et al.Thinking about complexity in health: a systematic review of the key systemsthinking and complexity ideas in health. J Eval Clin Pract. 2018;24:600–6.https://doi.org/10.1111/jep.12856.

41. Long KM, Meadows GN. Simulation modelling in mental health: asystematic review. J Simul. 2017. https://doi.org/10.1057/s41273-017-0062-0.

42. Banco Mundial. World Bank: country and lending groups - DataBank. WorldBank Gr. 2018.

43. Brailsford SC, Desai SM, Viana J. Towards the holy grail: Combining systemdynamics and discrete-event simulation in healthcare: Proc. 2010 Winter Simul.Conf., IEEE; 2010. p. 2293–303. https://doi.org/10.1109/WSC.2010.5678927.

44. Mielczarek B, Zabawa J. Modeling Healthcare Demand Using a Hybrid SimulationApproach, vol. 2016: Proc. 2016 Winter Simul. Conf., IEEE Press. p. 1535–46.

45. Zulkepli J. Hybrid simulation for modelling large systems: an example ofintegrated care model: Proc. 2012 Winter Simul. Conf; 2012.

46. Viana J, Simonsen TB, Dahl FA, Flo K. A Hybrid Discrete Event Agent BasedOverdue Pregnancy Outpatient Clinic Simulation Model. Proc. 2018 WinterSimul. Conf. Piscataway: IEEE Press; 2018. p. 1488–99.

47. Djanatliev A, German R, Kolominsky-Rabas P, Hofmann BM. Hybridsimulation with loosely coupled system dynamics and agent-based modelsfor prospective health technology assessments: Proc. Winter Simul. Conf.,Winter Simulation Conference; 2012. p. 69:1–69:12.

48. Al-Khatib IA, Eleyan D, Garfield J. A system dynamics approach for hospital wastemanagement in a city in a developing country: the case of Nablus, Palestine.Environ Monit Assess. 2016;188:503. https://doi.org/10.1007/s10661-016-5487-9.

49. Ansah JP, Eberlein RL, Love SR, Bautista MA, Thompson JP, Malhotra R, et al.Implications of long-term care capacity response policies for an agingpopulation: a simulation analysis. Health Policy. 2014;116:105–13. https://doi.org/10.1016/j.healthpol.2014.01.006.

50. Brailsford SC. Emergency and on-demand health care: modelling a largecomplex system. J Oper Res Soc. 2004;55.

51. Cepoiu-Martin M, Bischak DP. Policy choices in dementia care - anexploratory analysis of the Alberta Continuing Care System (ACCS) usingsystem dynamics. Spec Issue Complex Forum “real” World Heal Syst ImplicComplex. Theor Sci. 2018;24:278–84.

52. Ciplak N, Barton JR. A system dynamics approach for healthcare wastemanagement: A case study in Istanbul Metropolitan City, Turkey. WasteManag Res. 2012;30:576–86. https://doi.org/10.1177/0734242X12443405.

53. de Andrade L, Lynch C, Carvalho E, Rodrigues CG, Vissoci JRN, Passos GF, et al.System dynamics modeling in the evaluation of delays of care in ST-segmentelevation myocardial infarction patients within a tiered health system. PLoSOne. 2014;9:e103577. https://doi.org/10.1371/journal.pone.0103577.

54. Desai MS, Penn ML, Brailsford S, Chipulu M. Modelling of Hampshire adultservices—gearing up for future demands. Health Care Manag Sci. 2008;11:167–76. https://doi.org/10.1007/s10729-007-9049-0.

55. Eleyan D, Al-Khatib IA, Garfield J. System dynamics model for hospital wastecharacterization and generation in developing countries. Waste Manag Res.2013;31:986–95. https://doi.org/10.1177/0734242X13490981.

56. Lane D. Emergency but no accident: a system dynamics study of casualtywaiting times in the British NHS. Eurohealth (Lond). 1998;4.

57. Lane DC. Looking in the wrong place for healthcare improvements: asystem dynamics study of an accident and emergency department. J OperRes Soc. 2000;51.

58. Mahmoudian-Dehkordi A, Sadat S. Sustaining critical care: using evidence-based simulation to evaluate ICU management policies. Health Care ManagSci. 2017;20:532–47. https://doi.org/10.1007/s10729-016-9369-z.

59. Meker T, Barlas Y. Dynamic consequences of performance-based paymentSystems in Public Hospitals. Syst Res Behav Sci. 2015;32:459–80. https://doi.org/10.1002/sres.2338.


https://doi.org/10.1002/sdr.241

https://doi.org/10.5322/JESI.2017.26.10.1185

https://doi.org/10.5322/JESI.2017.26.10.1185



https://doi.org/10.1016/j.cosrev.2017.03.001

https://doi.org/10.1260/2040-2295.3.4.649

https://doi.org/10.1590/1414-431X20176961

https://doi.org/10.1590/1414-431X20176961

https://doi.org/10.1007/s10729-016-9383-1

https://doi.org/10.1155/2016/4189206

https://doi.org/10.1155/2016/4189206

https://doi.org/10.1007/s10729-014-9279-x

https://doi.org/10.1007/s10729-014-9279-x

https://doi.org/10.3310/hsdr03200

https://doi.org/10.1109/WSC.2007.4419754

https://doi.org/10.1109/WSC.2007.4419754

https://doi.org/10.1016/j.ejor.2017.09.019

https://doi.org/10.2105/AJPH.2005.062059

https://doi.org/10.2105/AJPH.2005.062059

https://doi.org/10.1093/heapol/czx122


https://doi.org/10.1111/1475-6773.12040

https://doi.org/10.1057/palgrave.jors.2601862

https://doi.org/10.1057/palgrave.jors.2601862

https://doi.org/10.1136/bmjopen-2016-015007

https://doi.org/10.17061/phrp2531531

https://doi.org/10.1136/emj.2002.003673

https://doi.org/10.1016/j.wasman.2007.01.007

https://doi.org/10.1016/j.wasman.2007.01.007

https://doi.org/10.1136/bmj.b2535

https://doi.org/10.1016/j.socscimed.2017.09.005

https://doi.org/10.1111/jep.12856

https://doi.org/10.1057/s41273-017-0062-0

https://doi.org/10.1109/WSC.2010.5678927

https://doi.org/10.1007/s10661-016-5487-9

https://doi.org/10.1016/j.healthpol.2014.01.006

https://doi.org/10.1016/j.healthpol.2014.01.006

https://doi.org/10.1177/0734242X12443405

https://doi.org/10.1371/journal.pone.0103577

https://doi.org/10.1007/s10729-007-9049-0

https://doi.org/10.1177/0734242X13490981

https://doi.org/10.1007/s10729-016-9369-z

https://doi.org/10.1002/sres.2338

https://doi.org/10.1002/sres.2338

60. Semwanga AR, Nakubulwa S, Adam T. Applying a system dynamicsmodelling approach to explore policy options for improving neonatalhealth in Uganda. Heal Res Policy Syst. 2016;14:35. https://doi.org/10.1186/s12961-016-0101-8.

61. Walker BC, Haslett T. The dynamics of local rules in hospital admissionprocesses. Aust Health Rev. 2003;26:98–106.

62. Wong HJ, Wu RC, Caesar M, Abrams H, Morra D. Smoothing inpatient dischargesdecreases emergency department congestion: a system dynamics simulationmodel. Emerg Med J. 2010;27:593–8. https://doi.org/10.1136/emj.2009.078543.

63. Worni M, Pietrobon R, Zammar GR, Shah J, Yoo B, Maldonato M, et al.System dynamics to model the unintended consequences of denyingpayment for venous thromboembolism after total knee arthroplasty. PLoSOne. 2012;7:e30578. https://doi.org/10.1371/journal.pone.0030578.

64. Yu W, Li M, Ge Y, Li L, Zhang Y, Liu Y, et al. Transformation of potentialmedical demand in China: A system dynamics simulation model. J BiomedInform. 2015;57:399–414. https://doi.org/10.1016/j.jbi.2015.08.015.

65. Einzinger P, Popper N, Breitenecker F, Pfeffer N, Jung R, Endel G. The GAP-DRG Model: Simulation of Outpatient Care for Comparison of DifferentReimbursement Schemes. Proc. 2013 Winter Simul. Conf. Simul. Mak. Decis.A complex world. Piscataway: IEEE Press; 2013. p. 2299–308.

66. Hutzschenreuter AK, Bosman PAN, Blonk-Altena I, van Aarle J, La Poutré H.Agent-based patient admission scheduling in hospitals. Belgian/NetherlandsArtif Intell Conf. 2008:315–6. https://doi.org/10.1007/3-540-32062-8.

67. Yousefi M, Ferreira R. An agent-based simulation combined with groupdecision-making technique for improving the performance of anemergency department. Brazilian J Med Biol Res. 2017;50:1–10. https://doi.org/10.1590/1414-431X20175955.

68. Günal MM, Pidd M. Discrete event simulation for performance modelling inhealth care: a review of the literature. J Simul. 2010;4:42–51. https://doi.org/10.1057/jos.2009.25.

69. Salleh S, Thokala P, Brennan A, Hughes R, Dixon S. Discrete event simulation-basedresource Modelling in health technology assessment. Pharmacoeconomics. 2017;35:989–1006. https://doi.org/10.1007/s40273-017-0533-1.

70. Zhang X. Application of discrete event simulation in health care: asystematic review. BMC Health Serv Res. 2018;18:687. https://doi.org/10.1186/s12913-018-3456-4.

71. Bennett PG. On linking approaches to decision-aiding: issues and prospects.J Oper Res Soc. 1985;36:659. https://doi.org/10.2307/2582261.

72. Chahal K, Eldabi T. Applicability of hybrid simulation to different modes ofgovernance in UK healthcare: 2008 Winter Simul. Conf., IEEE; 2008. p. 1469–77. https://doi.org/10.1109/WSC.2008.4736226.

73. Lättilä L, Hilletofth P, Lin B. Hybrid simulation models – when, why, how?Expert Syst Appl. 2010;37:7969–75. https://doi.org/10.1016/j.eswa.2010.04.039.

74. Brailsford SC. Hybrid simulation in healthcare: New concepts and new tools:2015 Winter Simul. Conf., IEEE; 2015. p. 1645–53. https://doi.org/10.1109/WSC.2015.7408284.

75. Morgan JS, Howick S, Belton V. A toolkit of designs for mixing discreteevent simulation and system dynamics. Eur J Oper Res. 2017;257:907–18.https://doi.org/10.1016/j.ejor.2016.08.016.

76. Brailsford SC, Eldabi T, Kunc M, Mustafee N, Osorio AF. Hybrid simulationmodelling in operational research: a state-of-the-art review. Eur J Oper Res.2018. https://doi.org/10.1016/j.ejor.2018.10.025.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.


https://doi.org/10.1186/s12961-016-0101-8

https://doi.org/10.1186/s12961-016-0101-8

https://doi.org/10.1136/emj.2009.078543

https://doi.org/10.1371/journal.pone.0030578

https://doi.org/10.1016/j.jbi.2015.08.015

https://doi.org/10.1007/3-540-32062-8

https://doi.org/10.1590/1414-431X20175955

https://doi.org/10.1590/1414-431X20175955

https://doi.org/10.1057/jos.2009.25

https://doi.org/10.1057/jos.2009.25

https://doi.org/10.1007/s40273-017-0533-1

https://doi.org/10.1186/s12913-018-3456-4

https://doi.org/10.1186/s12913-018-3456-4

https://doi.org/10.2307/2582261

https://doi.org/10.1109/WSC.2008.4736226

https://doi.org/10.1016/j.eswa.2010.04.039

https://doi.org/10.1109/WSC.2015.7408284

https://doi.org/10.1109/WSC.2015.7408284



Date post:	18-Feb-2022
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Mathematical modelling for health systems research: a ...

Documents