©CopyrightJASSS

Ju-SungLeea,TatianaFilatovaa,ArikaLigmann-Zielinskab,BehroozHassani-Mahmooeic,ForrestStonedahld,IrisLorscheide,AlexeyVoinova,GaryPolhillf,ZhanliSungandDawnC.Parkerh(2015)aUniversityofTwente,TheNetherlands; bMichiganStateUniversity,UnitedStates; cMonashUniversity,Australia; dAugustanaCollege,UnitedStates; eHamburgUniversityofTechnology,Germany;fTheJamesHuttonInstitute,UnitedKingdom; gLeibnizInstituteofAgriculturalDevelopmentinTransitionEconomies,Germany; hUniversityofWaterloo,Canada

TheComplexitiesofAgent-BasedModelingOutputAnalysis

JournalofArtificialSocietiesandSocialSimulation 18(4)4<http://jasss.soc.surrey.ac.uk/18/4/4.html>DOI:10.18564/jasss.2897

Received:28-Apr-2015Accepted:29-Jun-2015Published:31-Oct-2015

Abstract

Theproliferationofagent-basedmodels(ABMs)inrecentdecadeshasmotivatedmodelpractitionerstoimprovethetransparency,replicability,andtrustinresultsderivedfromABMs.ThecomplexityofABMshasriseninstridewithadvancesincomputingpowerandresources,resultinginlargermodelswithcomplexinteractionsandlearningandwhoseoutputsareoftenhigh-dimensionalandrequiresophisticatedanalyticalapproaches.Similarly,theincreasinguseofdataanddynamicsinABMshasfurtherenhancedthecomplexityoftheiroutputs.Inthisarticle,weofferanoverviewofthestate-of-the-artapproachesinanalysingandreportingABMoutputshighlightingchallengesandoutstandingissues.Inparticular,weexamineissuessurroundingvariancestability(inconnectionwithdeterminationofappropriatenumberofrunsandhypothesistesting),sensitivityanalysis,spatio-temporalanalysis,visualization,andeffectivecommunicationofallthesetonon-technicalaudiences,suchasvariousstakeholders.

Keywords:Agent-BasedModelling,Methodologies,StatisticalTest,SensitivityAnalysis,Spatio-TemporalHeterogeneity,Visualization

Introduction1.1 Agent-basedmodels(ABMs)havebeengainingpopularityacrossdisciplinesandhavebecomeincreasinglysophisticated.The

lasttwodecadeshaveseenexcellentexamplesofABMapplicationsinabroadspectrumofdisciplinesincludingecology(Grimm&Railsback2005;Thiele&Grimm2010),economics(Kirman1992;Tesfatsion&Judd2006),healthcare(Effkenetal.2012),sociology(Macy&Willer2002;Squazzoni2012),geography(Brown&Robinson2006),anthropology(Axelrod&Hammond2003),archaeology(Axtelletal.2002),bio-terrorism(Carleyetal.2006),business(North&Macal2007),education(Abrahamsonetal.2007),medicalresearch(An&Wilensky2009),militarytactics(Ilachinski2000),neuroscience(Wangetal.2008),politicalscience(Epstein2002),urbandevelopmentandlanduse(Brownetal.2005),andzoology(Brysonetal.2007).Thismethodologynowalsopenetratesorganizationalstudies(Carley&Lee1998;Lee&Carley2004;Chang&Harrington2006),governance(Ghorbanietal.2013),andisbecomingactivelyemployedinpsychologyandotherbehaviouralstudies,exploitingdatafromlaboratoryexperimentsandsurveys(Duffy2006;Continietal.2007;Klingert&Meyer2012).

1.2 ABMsproducearichsetofmultidimensionaldataonmacrophenomena,comprisingamyriadofdetailsonmicro-levelagentchoicesandtheirdynamicinteractionsatvarioustemporalandspatialresolutions.DespitesignificantprogressmadeinempiricallygroundingABMmechanismsandagentattributes(Robinsonetal.2007;Windrumetal.2007;Smajgletal.2011),ABMscontinuetoharbouraconsiderableamountofsubjectivityandhencedegreesoffreedominthestructureandintensityofagents'interactions,agents'learningandadaptation,andanypotentialthresholdsaffectingswitchinginstrategies.TheincreasingcomplexityofABMshasbeenfurtherstimulatedbyimprovementsincomputingtechnologyandwideravailabilityof

http://jasss.soc.surrey.ac.uk/18/4/4.html 1 25/01/2016

advancedcomputingresources.Thesequalitiesdemandacomprehensive(oratleastsufficient)explorationofthemodel'sbehaviour.

1.3 Tocomplicatemattersfurther,anABMistypicallyastochasticprocessandthusrequiresMonteCarlosampling,inwhicheachexperiment(orparametersetting)ismultiplyperformedusingdistinctpseudo-randomsequences(i.e.,differentrandomseeds)inordertoachievethestatisticalrobustnessnecessaryfortestinghypothesesanddistinguishingmultiplescenariosundervaryingexperimentalorparametersettings.By"randomseed",wemeantheseedfortherandomnumbergenerator.Thus,anABMdeliversahighvolumeofoutputdatarenderingtheidentificationofsalientandrelevantresults(suchastrends)andtheassessmentofmodelsensitivitiestovaryingexperimentalconditionsachallengingproblem.

1.4 AllthesecomplicationsapplynotonlytotheanalysisofABMoutputdatabutalsotothemodel'sdesignandimplementation.Themassivediversityinoutputs,oftenexhibitingtemporalandspatialdimensions,necessitatesjudiciousmodeldesign,planning,andapplication.Poororunstructureddesignmayleadtounnecessarilylargeroutputstoreachthesame(orevenless)preciseconclusionsthatonemayinferfromoutputsofwell-designedmodelsandexperimentation.ThestandardsemployedintheABMfield,suchasODD(Overview,Designconcepts,andDetails)(Grimmetal.2006,2010;Mülleretal.2013)andDOE(designofexperiments)(Fisher1971),havesignificantlyimprovedtransparency,replicability,andtrustinABMresults.However,thefieldcontinuestolackspecificguidanceoneffectivepresentationandanalysisofABMoutputdata,perhapsduetothisissue'shavinglesspriorityinABMsocialscienceresearchorduetotechnicalbarriers.Furthermore,convergingonuniversalstandardsremainselusivepartlyduetothebroadspectrumofresearchfieldsemployingABMs.Domain-relevantmetrics,analyticaltechniques,andcommunicationstylesarelargelydrivenbyeachdiscipline'stargetaudience.

1.5 Yet,therearecommonmethodologicalchallengesfacingABMmodellersintheirpathtowardunderstanding,refining,anddistillingthemostrelevantandinterestingresultsfromanearly-endlessseaofoutputdata.WhileamodellerinvestsasignificantamountoftimeandeffortinthedevelopmentofanABMitself,acomprehensiveorcompellinganalysisoftheABMoutputdataisnotalwaysconsideredasdeservingthesameresource-intensiveattention.Properoutputanalysisandpresentationarevitalfordevelopingadomain-specificmessagecontaininginnovativecontributions.

1.6 Thispaperaimstoprovideanoverviewofthestate-of-the-artinhowagent-basedmodellerscontendwiththeirmodeloutputs,

theirstatisticalanalysis,andvisualizationtechniques.[1]Wediscusschallengesandofferexamplesforaddressingthem.Thefirsttopicdealswithseveralissuessurroundingthestudyofvarianceinthemodeloutputs(i.e.,stability)anditsimpactonbothmodeldesign(e.g.,simulationrunsor"samples")andanalysis(e.g.,hypothesistesting).Thenextoneaddressesthestateofsensitivityanalysisandthecomplexitiesinherentintheexplorationofthespacethatencapsulatesboththeparametersandtheoutcomes.Thethirdtopicfocusesontheanalysisandpresentationofspatial(includinggeospatial)andtemporalresultsfromABMs.

1.7 Wealsosurveytheroleofeffectivevisualizationasamediumforbothanalysisandexpositionofmodeldynamics.Commentsonvisualizationappearwithineachmaintopicasvisualizationstrategiestendtobestronglydefinedandconstrainedbythetopicmatter.Finally,weoutlineoutstandingissuesandpotentialsolutionswhicharedeemedasfutureworkforourselvesandotherresearchers.

DeterminingMinimumSimulationRunsandIssuesofHypothesisTesting2.1 ABMresearchersstrivetoexposeimportantandrelevantelementsintheirmodels'outputsandconsequentlytheunderlying

complexdynamicsinbothquantitativeandqualitativeways.CompellingstatementsaboutanABM'sbehavioursmaybedrawnfromdescriptivestatisticsofdistinctoutcomes(e.g.,meanandstandarddeviation)orstatisticaltestsinwhichoutcomesarecompared(e.g.,t-test),predicted(inthestatisticalsense,e.g.,multipleregression),orclassified(e.g.,clusteringorprincipalcomponentanalysis).GiventhestochasticnatureofmostABMs,theseanalyticalexercisesrequireanoutcomepooldrawnfromasufficientnumberofsamples(i.e.,simulationruns).

2.2 ThequantityofABMoutputsampleshasseveralramificationstoexperimentaldesignandthequalityoftheanalysis.ForthoselargeandcomplexABMswhoselongerruntimesprohibittheproductionoflargesamples,therelevantquestionistheminimumnumberofrequiredruns.Conversely,expedientABMsofferthetemptationofproducingfargreatersamplecountstherebyincreasingthesensitivityofstatisticaltestspossiblytothepointofabsurdity.Thatis,onemightproducesomanysamplessuchthattraditionaltestsexposeextremelysmallandcontextuallyinconsequentialdifferences.Wefocusourdiscussiononthemethodsforthedeterminingthenumberofminimumruns.ImplicationsofhavingtoomanysamplesarediscussedinAppendixB.

MinimumSampleSize(NumberofRuns)

2.3 Thedeterminationoftheminimumsamplesizepartlyreliesontheanalyticalobjective.Onecommonobjectiveisastatisticaldescriptionoftheoutcomestypicallyintheformofmeansandstandarddeviations(oralternatively,variances).Sincetheshapeofamodel'soutputdistributionsareusuallyaprioriunknown,thepointorsamplesizeatwhichoutcomemeanandvariancereachesrelativequiescenceorstabilityiscrucialtoaccuratereportingofthedescriptivestatistics.Otherwise,thestatisticswouldharbourtoomuchuncertaintytobereliable.Unfortunately,allofthedifferentcriteriafordeterminingthispointofstabilitysuffersfromsomedegreeofsubjectivity,andthusitfallsontheanalysttowiselymaketheselection.Furthermore,theseapproachesappeartoremaineitherunderusedorunknowntomanyABMresearchers(Hamill2010).Infact,asurveyofsomeofABMliteraturerevealssamplesizestobetoolow,convenientlyselected(samplesizesof100orlessarecommon),orexorbitantly

http://jasss.soc.surrey.ac.uk/18/4/4.html 2 25/01/2016

high(Angus&Hassani-Mahmooei2015).

VarianceStability

2.4 Assessingvariancestabilityrequiresametrictomeasuretheuncertaintysurroundingthevariance(orthevarianceofvarianceifyouwill).LawandKelton(2007)andLorscheidetal.(2012)offersuchmetricsbothofwhichrelyonsomefunctionalratiobetweenthevarianceandthesamplemean.Law&Kelton'sapproachseeksasamplesizeinwhichthevariabilityremainswithinsomepredefinedproportionoftheconfidenceintervalaroundthemean(confidenceintervalboundvariance),thusitisboundtotheassumptionsofnormality,namelythatthemeanhasaGaussiandistribution.Hence,theresearchermustselectthisproportionfromoutputsofatrialsetofruns.Lorscheidetal.'smethodeschewsthoseassumptionsbyemployingthecoefficientofvariationandafixedepsilon(E)limitofthatmetric.Whilethesearedimensionlessmetrics,theiruseisproblematicforsimulationstudiesinwhichmultivariateoutcomesfromparametereffectsareanalyzed.

2.5 Thecoefficientofvariationissimplytheratioofthestandarddeviationofasample(σ)toitsmean(μ):

cV =

σμ

2.6 Thisscalingoffersasimilarinterpretationastheconfidenceinterval(C.I.):cV→ 0isequivalenttot(0 ∉ C. I. ) → ∞andthep-valueapproaches0.Thatis,whenthestandarddeviationshrinksrelativetothemean,theprobabilityoftheconfidenceintervalspanningacrossthevalueof0dropsprecipitously.Thecoefficientofvariation(cV)willexhibitsubstantialvarianceforsmallsamplesizesjustlikethestandarderrorofthemean.Forexample,thecVofasingleABMoutcomeobtainedfromasetoffiverunswillvarymorewiththesamemetricstakenfromothersetsoffiverunsthanifeachsetcontainedfarmoreruns,say100.Lorscheidetal.comparethecV'sofdifferentlysizedsetsofruns(e.g.,thecVfrom10runs,then100,500,andsoforth).ThesamplesizeatwhichthedifferencebetweenconsecutivecV'sfallsbelowacriterion,E,andremainssoisconsideredaminimumsamplesizeorminimumnumberofABMruns.Forexample,ifanoutcomedrawnfromrunsofdifferentsamplesizes,n ∈ {10, 500, 1000, 5000, 10000},yieldsthecV's(roundedto1/100){0.42, 0.28, 0.21, 0.21, 0.21}andweselectE = 0.01,wewouldconsiderthethirdsamplesize(n = 1000)asthepointofstability.ThesecVstabilitypointsareobtainedforallABMoutcomesofinterest(inamultivariatesetting),andthustheminimumnumberofrunsfortheABMisthemaximumofthesepoints:

nmin = argmaxn |cx ,nV − c

x ,mV | < E, ∀xand∀m > n

wherenmin istheestimatedminimumsamplesize;xisadistinctoutcomeofinterest;andmissomesamplesize > nforwhichthecV(ofeachoutcome)ismeasured.

2.7 However,thefixedEfavorsthoseμsufficientlylargerthan0andalsolargerthanitscorrespondingσandpenalizesnmin foroutcomeswithμcloserto0.Thatis,themorelikelyanoutcome'sconfidenceintervalencompasses0,theerroneouslylargertheestimationofnmin willbe.Conversely,thefixedErenderstheproceduretooeffectiveandresultsinanunderestimationofnmin forthoseC.I.'sthatresidefarfrom0,relativetoσ.Therefore,weurgesomecautioninusingcVtodeterminetheminimumsamplesizeandapplyingitonlytoABMsforwhichtheoutcomesofinterestareprejudicedagainstattainingavalueof0.Assuch,Lorscheidetal.'sapproachdeterminesaminimumsamplesizenotjustbasedonvariancestabilitybutalsothelikelihoodanoutcome'sconfidenceintervalcontains0.

2.8 Alternatively,onemightconsiderjustvariancestabilitywithoutanyconsiderationofthemeanvalue.FollowingLorscheidetal.'sstrategyofassessingstabilityfrommetricsonanoutcomeforasequenceofsamplesizes,wetrackthewindowedvarianceofsimpleoutcomesfromseveralcanonicalstatisticaldistributionsaswellasanABMmodel.Thedistributionsweemployherefor

demonstrationpurposesarethenormal(orGaussian),uniform,exponential,Poisson,χ2,andtdistributions.[2]WhileABMoutputsdonotalwaysconformtooneoftheseparametricdistributions,theonesweexamineherearedistinctenoughtoprovideuswithasenseofvariancestabilityforaspectrumofdistributions.ThesedistributionswillserveasproxiesforABMs'outcomesand,forourpurposes,havebeenparameterizedsothattheirtheoreticalvarianceσ2 ≈ 1.WealsoincludeanoutcomefromasimpleABMmodelofBirthRates(Wilensky1997).ThisABMentailstwodynamicallychangingpopulations(labelled"red"and"blue").Here,ouroutcomeisthesizeofthe"red"population.Forfurtherdetails,seeFigure17inAppendixC.

2.9 InFigure1a,weoffervariancesforvaryingsizedsamplesofGaussianvariates(i.e.,scalarsdrawnfromtheGaussiandistributionparameterizedtohaveavarianceof1).Atlowsamplesizes,thereisconsiderablevariancesurroundingthesamplevarianceitselfasevidencedbythe"noisiness"ofthevariancefromonesamplesizetothenext.Afteracertainpoint(roughlyn = 400),thisoutervarianceappearstostabilizeandcontinuestofurtherconvergeto0albeitslowly.

2.10 Theoutervarianceateachsamplesizecanthenbemeasuredusingthevarianceofproximalsamplesizes.Forexample,theoutervarianceforn = 10iscalculatedfromvarianceofthesamplevariancesofn ∈ {10, …, 10 + (W − 1)g},whereWissome

http://jasss.soc.surrey.ac.uk/18/4/4.html 3 25/01/2016

predeterminedsizeofthewindowandgisoursamplesizeincrement;weselectg = W = 10,andwesoconsidern ∈ {10, 20, …, 100}forthevariancesurroundingn = 10.Notationally,thisoutervariancemaybeexpressedasσ2(s2)butgivenitsestimationusingwindows,weassignitω2n,wherenisthesamplesize.InFigure1,wecharttheω2nrelativetoitsmaximumvalue

(

ω2n

maxω2 )foreachdistribution.

Figure1.SampleSizevs.VarianceStability.Thecoloursintherightplotdenotethedistribution:normal,uniform,

exponential,Poisson,χ2,Student'st,andBirthRateABM.

2.11 Thecoloursdistinguishthesevendistributions.Thegrey,dashedhorizontallineexpressesoursemi-subjectivecriterion(of0.20)underwhichtherelativeω2nmustresideinorderforntobedeemeda"minimumsamplesize".Thespeedatwhichthisconditionismetvariesconsiderablyamongthedistributions,highlightingtheneedtoforegodistributionalassumptionsregardingonesABMoutputsinvariance-basedminimumsamplesizedetermination,unlessthedistributionalshapesarewell-identified.Thisapproachforassessingvariancestabilitybearstwoelementsoffurthersubjectivity.Firstly,thepointofstabilitymaybeidentifiedbyeitherthefirstnatwhichthecriterionconditionismetorthefirstnatwhichallfurthersamplesizesmeetthecondition.TheredandgreenverticallinesinFigure1arespectivelydenotethesepointsofstabilityfortheGaussiandistribution.Secondly,theoutervariancecouldhavebeenassessedusingalargerpoolofvariatesateachnratherthanestimatedunderwindows.However,inkeepingwiththestrategyofminimizingthenumberoftestsimulationrunsnmin ,wereportthefindingsofwindow-basedω2ratherthanσ2(s2).

EffectSize

2.12 Intraditionalstatisticalanalysis,thecommonapproachfordeterminingminimumsamplesizerequiresonetofirstselectthesizeofadetectableeffect(i.e.,astatisticsuchasthemeanordifferenceofmeansscaledbyapooledstandarddeviation).ThisapproachalsorequiresaselectionofacceptablelevelsofthetypeIandtypeIIerrors.AtypeIerroriscoarselytheprobability(inthefrequentistsense)thatthenullhypothesisisrejectedwheninfactitistrue.ThetypeIIerroristheconverse:thelikelihoodthatthenullhypothesisisretainedwhenthealternativehypothesisistrue.Forexample,onemightcomparethemeansoftwoABMoutcomeseachfromseparatesetsofsampleofruns.Theseoutcomesmaybeborneofdistinctmodelparameterizations(anecessarythoughnotsufficientconditionforyieldingatruedifference)andmeasuredtobesignificantlydifferentunderarudimentaryt-test.However,asmallsamplesizewillpenalizethetestwhichmaynotreportastatisticallysignificantdifferencebetweenmeansofthoseoutcomes:atypeIIerror.Alternatively,theseoutcomesmayarisefromidenticalparameterizationsyetthet-testerroneouslyrevealsasignificantdifference:atypeIerror.TheratesoftypeIandIIerrorsareexpressedasαandβ.TheconverseofthetypeIIerror(1 − β)iscalledthe"power"level.

2.13 Theminimumsamplesizenmin canthenbecomputedas

nmin ≥ 2

s2

δ (tV,1 −α / 2 + tV,1 −β/ 2)2

wheresisthestandarddeviationoftheoutcomeorthepooledstandarddeviationoftwooutcomes,δislowerboundontheabsolutedifferenceinmeansthatistobeclassifiedassignificantlydifferent;tisthet-statistic(orthequantilefunctionofthetdistribution);νisthedegreesoffreedom(herenmin − 1),andαandβarethelevelsoftypeIandIIerrorsrespectively.Inlayterms,theminimumsamplesizeoccursatthepointatwhichbothtypeIandtypeIIerrorsoccuratthedesiredcriticallevelsasdeterminedbythet-test,hencetheemploymentofthetdistribution.ThisapproachhasbeensuggestedintheABMliterature(e.g.,Radax&Rengs2010).Asnmin appearsonbothsidesoftheequation,trial-and-errororalgorithmiciterationscanusually

http://jasss.soc.surrey.ac.uk/18/4/4.html 4 25/01/2016

convergeonnmin .Asimilarequationisemployedwhentheoutcomeisaproportion∈ [0, 1].Thenon-centraltdistributioncanalsobeusedforsamplesizedetermination(asintheRstatisticalpackage).

2.14 Acloseinspectionofthisapproachrevealstestsensitivitytotheoutcomedistribution'sdeparturefromthenormal.InTable1,weempiricallymeasurethepowerlevel(andhencetheleveloftypeIIerror)bydrawingpairsofvariatesetsfromsometypical

distributionsandtheBirthRatemodel,parameterizedsuchthatidenticaleffectsizesof0.5.[3]Thus,aninsignificantt-testcomparisonforapairofvariatesetsistantamounttoatypeIIerror.Foreachdistributiontypeandsamplesizen,thecomparisonwasperformedfor5000pairsofvariatesetseachofsizen.Eachsetpairwascompared,andwemonitortheproportionofpairsofthesesetsthatyieldedasignificantp-value:anempirically-derivedpowerlevel.TheempiricalpowerlevelsforarangeofsamplesizesnaregraphicallyshowninFigures15and16inAppendixA.

Table1:MinimumSampleSizesforOutcomeDistributions.nt,ne,andnWarethetheoretically-derived,empirically-derived,andWilcox-test-basedminimumsamplesizes,nmin .

Distr. nt ne nW ne − nt nW− nt

Normal 64 65 68 1 4Exponential 64 59 78 −5 14Poisson 65 61 64 −4 −1

BirthRate 64 65 70 1 6BirthRate(d = 1.0) 18 19 26 1 8

2.15 Weobserveincongruitiesbetweenthetheoreticalnmin (nt)andtheempirically-derivedne.Infact,thepowercalculationoverestimatesnmin fortheskeweddistributions(i.e.,theexponentialandthePoisson).Whilethedifferencesinthesenmin arerelativelyminor,theycouldhaveamaterialbenefitforlargescaleABMsforwhicheachruniscostly.However,inthesecases,usingthet-testforexposingapredeterminedeffectsizehastobedeemedappropriate.Forthesesurveyeddistributiontypes,theempiricaldistributionsofthemeansthemselvesfrequentlypasstheShapirotestofnormalityhenceallowingfortheuseofthet-test.

2.16 Giventhesensitivityofthetraditionalt-testtodistributionalskewnessaswellastheuncertaintyofthedistributionalshapeinABMoutcomes,onemightturntoamoreconservativetest,theWilcoxonrankedsumtest (alsoknownastheMann-Whitneytest);thenWcolumnreportsthistest'ssuggestedminimumsamplesizes.Interestingly,themoreefficientWilcoxontestappearstoproposealowerminimumsizeforthePoissondistribution.

2.17 Whenweassesstheefficiencyofcalculatingnmin fortheBirthRateABM,wefindthat,despitetheflatnessplusbimodalityoftheoutcomedistribution,thecalculationofnmin isalmostaccurate,andtheWilcoxtestismodestlyconservativecomparedtotheexponentialdistribution.

MultivariateStability

2.18 SincemostABMstypicallyproducemultipleoutcomes,thecalculationoftherequiredsamplesizewouldhavetoconsideralloutcomesofinterest.Furthermore,analysisofABMsoftenentailsanexplorationofparametersettings(ortheparameterspace)inordertounderstandthedependenciesbetweenkeyinputparametersandtheiroutcomesincludingtheirvariability.GiventhecomplexityofABMs,theoutcomes'variancemayormaynotbeconstant(i.e.,homoskedastic)acrosstheparametersettings.Hence,thetaskofunderstandingmodeloutputsensitivitytodifferentexperimentalconditionsthatarerelevanttotheresearchquestionisvital.Whilethetopicofsensitivityanalysisisfurtherelaboratedin3.1,wediscussitherebrieflygivenitsroleindeterminingtheadequaterunsamplesizeapplicabletoallthechosenparametersettings.Well-structuredDOE(describedfurtherinSection3)canbeveryhelpfultocomprehensivelyexploremodelvariabilitiescorrespondingtomultiplemodelparameters.

Visualizationforstatisticalissues

2.19 OnevisualizationapproachforexaminingunivariateABMoutcomedistributionsistheviolinplotwhichcombineselementsofaboxplotandakerneldensityplot,withasmoothedestimationofoutcomes'variancesacrosstherangesoffactors/parameters(see,e.g.,Kahl&Hansen2015).Figure2illustratesthemultifacetedinfluenceofsixvariables(eachwith3valuesfollowinga3k

factorialdesign)ontheoutputmeasure("compensationpayment").DetailsmaybefoundinLorscheid(2014).

http://jasss.soc.surrey.ac.uk/18/4/4.html 5 25/01/2016

Figure2.Violinplotsforaunivariateanalysisofpotentialcontrolvariables.Source:Lorscheid(2014)

2.20 Thegreyareaindicatesthedispersionofvalues.Thewhitedotintheplotindicatesthemedianofthedataset.Theblacklineabovethemedianistheareaofthesecondquartile,andtheblacklinebelowthemedianisthethirdquartileofthedataset.ForadetaileddescriptionofviolinplotsseeHintzeandNelson(1998).

Solutionspaceexplorationandsensitivityanalysis3.1 Thereareavarietyoftopicsthatmaybegatheredunderthebroadheadingofparameterorinput-outputspaceexploration,

includingoptimization,calibration,uncertaintyanalysis,sensitivityanalysis,aswellasthesearchforspecificqualitativemodelfeaturessuchas"regimeshifts"and"tippingpoints".Inallcases,thegoalistoprovideadditionalinsightintothebehaviouroftheABMthroughtheexaminationofcertainparametersettingsandtheircorrespondingoutputmeasures.

3.2 Wewillfirstdiscuss"exploration"inabroadsense,followedbyamoredetaileddiscussionofavarietyofmethodsforperformingoneparticularlyimportantexplorationtask:sensitivityanalysis.

Input/OutputSolutionSpaceExploration

3.3 Oneofthemostcommonformsofparameterspaceexplorationismanual(orhuman-guided)exploration(alsocalledthe"trial-and-errormethod"or"educatedguessing").Thisapproachcanbecomputationallyefficientifguidedbyanindividualfamiliarwiththemodel'sdynamicsand/oroutputs.Theintuitionsofamodel'sauthor/developer,adomainexpert,orastakeholdercaninformparsimonious,iterativeparameterselectioninsuchawayastogenerateandtestrelevanthypothesesandminimizethenumberofregionssearchedorthenumberofrequiredsimulationruns.However,thisexplorationstrategyisvulnerabletohumanbiasandfatigueleadingtoadisproportionateamountofattentionpaidtothetargetphenomenaandtheneglectoflargeportionsofthemodel'sbehaviourspace.Distributingtheburdenoftheexplorationtask(e.g.,crowdsourcing)mayaddresssomeofitslimitations.Nevertheless,moresystematic,automated,andunbiasedapproachesarerequiredtocomplementtheshortcomingsofsolelyhuman-guidedsearching,whichwillalwaysplayaroleinspaceexploration,especiallyinitspreliminarystages.

3.4 Thesimplestofthesystematicexplorationapproachesfallundertheclassofregularsamplingtechniquesinwhichtheparametersarechosen(orsampled)inasystematicmannertoensuretheirhavingcertainstatisticalorstructuralproperties.Someofthesetechniquesarerandom,quasirandom,gridded/factorial,Latinhypercube,andsphere-packing.Samplingmaygloballyconsidertheentireparameterspaceorfocuslocallyonaparticularregion(e.g.,alteringoneparameteratatimefor"univariatesensitivityanalysis").Thewell-establishedmethodologicalhistoryofDOEandtherecentliteratureondesignandanalysisofsimulationexperiments(e.g.,Sacksetal.1989)canguidethesamplingstrategy.However,thecomplexityofABMs(andtheiroutputs)canrenderclassicDOEinappropriateasnotedbySanchezandLucas(2002).ClassicDOE(a)assumesonlylinearorlow-orderinteractionsamongexperimentalparameters(orfactors)andoutputs,(b)makeslittleornoprovisionfortheiterativeparameterselectionprocess(i.e.,sequentialvirtualexperiments),and(c)alsoassumestypicalerror(Gaussianandunimodal)intheoutput.Anexampleofmulti-modalABMoutputispresentedinAppendixC.Thus,traditionalDOEmethods,whileusefulforABMs,oughttobeimplementedwithcautionandconsiderationforappropriatemethodsthataddresscomplexitiesintheoutput.ResearchintoDOEmethodsforABMsisstillevolving(Kleinetal.2005;Kleijnenetal.2005;Ankenmanetal.2008;Lorscheidetal.2012).

3.5 Researchintomoresophisticatedexplorationtechniqueshasbeeninformedbymeta-heuristicsearching,optimizationalgorithms,andmachinelearning.Researchersproposetheuseofgeneticalgorithms(GA)(Holland1975)forawiderangeofexplorationtasksincludingdirectedsearchesforparametersthatyieldspecificemergentbehaviours(Stonedahl&Wilensky2011),parameteroptimization(Stonedahletal.2010),calibration/parameterestimation(Calvez&Hutzler2006;Heppenstalletal.2007;Stonedahl&Rand2014)andsensitivityanalysis(Stonedahl&Wilensky2010).Thequery-basedmodelexploration(QBME)paradigmprovidedbyStonedahlandWilensky(2011)expandsMiller's(1998)applicationofGAinABMoutputexploration.InQBME,parametersproducinguser-specifiedmodelbehavioursarediscoveredthroughautomationsuchasGAs,thusinvertingthetraditionalworkflow(seeFigure3).StonedahlandWilensky(2011)demonstrateQBMEforidentifyingconvergence,divergence,temporalvolatility,andgeometricformationsinmodelsofcollectiveanimalmotion(i.e.,flocking/swarming).

http://jasss.soc.surrey.ac.uk/18/4/4.html 6 25/01/2016

Figure3.Query-BasedModelExploration(QBME).ThisframeworkforexploringABMparameter-spacesexploitsgeneticalgorithms(orothermeta-heuristicsearchalgorithms)toefficientlysearchforparametersthatyieldadesiredmodelbehaviour.

Source:StonedahlandWilensky(2011)

Sensitivityanalysis:Approachesandchallenges

3.6 Sensitivityanalysis(SA)isavariationofparameter/input-outputspaceexplorationthatfocusesonmodelresponsetochangesintheinputparameters(Figure4).Specifically,theresearcherseekstoidentifyparametersforwhichsmallvariationsmostimpactthemodel'soutput.

Figure4.Uncertaintyandsensitivityanalysisaspartofthemodellingprocess.Source:Ligmann-Zielinskaetal.(2014)

3.7 Thisdiscoverycanaidinprioritizingprospectivedatacollectionleadingtoimprovedmodelaccuracy,reductionofoutputvariance,andmodelsimplification(Ligmann-Zielinskaetal.2014).Modelinsensitiveparametersmayevenberelegatedtomerenumericalconstantstherebyreducingthedimensionalityoftheinputparameterspaceandpromotingmodelparsimony;thissimplificationprocessisreferredtoas"factorfixing"(Saltellietal.2008).Ignoringthesenon-influentialinputparameterscanhaveill-effectsonthemodelbyincreasingitscomputationalcostandalsoonitsreceptionwhentheseparametersarecontroversialforstakeholdersnotinvolvedwiththemodel'sdevelopment(Saltellietal.2008).

3.8 TheproliferationofvariousSAmethodsstemsfromthevarietyofABMstylesandresearchproblemsABMsaddressaswellasfromtheavailabilityofincreasingcomputationalcapacity(Hamby1994;Saltellietal.2000).Currently,theABMpracticeofSAhasentailedoneormoreofthefollowingmethods:one-parameter-at-a-time,elementaryeffects,standardizedregressioncoefficients,meta-modelling,andvariance-baseddecomposition(Thieleetal.2014;tenBroekeetal.2014).Below,webriefly

http://jasss.soc.surrey.ac.uk/18/4/4.html 7 25/01/2016

discusseachofthese,theiradvantages,anddrawbacks,inthecontextofABMs.

3.9 Inone-parameter-at-a-time(OAT),eachinputparameterinturnisexaminedoverasetofvalues(definedeitherexantetotheSAordynamicallyduringSA)andinisolationbyholdingtheotherparametersataconstantbaseline.Meanwhile,theeffectsofthesemarginal(i.e.,one-at-at-time)parameterchangesaremonitored,andrepeatediterationsincreasetheprocedure'srobustness.Hassani-MahmooeiandParris(2013),forexample,appliedOATtotheirABMofmicro-levelresourceconflictstoidentifypreferableinitialconditionsandtoevaluatetheinfluenceofstochasticityonthemodel;thesimilarityofoutcomeswithinathresholddemonstratedthemodel'sinsensitivitytorandomness.OAT'ssimplicitywhileattractivealsoexposesitslimitationsinABMSA(Ligmann-Zielinska2013).Forone,theimpactfulandrelevantvaluesforeachinputparametermaybeaprioriunknownthusrenderinganyprioritizationofparametersdifficultandthesearchforkeyparametricdriversinefficient.Also,themarginalnatureoftheparametersearchspacesurroundingthebaselineobscuresparameterinteractionsandseverelyshrinkstheinputhypercubewithlargerparametersets.Forexample,withasfewas10inputparameters,OATcoversonly0.25%oftheinputspace(forageometricproof,seeSaltelliandAnnoni(2010)).

3.10 Elementaryeffects(EE)expandsonOATbyrelinquishingthestrictbaseline.Thatis,achangetoaninputparameterismaintainedwhenexaminingachangetothenextinputratherthanresumingthebaselinevalue(asdoneinOAT).Passingovertheparametersetismultiplyrepeatedwhilerandomlyselectingtheinitialparametersettings.TheseperturbationsintheentireparameterspaceclassifyEEasglobalSA(Saltellietal.2008).OriginallyproposedbyMorris( 1991)andimprovedbyCampolongoetal.(2000),EEissuitedforcomputationallyexpensivemodelshavinglargeinputsetsandcanscreenfornon-influentialparameters.

3.11 GlobalSAmayalsobeperformedthroughestimationofstandardizedregressioncoefficients(SRC),whichinitsbasicformsuccinctlymeasuresthemaineffectsoftheinputparametersprovidedtherelationshipsbetweentheparametersandtheoutcomesareprimarilylinear.Astandardizedregressioncoefficientexpressesthemagnitudeandsignificanceoftheserelationshipsaswellastheexplainedvariance.Moreprecisely,thesquareofthecoefficientisthevarianceexplained.

3.12 OneglaringlimitationofSRCisitsabilityhandlespatialinputs(ofspatialABMs)unlessasmallsetofscalarsorindicescansufficientlyserveasproxiesforentiremaps(Lilburne&Tarantola2009).Also,SRCcanexposelower-ordereffectsbutnotcomplexinterdependencies(Happeetal.2006).

3.13 Meta-modelling(oremulation)canaddressthelow-orderlimitationsofSRC.Ameta-model(emulator,amodelofamodel)isthesurrogaterepresentationofthemorecomplexmodel(likeABM)createdinordertoreducethecomputationaltimeofthesimulationsnecessaryforSA.Forexample,Happeetal.2006collatedmodelresponsesfroma2kfactorialdesignonarelativelysmallsetofparametersandfittedasecond-orderregression.Meta-modellingcanbecomputationallyefficientandnotnecessarilyrequirelargeamountsofdata.Forevenhigherordereffects,meta-modellingmethodssuchasGaussianprocessemulatorsarerequired(Marreletal.2011).

3.14 Variance-basedSA(VBSA)isconsideredthemostprudentapproachforevaluatingmodelsensitivitiesasitdoesnotassumelinearity(Ligmann-Zielinska&Sun2010;Fonoberovaetal.2013;Tang&Jia2014).InVBSA,thetotalvarianceofagivenoutputisdecomposedandapportionedtotheinputparametersincludingtheirinteractions(Saltellietal.2000;Lilburne&Tarantola2009).Twoindicesperinput(i)aredrawnfromthepartialvariances:afirst-order(maineffects)indexSiandatotaleffectsindexSTi.Siistheratioofi'spartialvariancetothetotalvariance.STiisthesumofallnon-iindices(∑S − i)andcapturesinteractionsbetweeniandtheotherinputs(Homma&Saltelli1996).Sialoneissufficientfordecomposingadditivemodels,soVBSAisunnecessaryformodelsknowntoproducelargelylinearoutputascalculationoftheindexpairsiscomputationallyexpensiverequiringlargesamplesizes(Ligmann-Zielinska&Sun2010);forexample,VBSAforkparametersrequiresM(k + 1)modelrunswhereM > 1000.Thiscomputationalloadmaybereducedthroughparameter(orfactor)grouping(Ligmann-Zielinska2013;Ligmann-Zielinskaetal.2014),parallelization(Tang&Jia2014),orquasi-randomsampling(Tarantola&Zeitz2012).Situationsinwhichinputsareexogenouslycorrelatedmayresultinnon-uniqueVBSA(Mara&Tarantola2012).Whilerecentmethodsaddressingthesedependencieshavebeendeveloped(Kucherenkoetal.2012;Mara&Tarantola2012;Zunigaetal.2013),theystillrequireelaborateDOEcoupledwithaverylargesamplesizeandalgorithmiccomplexity.

3.15 Geneticalgorithms(GAs)mayalsobeusedforSA(Stonedahl&Wilensky2010).Parametersarealteredunderthegeneticparadigmofreproductioninwhichpairsof"fitter"parametersetsexchangesubsets.Thefitness(orobjective)functionmaybetailoredtoexposemodelsensitivitiestoitsparameters(asopposedtocalibration).Forexample,StonedahlandWilensky(2010)allowedGAstosearchthrough12parametersofthe"ArtificialAnasazi"ABM(Deanetal.2000;Janssen2009)inordertoinduceresponsesdepartingfarfromtheempirical,historicalvalues,whileconstrainingthesearchtoalimitedrange( ± 10%oftheircalibratedsettings).SeeFigure18inAppendixDforaplotoftheoutlierresults.

3.16 TherelianceonmeanandvariancefordistributionalinformationinmanyofthepracticedmethodsofSAisinsufficientformorecomplexdistributions.Futureresearchshouldinvestigatemoment-independentmethods(Borgonovo2007;Baucells&Borgonovo2013).

Visualizationinsensitivityanalysis

3.17 Visualizationsoftheinputparameter-outputrelationshipsareanintegralpartofSA.Inscatter-plots,theserelationshipsaredirectlyandsimplyplottedpotentiallyrevealingdependencies(seeFigure5).WeusedthesimpleSchellingsegregationmodel

http://jasss.soc.surrey.ac.uk/18/4/4.html 8 25/01/2016

(Schelling1969)ofredandgreenagentsona100x100gridimplementedinAgentAnalyst(http://resources.arcgis.com/en/help/agent-analyst/).Themodelcontainsfouruniformparameters(lowerandupperboundsinparentheses):numberofagents(3000,6000;discrete),tolerancetoagentsofdifferentcolour(0.2,0.5),randomseed(1,10000;discrete),andpercentofgreenagents(10,50).Wemeasuredagentmigration(totalnumberofagentmovesduringmodelexecution)forN = 1280modelruns.

Figure5.Sensitivityanalysiswithscatterplots.Scatterplotsoftotalmigrationversusthefourmodelinputsatt = 100;notethatnumberofagentshasmoreinfluenceonthevariabilityoftotalagentmigrationthantheotherinputs.

3.18 Typically,they-axisandx-axis(foratwo-dimensionalscatterplot)expressvaluesforanoutputandaninputparameter,respectively.Scatterplotsareespeciallyusefulwhenthedependenciesarestructured,theoutputisascalar,andthenumberofmodelparametersislimitedallowingforanunencumberedenumerationofparametersandoutputcombinationsinseparatescatterplots.Morecomplexmodelbehavioursrequirealternativevisualizationstylessuchaspiechartstodisplayvariancepartitions(seeFigure6).

Figure6.Sensitivityindicesobtainedfromdecompositionofmigrationvarianceatt = 100.Notethatalmost30%ofmigrationvarianceiscausedbyinteractionsamonginputs–mainlytoleranceandnumberofagents.

3.19 Asnapshotofvariancedecompositionattheendofthemodelrun(e.g.,Figure6)maybeinsufficientinassessingtheimportanceofparametersandconsequentlytheirprioritization(Ligmann-Zielinska&Sun2010).Thus,visualizingvariancedecompositiontemporallywillrevealparameterstabilityoverthecourseofthemodelrun(seeFigure7).Spatialoutputssuchaslandusechangemapsmayalsoreceivesimilartreatmenttorevealtheextentofoutcomeuncertaintyinregions(orclusters)duetospecificparameters(Ligmann-Zielinska2013).

http://jasss.soc.surrey.ac.uk/18/4/4.html 9 25/01/2016

Figure7.Timeseriesofsensitivityindicesobtainedfromdecompositionofmigrationvariancemeasuredovertime.Theexampledemonstratesthatparametersensitivitiescanconsiderablyvaryduringsimulation,withnumberofagentsdominatingthe

varianceatt = 10,andtolerancedominatingthevarianceatt = 100.

3.20 SAformultipleoutcomevariablesinanABMincursadditionalchallengesduetodifferencesineachparameter'simpactontheoutcomes(seeFigure4).Thisissueisofparticularconcernformodelsimplificationanddemandseitheronesingle,well-chosenoutcomethatisadequatelyrepresentativeofthemodel'sbehaviourormoreconservatively,thedifficulttaskofundergoingSAacrossthewholespectrumofoutputs:scalars,time-andspace-dependentmeasures.

Spatio-TemporalDynamics

TemporalandspatialdimensionsofABMoutputdata:challenges

4.1 Asastochasticprocess,anABMgenerates(oriscapableofgenerating)timeseries(TS)data.Toaslightlylesserextent,ABMsalsooperatewithintopologicalboundariesoftenexpressedasaspatiallandscape,whetherempirical(e.g.,Parkeretal.2002;Bousquet&LePage2004;Heppenstalletal.2012;Filatova2014)orstylized(e.g.,earlyusebySchelling1969,1978;Epstein&Axtell1996).SimilartootherABMoutcomes,boththeTSandspatialdataareborneoutofcomplexendogenousdynamicsoverwhichthemodellerexertsfullcontrol.Thus,itisrarelythecasethattheoutputdataisproducedbyasinglecomponentofthemodel.Instead,mostofABMTSandspatialoutputrecordedembodythelonglistofuniquefeaturesthatABMsexhibitsuchasemergenceratherthanaggregationatthemacro-level,interactionratherthanreactionatthemeso-level,andnon-linearityratherthanlinearityofprocessesanddecision-makingatthemicro-level.SpatialmapsgeneratedbyABMsmayalsocaptureeventualspatialexternalities,path-dependencies,andtemporallageffects.ThesecharacteristicsrendertheanalysisofABMoutputsless

appropriateformoretraditionaltools.TheseissuesderivefromthereasonsABMareusedinthefirstplace.[4]WeconsiderthischallengingnatureofTSandspatialoutputanalysisinthefollowingsections.

Time:approachesandvisualizationtechniques

TimeseriesgeneratedbyABMs(ABMTS)representamyriadoftemporaloutcomes,suchasevolvingagentcharacteristics(e.g.,sociodemographics,utility,opinions);agentbehaviours/decisions(e.g.,strategies,movements,transformations);ormeasuresdescriptiveofthemodelstate(e.g.,agentpopulationorsubpopulationcounts).Alltheseareoftenpresentedassimplelinegraphsrepresenting1)outcomesofindividualagentsofspecialinterest(Squazzoni&Boero2002,Fig.9)or2)aggregatedstatistics(suchasmeanormedian)overtheentireagentpopulation,subgroups,oranindividual(withmeasurementsoverseveralruns)undervaryinglevelsoftemporalgranularity(e.g.,amovingwindowcoveringseveraltimepoints)(Izquierdoetal.2008).ComparisonsofmultipleTSareoftenfacilitatedbytheinclusionofconfidenceintervals(Raczynski2004,Fig.4)andoccasionallyperformedagainstexperimental/empirical(Richiardietal.2006;Boeroetal.2010,Fig.1)ortheoretical(Takahashi&Terano2003,Fig.4)outcomesorexpectations(Angus&Hassani-Mahmooei2015).TScomparisonshavealsobeenperformedforcalibrationpurposes(Richiardietal.2006).

4.2 DespitetheproliferationofABMs,effectiveTSanalyticaltechniquesremainunder-used(Grazzini&Richiardi2015).AngusandHassani-Mahmooei(2015)surveyedover100ABMpublicationsinJASSSandfoundveryfewinstancesofadditional(statistical)modellingofTSdata.Onlyinagent-basedfinancialmarketresearchdoesonefindrelevantTSstatisticalandeconometricanalysis(Yamada&Terano2009;Chenetal.2012;Neri2012).

4.3 Inthissection,wediscusselementsofTSanalysisinthecontextofABMsandpresentsomebasicandsomecompellingexamples,whilestoppingshortofexpoundingformalTSmodellingsuchasauto-regressivemodels.

http://jasss.soc.surrey.ac.uk/18/4/4.html 10 25/01/2016

4.4 AmongthetechniquesforanalysingABMTS,weconsidertimeseriesdecompositiontobeoneofthemoreusefulmethods.DecompositionentailspartitioningaTSintofourcomponents:trend,cyclical,seasonal,andrandomcomponents.Theprominenttrendisthemoststructuredofthesecomponentsanddepictsthelong-termlinearornon-linearchangeintheTSdata.Theseasonalcomponentexhibitsregularperiodicityduetosomefixedexternalcyclesuchasseasons,months,weeks,ordaysoftheyear.Exogenousmodeleventssuchasregularadditionsofafixedcountofagentstotheagentpoolarealsoconsideredseasonal.Cycleshavingirregularperiodicityconstitutethecyclicalcomponent.Finally,theresidualorrandomcomponentcapturestheunexplainedvariationremainingafterthepriorthreecomponentsarefilteredfromtheTS.

4.5 ThespectrumofTSanalysistechniquesrunsfromthecalculationofmovingaveragesandlinearfiltering(seeFigures8aand8b)tothemoresophisticatedexponentialsmoothingandautoregressivemodelling.Furthermore,ABMrelevantergodicitytestsmaybeappliedtoinferstationarityofstatisticalmoments(e.g.,mean,variance,skewness,kurtosis,etc.)henceequilibriumofthosemomentsacrossapoolofsimulationruns(Grazzini2012).

4.6 ForanexampleofABMTSanalysis,weexaminetheoutputofthewell-knownEl-Farol"barpatron"game-theoreticABM(Rand&Wilensky2007).Theinitialconditionsare:memorysize=5,numberofstrategies=10,overcrowdingthreshold=50.Thenumberagentsincreasesby2%every52steps.Figure8apresentstheoriginaldataanditslinearincreasingtrend(inred).SubtlestructureintheTSisexposedwhenweplotamovingaveragealongwithitsdecomposedtrend(obtainedviaexponentialsmoothing)(Figure8b).

(a)Lineartrend(b)Movingaverage

http://jasss.soc.surrey.ac.uk/18/4/4.html 11 25/01/2016

(a)Lineartrend(b)MovingaverageFigure8.TimeSeriesofBarAttendanceinElFarolABM.Theblackandgreyseriesrepresenttheoriginaldata;theredlineisthelinearfit(left)ordecomposedtrend(right);andthebluelineisamovingaverage.Theuppersubfiguresarelabelled(a)and

(b).Thelowersubfiguresare(c)and(d).

4.7 Themodelbehaviourisaugmentedtoincludeanarbitrary,one-timeincreaseinpopulationincreasewhilethegradualincreaseisreducedto0%.Whilethelineartrendforthenewresults(Figure8c)coarselycapturesthepopulationincrease,thesuddenchangeismadestarklyvisibleusingamovingaverage(plusdecomposedtrend)(Figure8d).

4.8 ComparisonsbetweenTSdrawnfromdistinctmodelparameterizationscanbeeasilyperformedthroughdirect(albeitnaïve)

visualcomparisonofthetwoseries,plottingthedifferences,orcalculatingtheirEuclideandistance[5]orcrosscorrelation.However,thesecomparisonapproachesareappliedtoexacttemporalpairwisedataandthusfailtoaccountforthecomplexitiesofABMsthatmayproduceTSthataredissimilaronlythroughinterspersedlags.InadditiontoRichiardi's(2012)suggestionsforrobustTScomparison,weadvocatetheuseofdynamictimewarping(DTW)toaddresstheabovecomplication(Keogh&Ratanamahatana2005).DTWisnowextensivelyusedinareassuchasmotionandspeechrecognition.UsingastylizedpairofTS,Figure9depictstheeffectivenessofDTW,whichidentifiescomparablepairsofdataoccurringatdifferingtimescales.

Figure9.Comparingdistance(similarity)measurementmethodsbetweenEuclideanDistance(left)andDTW(right)

4.9 TofurtherdemonstrateDTW'seffectiveness,wecompare40distinctparametersettingsinEpstein'sABMofcivilviolence(Epstein2002;Wilensky2004).Theprimarydistinctionistheinitialcopdensitywhichrangesfrom4.05and5.00inincrementsof0.05%andresultsinvariedsizesofthepopulationofquietcitizens,ouroutcomemeasureofinteresthere.Weobtain10TSsamples(i.e.,modelruns)underNetLogoeachhavingadurationof200ticks/timepoints.InFigure10,wepresentfourindividualrunsinordertohighlightthedifficultyofdirectvisualcomparison.

http://jasss.soc.surrey.ac.uk/18/4/4.html 12 25/01/2016

Figure10.ComparingTimeSeriesfromDistinctParameterizationsofQuietCitizensofCivilViolenceABM

4.10 Whiletheoutcomesareinitiallysimilar,theyrapidlydivergesharingonlythecharacteristicoftheoutcomes'exhibitingsomefluctuation.Thetworunsunderthesameparametersetting(4.1%)naturallydivergeduetodistinctrandomsequences(fromaRNGusingdistinctrandomseeds).

4.11 ComparisonsofallpairsoftheexperimentalconditionsarepresentedinFigure11.Thesecomparisonsareperformedonnormalizedoutcomesaveragedoverthesuiteofruns.ThedistancemeasurementsintheleftmatrixaredirectcorrelationsandEuclideandistancesintherightmatrix.ThecellsintheuppertriangleofeachmatrixcorrespondtodirectcomparisonoftheTSwhilethoseinthelowertriangleindicatesthemeasurementsunderDTW.Therowandcolumnsdenoteincreasingcopdensities.

Figure11.ComparingDTWagainstnon-DTWforcorrelationandEuclideandistance

4.12 ThefiguresillustratethesuperiorityofDTWincapturingTSdifferencesunderthesemeasurements,overthedirectuseofexacttemporalpairs.DTWclearlyexposesgreatersimilarityoftheoutcomes'TSwhentheexperimentalparameters(copdensity)arealsosimilarwhereasthedirectmeasurementsdonot.Thisrelationshipislargelymonotonicasonewouldexpect.Thus,DTWisappropriateformodelsforwhichtheoutcomeTS'sstructure(specifically,bothitsseasonalandirregularperiodicities)isgreatlyaffectedbytheexperimentalconditions.

4.13 Finally,TSanalysisprovidesahighlyinformativeopportunitytopreciselyestimatetheimpactofchangesintheinputvariablesontheoutputsofanABMmodel.Techniquessuchaspaneldata(orlongitudinal)analysis,whichtakeintoaccountbothTSandcrosssectionalcomponentsofthedata,canenabletheagent-basedmodellerstouncoverrobustevidenceonhowmodelbehavioursareassociatedwiththechangesinthevariablesoftheagentsand/orthemodelovertime.OtherprominentcomponentsofTSanalysis(suchasforecasting,classificationandclustering,impulseresponsefunction,structuralbreakanalysis,laganalysis,andsegmentation)mayalsobeusedalongwithestimationandauto-regressivemethodsinordertoprovideabetterunderstandingofseriesgeneratedbyABMs.

ProcessingspatialABMoutput:approachesandvisualizationtechniques

4.14 ThespatialenvironmentinABMsvaryfromcellulargrids(inwhichonlyinter-agentdistancematter)torasterorvectorrepresentationsofmultiplelayersofarichGISdata.LocationsinaspatialABMmayrelatetooutputmetricsattheindividualoraggregatedagentlevel(e.g.,income,opinion,orastrategy)orotherspatialqualities(e.g.,land-usecategories).ItischallengingtoseekpatternsandtocompareacrossexperimentalconditionsinasearchforacompellingnarrativewhilescreeningthroughhundredsofmapsproducedbyanABM.SpatialABManalysisisofteninformedbymethodsfromgeographyandspatialstatistics/econometrics.Here,wereviewsomespatialmetricsandvisualizationapproachesforspatialanalysis.

4.15 QuantitativeindicessuchastheKappaindexofagreement(KIAorCohen'sκ)havebeenwidelyemployedforcell-by-cellcomparisonofABMoutcomesonspatialmaps(Manson2005).Morerecentworksuggestalternativestotheκsuchasamovingwindowalgorithm(Kuhnertetal.2005)andexposeitslimitations(Pontius&Millones2011).Pontius(2002)proposesmorecomprehensivemethodsandmeasuresfortrackinglandusechangesandcomparingmapsundermultipleresolutions(fromcoarsetofine).

4.16 Variousspatialmetricsareoftenusedtomeasureland-usechangeanddetectspatialpatternssuchasfragmentationandsprawl(Parker&Meretsky2004;Torrens2006;Liu&Feng2012;Sunetal.2014).Thesemetricsincludemerecountingofland-usecategories,landscapeshapeindex,fractaldimension,edgedensity,aswellasadjacency,contiguity,andcentralityindexes.Moran'sIisanotherspatialautocorrelationstatisticindicatingtheextentofdispersionorclustering(Wu2002).Millingtonetal.(2008)usedacontagionindexalongwithbasicpatchcountmetricstoidentifyfragmentation.Griffithetal.(2010)usedGetis-OrdGi*"hotspot"analysistoidentifystatisticallysignificantspatialclusteringofhigh/lowvalueswhileanalysingthespatialpatternsofhominids'nestingsites.ZinckandGrimm(2008)usedspatialindices(shapeindex,edgeindex)andbasicmetrics(countsandareasofdiscrete"island"regions)tosystematicallycompareempiricaldatatosimulationresultsfromtheclassicDrossel-

http://jasss.soc.surrey.ac.uk/18/4/4.html 13 25/01/2016

SchwablforestfirecellularautomataABM.SoftwareofferingthesemetricsincludeC++Windows-based'FRAGSTATS'andthe'SDMTools'package(fortheplatform-independentR),thefunctionalityofwhichmaybeaugmentedbyBio7andImageJforimageprocessing.

4.17 Inanotherexample,Sunetal.(2014,Table6)reportedbasicstatistics(i.e.,meanandstandarddeviation)onninespatialmetricsestimatedovermultipleparametersettingsforvaryinglevelsoflandmarketrepresentationintheirurbanABM.Thesignificanceofmeandifferences(acrossconditions)wereassessedwiththeWilcoxonsignedranktest.Meansofthemetricswerejointlyvisualizedacrossallparametersettingsinalinegraphstylecalled"comprehensiveplotting"(Sunetal.2014,Figures5–8),whichthenallowsforidentificationofconditionsproducingoutlyingbehaviourfacilitatingvisualsensitivityanalysis.

4.18 Obviously,statisticalmodelssuchasregressionsandANOVAsmaybeemployedtorelatemodelparametersandoutcomestospatialoutcomes(e.g.,Filatovaetal.2011).However,dependenciesamongspatiallydistributedvariablesrequirespecialtreatmentintheformofaweightedmatrixincorporatedintoaspatialregressionmodelasapredictororaspartoftheerrorterm.Localesmaybedisambiguatedfurtherinstatisticalpredictionthroughgeographicallyweightedregression(GWR).Thesemethodsalsofallundertheauspicesofspatialeconometrics.

ReportingandvisualizingABMresultsovertimeandspace

4.19 Typically,spatialABMoutputisrenderedastwo-dimensionalmaps.Acollectionofthesecanhighlightmodeldynamism/progressionorallowforcomparisonofmetricsandexperimentalconditions(Parry&Bithell2012,Figure14.11)orexposekey,informativetrajectories(e.g.,Barros2012,Figure28.3).3Dviewsarealsousedtoportrayoutputsparticularlyinevacuationandcommuting(Patel&Hudson-Smith2012).Often,visualinspectionofthedataoffersfacevalidationandhigh-levelinference.Forexample,aspatialoverlayiscommonlyusedtoanalyserasteroutputsandevaluatethespatialdistributionofmultiplemodelbehavioursandoutcomes.

4.20 ThespatialdistributionofoutcomesissubjecttothestochasticandpathdependentnatureofABMsandoftendemandaggregationforeffectivepresentationofmodelbehaviour(Brownetal.2005).Naturally,ameanwithaconfidenceintervalforanoutcomeineachspatiallocationcanbesufficientforreportingasetofmaps(overtimeoracrossexperimentalconditions)(e.g.,Tameneetal.2014).Alternatively,afrequencymapforasingleparametersettingrevealseachlocation'sstatetransitionprobabilities(asaproportionoftotalsimulationruns)(Brownetal.2005).Thesetransitionsmayeasilybeportrayedasacolour-gradientmap(e.g.,Magliocca2012).PlantingaandLewis(2014)warnagainstconstructingdeterministictransitionrulesoutoftheseprobabilities.Brownetal.(2005)alsosuggestdistinguishingareasofnon-transition(orinvariantregions)fromvariantregions;themethodforidentifyingtheseareasiscalledthevariant-invariantmethod.

4.21 Judiciousselectionofatemporalsequenceofmapscanrevealmodeldynamics.Softwaresuchasa"mapcomparisonkit"(RIKSBV2010)canperformautomatedteststoidentifytheextenttowhichtworastermapsaredifferent.Pontiusetal.(2008)extendsthecomparisonexercisetoincludethreemapswhileconsideringpixelerrorandlocationerror.Comparisonsincludeallpairsofsimulationoutputsattimes1and2,a"true",referencemapattime2,andallthreemapsjointly.

4.22 WhileshowcasingamapasanoutputofanABMisalwaysappealing,supplementingitwithsummarystatisticsofspatialpatternsallowsfordeeperunderstandingofexperimentaleffectsandconsequentlythemodel'sbehaviour.Inadditiontothespatialmetricsdiscussedin4.3,quantitiesoflanduseandconversionsmaybereportedashistogramsacrossdifferentscenariosandland-usetypesovertime(e.g.,Figure4inVillamoretal.(2014)).

4.23 Color-gradientsinlandscapevisualizationscanbeusedtorepresenttemporalchangesinametricofinterest.Filatova(2014)employsthesespatio-temporalchangegradientstopresentlandpricechanges(duetomarkettradesinanABMfocusingoncoastalproperties)betweentwopointsintimeinanempiricallandscape(Figure12).

Figure12.Changesinpropertypricesovertime

4.24 Theblueandredgradientscorrespondtothevalenceofthechange(i.e.,fallingandrisingprices,respectively);theirdarknessor

http://jasss.soc.surrey.ac.uk/18/4/4.html 14 25/01/2016

intensityindicatelevelofchange.Suchvisualizationsmayoffereasyidentificationofclustersandboundaries.

4.25 AnotherobviouswaytodepictthedifferencesinspatiallydistributedABMoutputdataistovisualizein3Dplots.The3rddimensionclearlypermitstheinclusionofadditionalinformationsuchastimeoroneofagents'innerattributes.Forexample,Huangetal.(2013)employed3Dcoloredbarplots(Figure13)inwhichthe2Dlayoutcorrespondstophysicallocation.

4.26 Such3DvisualizationsofferinsightsintotheABM'sdynamicsorhighlightvitalstructuraldifferencesacrossexperiments.Forexample,DeardenandWilson(2012)usea3DsurfacetoplotthemacrometricsofinterestoftheirretailABMasafunctionofdifferentvaluesofthetwomostcriticalparametersaffectingagents'choices.3Dsurfacesoverdifferentparametersspacescanalsobecomparedwitheachother,asdemonstratedbyDeardenandWilsonintheircomparisonsofactivitywithinconsumerandretaileragentclasses.

Figure13.Priceoflandandsequenceoftransactionsundervariousmarketconditionswhenagentshaveheterogeneous

preferencesforlocation.Left:Allocationoflandonthismarketisonlypreferencedriven.Right:Landallocationhappensthroughcompetitivebidding.Thehigherthebar,thehigherthelandprice.coloursdenotethetimewhenlandwasconvertedintourban

use.Source:Huangetal.(2013)

4.27 Inthisexample,thecoloraddsafourthdimension(timeofanevent)totheconveyedinformation.Furthermore,functionalshapesaremoreeasilydiscerniblein3D;inthiscase,theuppersectionofFigure13bmightbefittedtoaparaboloidorasimilarsolidofrevolution.

4.28 Anotherefficientuseof3DvisualizationforABMswithgeo-spatialelementsisshowninFigure14.Malleson'smodelofburglaryinthecontextofurbanrenewal/regenerationcomprisesagentswhotraveltocommitcrimeinanurbanlandscape(Mallesonetal.2013).

Figure14.Spatio-temporalTrajectoriesofBurglars.Source:NicolasMalleson.http://nickmalleson.co.uk/research

4.29 Thecolored,segmentedlinesinthefigurewhenoverlaidontothe2Dcitymapdepictthetrajectoriesofburglaragentsseekingtargetswhilethez-axisdenotesthetemporaldimensiontotheirjourney.Thisjointportrayaloftimeandspacesuccinctlycommunicatesimportantfeaturessuchaskeylocationsofactivityfortheagents(i.e.,wherelinesappearvertical)aswellastheoriginoftheagentanditsdestination,inthiscaseapresumably"safe"location.

4.30 Finally,giventhedynamicnatureofABMs,theirspatialandtemporaloutputsareappropriatefordynamicpresentationmodalities.Interactive3Dvisualizationsoffertheabilitytoexaminethedatafromalternativeperspectives.Increasingly,video(oranimation)hasbeenusedtocaptureABMbehaviourasameanstocommunicateABMoutputtobothpractitionersaswellasthe

http://jasss.soc.surrey.ac.uk/18/4/4.html 15 25/01/2016

broaderlay-audience.[6]ModelbehaviourmayalsobecapturedasanimatedGIFfiles,thesmallsizesofwhichfacilitatetheiruseinpresentationsandwebpages(Lee&Carley2004;Lee2004).

Discussionandconclusions5.1 WhileABMasatechniqueoffersmanyexcitingopportunitiestoopenresearchfrontiersacrossarangeofdisciplines,therearea

numberofissuesthatrequiresrigorousattentionwhendealingwithABMoutputdata.ThispaperhighlightstheoutstandingcomplexitiesintheABMoutputdataanalysisandconsolidatescurrently-usedtechniquestotacklethesechallenges.Inparticular,wegrouptheminto3themes:(i)Statisticalissuesrelatedtodefiningthenumberofappropriaterunsandhypothesistesting,(ii)Solutionspaceexplorationandsensitivityanalysis,and(iii)ProcessingABMoutputdataovertimeandspace.WealsobrieflydiscussstakeholderinvolvementinABMresearch (iv)below.

5.2 Statisticalissuesrelatedtodefininganumberofrunsandhypothesistesting:ForanalysingABMs,thecalculationofstatisticsfrommodeloutcomesacrossmultiplesimulationrunsisrequired.However,thestatisticalmethodsarechallengedbybothaplethoraofABMoutputdataforwhichtraditionalstatisticaltestswillexposeminuteeffectsandcomplexABMsforwhichrunsarecostly.Intheformerscenario,statisticalmethodsneedtobetempered(e.g.,amorecriticalp-value)oranacceptableceilingonthenumbersamples(orruns)shouldbeenforced.Forthelattercase,apredeterminationoftestsensitivity(e.g.,effectsize)mustbemadebeforecalculatingaminimumnumberofruns.However,thestabilityofoutcomevarianceneedstobesecured,andwedemonstrateandreviewapproachesforestimatingthepointatwhichthisisachieved.WealsorevealthatthetraditionalapproachtodeterminingminimumsamplesizeissensitivetotheshapeofthedistributionandwesuggestempiricalestimationofthepowerleveloruseofthemoreconservativeWilcoxonranksumtest.Anotherchallengeistheanalysisofmanyinfluencingvariables.Theanalysisofcomplexinterdependencieswithinasimulationmodelcanbeaddressedbysystematicdesignofexperimentprinciples,andunivariateanalysismaysupporttheanalysisbypre-definingparameterranges.Overall,ABMresearchersshouldbeawareofthestatisticalpitfallsintheanalysisofsimulationmodelsandofthemethodsdescribedtoaddressthesechallenges.

5.3 Solutionspaceexplorationandsensitivityanalysis:AnABMcannotbeproperlyunderstoodwithoutexploringtherangeofbehavioursexhibitedunderdifferentparametersettingsandthevariationofmodeloutputmeasuresstemmingfrombothrandomandparametricvariation.Accordingly,itisimportantforABManalystsandresearcherstobefamiliarwiththerangeofmethodsandtoolsattheirdisposalforexploringthesolutionspaceofamodel,andfordetermininghowsensitivemodeloutputsaretodifferentinputvariables.ABMsposeparticularchallengesforSA,duetothenonlinearityofinteractions,thenon-normalityofoutputdistributions,andthestrengthofhigher-ordereffectsandvariableinterdependence.Whilesomemodelanalysesmayfindsuccessusingsimple/classicSAtechniques,practitionerswoulddowelltolearnaboutsomeofthenewerandmoresophisticatedapproachesthathavebeen(andarebeing)developedinanefforttobetterservetheABMcommunity.

5.4 ProcessingABMoutputdataovertimeandspace:WhileeveryABMhasthepotentialtoproducehighresolutionpaneldataonaggregatedandagent-levelmetricsoverlongtimeperiods,thestandardtimeseriestechniquesarerarelyapplied.WearguethattheuseoftimeseriestechniquessuchasdecompositionandmovingaveragesanalysisnotonlyimprovethescientificvalueofABMresultsbutalsohelpgainingvaluableinsights–e.g.,theemergenceofthetworegimesinthedataovertime–thatarelikelytobeomittedotherwise.TheuseofEuclideandistancesimilaritymeasurementanddynamictimewarpingoffershighutilityespeciallywhentemporallyvaryingoutputsneedtobecomparedbetweenexperimentsorinasensitivityanalysisexercise.Whendealingwithspatialdataanalysis,ABMresearchersactivelyusemethodsdevelopedingeographysuchasspatialindexes,mapcomparisontechniques,seriesof2Dor3Dmaps,2Dhistograms,andvideos.Inaddition,ABM-specificmethodsarebeingactivelyproposed–suchas3Dhistogramsreflectingtemporalchangesoveraspatiallandscape,spatio-temporaloutputvariablechangegradients,aswellasoverlayingtemporalABMdynamicsovera2Dmap.

5.5 CommunicatingABMresultstostakeholders:TheutilityandeffectivenessofmanyABMsandtheiroutputsareoftenjudgedbythemodel'sconsumers:theuser,thestakeholder,ordecision-maker.Thus,aqualitativeunderstandingofthemodelisessentialasmodelacceptanceandadoptiondependstronglyonsubjective,qualitativeconsiderations(Bennettetal.2013).

5.6 TheclarityandtransparencyofABMmechanismsfacilitatestakeholderinvolvementinthemodellingprocess.Thisparticipatorymodellingisapowerfulstrategytofacilitatedecision-making,management,andconsensusbuilding(Voinov&Bousquet2010).IncontrasttoothertechniquessuchasSystemDynamics(SD),ComputationalGeneralEquilibrium,orIntegratedAssessmentModelling,ABMrulesareexplicit,directlyembeddedinthemodel,anddonotnecessarilyhavetobeaggregatedorproxiedbyobscureequations.Thus,ABMshavebeenhistoricallyattheforefrontofparticipatorymodelling.Communicativegraphicaluserinterfaces(GUIs)inplatformssuchasNetLogoandCormashavealsocontributedtotheclarityinpresentationandeaseofinterpretation(astheyhavedoneforSD'sStellaorVensim).AcloneofparticipatorymodellingwithABMsconceivedbyFrenchmodellersbecameknownascompanionmodelling(Bousquetetal.1999;Barreteauetal.2003;Étienne2014)andisappliedgloballyparticularlyindevelopingnations(Becuetal.2003;Campoetal.2010;Worrapimphongetal.2010).Asthefocusisonthemodelasaprocessratherthanaproduct(Voinov&Bousquet2010),co-learningbetweenstakeholdersandmodellersresultsinexpedientcyclesofmodelling,outputpresentationanddiscussion,andsubsequentamendmentofthemodel.

5.7 Moss(2008)notesthatevidenceshouldprecedetheory,whenevermodellingbecomesembeddedinastakeholderprocess.Thus,interpretationofmodeloutputsrequiresmorethanmerequantitativeevaluationandinterpretation,andthenecessarytaskofweighingmodeloutputsagainstvaluesandperceptionsofbothstakeholdersandmodellersalikecontinuestochallengeus

http://jasss.soc.surrey.ac.uk/18/4/4.html 16 25/01/2016

(Voinovetal.2014).

5.8 Directionsforfutureresearch:Mostofthechallengesandtechniquesconsideredinthispaperarequitecomputationallyintensive.Yet,thefactthatananalysisofABMoutputdataoftenrequiresmoretimeandattentionthanthedesignandcodingofanABMitselfisstilllargelyunderestimatedespeciallybyamateursintheABMfield.Thus,user-friendlysoftwareproductsthat

supportdesignofexperiments(e.g.,theAlgDesignpackageinR(Wheeler2011)),[7]parameterspaceexploration,sensitivityanalysis,temporalandspatialdataexplorationareinhighdemand.Forexample,theMEMEsoftwareisonesteptowardthisgoalandisavaluabletoolsforABMresearchers.ABMswouldideallysupportrealworlddecision-making,henceefficient,user-friendlyABMplatforms,supportingdataanalysisandvisualization,wouldreinforceuseofABMinparticipatorymodelling.Moreover,insightsintoadvancedstatisticaltechniquescouldassistinresolvingsomeofthestatisticalissuesdiscussedinSection2.

AcknowledgementsThismaterialisbaseduponworksupportedbyNWODIDMIRACLE(640-006-012),NWOVENIgrant(451-11-033),andEUFP7COMPLEX(308601).Furthermore,ourthanksgotoallparticipantsofWorkshopG2duringtheiEMSs2014ConferenceinSanDiego.

Notes

1Thispaperwasinspiredbyaworkshopatthe2014iEMSsconference:theG2workshop"AnalyzingandSynthesizingResultsfromComplexSocio-ecosystemModelswithHigh-dimensionalInput,Parameter,andOutputSpaces".Duringthatworkshop,severalkeyissuesfacingtoday'ssimulationmodellerswereidentifiedanddiscussed.Theonesdeemedtobemoreexigenthavebecomethefocusofthispaper.

2ThenormalorGaussiandistributionalsoknownasthe"bell-curve"isthemosteasilyrecognizedempiricaldistributioncontainingasinglemodeandoftencapturesmanynaturallyobservedoutcomes.Theuniformdistributionisaflat,artificialdistributionandcanbeconsideredtoserveasthecontroldistributionamongthisset.Theexponentialisoftenusedtomodelfailureratesandisamodalandskewed.ThePoissonexpressestheprobabilityofagivennumbereventsoccurringwithinaknowninterval.Theχ2(chi-squared)istypicallyemployedinstatisticaltestsaswellastheStudent'stdistribution.Weincludethesetwoastheyarereadilyrecognizablebymanypractitionersofappliedstatistics.

3TheeffectsizecalculationweemployisCohen'sd(Cohen1988):d =

μ1 −μ2spooled .

4LeBaronetal.(1999,p.1512),forexample,notethattheirartificialstockmarkethastimeseriescapturingphenomenaobservedinrealmarkets,includingweakforecastabilityandvolatilitypersistence.

5

Euclideandistance =

n

∑i =1(xi − yi)2

wherenisthenumberofdatapointsineachvector.

6Forexamples,weciteEpstein&Axtell(1996)andthecorrespondingvideo:https://www.youtube.com/watch?v=SAXWoRcT4NMandHelbingetal.(2005)andthecorrespondingvideo:https://www.youtube.com/watch?v=yW33pPius8E.

7FurtheroptionsinRforDOEmaybefoundathttp://cran.r-project.org/web/views/ExperimentalDesign.html.

AppendixA:MinimumSampleSizeforDistributions

√

http://jasss.soc.surrey.ac.uk/18/4/4.html 17 25/01/2016

Figure15.MinimumSampleSizeforThreeDistributions

Theblackcurvedenotestheempiricalpowerlevel.Theredlinedenotesthedesiredpowerlevel:1 − β = 0.80.Thesolidgreenverticallinedenotestheminimumsamplesizederivedfromthepowercalculationwhilethedottedgreenlineshowstheempirically-derivedsize.ThebluecurveandlinedenotethepowerandminimumsizeaccordingtothetwosampleMann-Whitney-Wilcoxontest.

Figure16.MinimumSampleSizesforBirthRateABM

Theredlinedenotesthedesiredpowerlevel:1 − β = 0.80.Thesolidgreenverticallinedenotestheminimumsamplesizefromthepowercalculation;thedottedgreenlineshowstheempirically-derivedsize;andthebluecurveandverticallinedenotethepowerandminimumsamplesizeaccordingtotheMann-Whitney-Wilcoxontest.

AppendixB:IssuesofHypothesisTestingThereexistsanongoingdebateovertheemphasisresearchersshouldplaceonsignificancetests.Alargesamplesizecaneasilyclassifyaminutedifferenceasbeingsignificant.Thus,many(intheABMfieldandoutside)argueforgreaterattentionpaidtowardstheeffectsizeitself(whetheritisCohen'sdorastandardizedregressioncoefficient)asthebenchmarkfora"significant"finding(Coe2002;Ziliak&McCloskey2008,2009;Sullivan&Feinn2012;Whiteetal.2013;Troitzsch2014).Infact,recentlythejournalofBasicandAppliedSocialPsychologyhasimplementedpolicytoremovep-valuesfromtheirpublications.Alternatively,researchersmayturntomethodsandmeasuresthatpenalize(orminimizetheimpactof)largesamplessizes.Rouderetal.(2009)demonstratetheeffectivenessofsuchpenaltiesinmeasuressuchastheBayesianinformationcriterionandtheJSZBayesfactor.CameronandTrivedi(2005,p.279)suggestusing√lognasamorestringent,criticalt-statistic.AnearliersuggestionbyGood(1982,1984,1992)entailsadjustingthecriticalp-valueusingasamplesizeofn = 100asareferencepointratherthanleavingitfixed(e.g.,p < 0.05)forallsamplesizes.Furthermore,theestimatedminimumsamplesizemaybereducedthroughvariancereductionbycontrolvariateswhichareoutcomeshavingknownmeanandvarianceandaresufficientlycorrelatedwithotheroutcomesofinterestforwhichthemeanandvarianceareunknown.ThistechniqueisdiscussedinthecontextofsimulationmodelsbyLawandKelton(2007).Finally,samplesizedeterminationforlargesimulationswhicharecostlytorun(i.e.,demandingheavycomputingresourcesandincurringlongexecutiontimes)maybeaddressedthroughbootstrappingofasmallersetofoutcomesforestimatingtheirvariance(Lee&Carley2013).

AppendixC:Multi-ModalABMOutputofBirthRateABMMulti-modalityintheoutputmayindicateseparateattractorsinthephasespacebridgedbytippingpoints.Forexample,theoutputdistribution(asubpopulation)ofasimple,populationgeneticsABM(Wilensky1997,1999)underasingleparametersettingexhibitsclearbimodality(Figure17).Thecarryingcapacitywassetto200;thefertilityrateforbothredandbluepopulationswas2.0;and10000runsof300stepswereperformed.

http://jasss.soc.surrey.ac.uk/18/4/4.html 18 25/01/2016

Figure17.BirthRateABMOutputDistribution.Histogramofthe"red"populationafter300timessteps,fittedtoaGaussiancurve(fromtheNetLogoSimpleBirthRatesModel(Wilensky1997))

Interpretingthemeanredagentpopulationof100.5(withalargedegreeoferror)asbeingsingularlyandtrulydescriptiveofthestochasticprocesswouldbegrosslyinaccurateandoverlookthesalientbifurcationinwhichtheredagentpopulationtendstoeitherdiminishtoextinction(0)ordominate(around200).ThisexampleillustrateshowABMstochasticitymayproducenon-normallydistributedoutputthatcannotbesensiblydescribedbymerelyitsmeanandvariance.

AppendixD:AnasaziModelOutliers

Figure18.Sensitivityanalysisexample.Source:StonedahlandWilensky(2010)

Theredlinedenotesthehistoricaldataandtheblacklinesrepresentoutcomesfrom100simulatedrunsparameterized(viaaGAsearch)toproduceoutputtime-seriesthatmaximallydifferfromthehistoricaldata.

ReferencesABRAHAMSON,D.,Blikstein,P.&Wilensky,U.(2007).Classroommodel,modelclassroom:Computer-supportedmethodologyforinvestigatingcollaborative-learningpedagogy.ProceedingsoftheComputerSupportedCollaborativeLearning(CSCL)Conference,8(1),46–55.[doi:10.3115/1599600.1599607]

AN,G.&Wilensky,U.(2009).Fromartificiallifetoinsilicomedicine:Netlogoasameansoftranslationalknowledge

http://jasss.soc.surrey.ac.uk/18/4/4.html 19 25/01/2016

representationinbiomedicalresearch.In:Artificiallifemodelsinsoftware(Adamatzky,A.&Komosinski,M.,eds.).Berlin:Springer-Verlag.

ANGUS,S.D.&Hassani-Mahmooei,B.(2015)."Anarchy"Reigns:AQuantitativeAnalysisofAgent-BasedModellingPublicationPracticesinJASSS,2001-2012.JournalofArtificialSocietiesandSocialSimulation,18(4),16http://jasss.soc.surrey.ac.uk/18/4/16.html

ANKENMAN,B.,Nelson,B.L.,&Staum,J.(2008).Stochastickrigingforsimulationmetamodelling.In:Proceedingsofthe40thConferenceonWinterSimulation.WinterSimulationConference,pp.362–370.

AXELROD,R.&Hammond,R.A.(2003).Theevolutionofethnocentricbehaviour.In:MidwestPoliticalScienceConvention,Chicago,IL.

AXTELL,R.L.,Epstein,J.S.,J.M.andfDean,Gumerman,G.J.,Swedlund,A.C.,Harburger,J.,Chakravartya,S.,Hammond,R.,Parker,J.&Parker,M.(2002).PopulationgrowthandcollapseinamultiagentmodeloftheKayentaAnasaziinlonghousevalley.In:ProceedingsoftheNationalAcademyofSciences,vol.99(suppl3),7275–7279.[doi:10.1073/pnas.092080799]

BARRETEAU,O.etal.(2003).Ourcompanionmodellingapproach.JournalofArtificialSocietiesandSocialSimulation6(2),1.http://jasss.soc.surrey.ac.uk/6/2/1.html.

BARROS,J.(2012).Exploringurbandynamicsinlatinamericancitiesusinganagent-basedsimulationapproach.In:Agent-BasedModelsofGeographicalSystems(Heppenstall,A.J.,Crooks,A.T.,See,L.M.&Batty,M.,eds.).Netherlands:Springer,pp.571–589..

BAUCELLS,M.&Borgonovo,E.(2013).Invariantprobabilisticsensitivityanalysis.ManagementScience,59(11),2536–2549.[doi:10.1287/mnsc.2013.1719]

BECU,N.,Perez,P.,Walker,A.,Barreteau,O.&LePage,C.(2003).AgentbasedsimulationofasmallcatchmentwatermanagementinnorthernThailand:DescriptionoftheCATCHSCAPEmodel.EcologicalModelling,170(2–3),319–331.[doi:10.1016/S0304-3800(03)00236-9]

BENNETT,N.,Croke,B.,Guariso,G.,Guillaume,J.H.,Hamilton,S.H.,Jakeman,J.&Marsili-Libelli,S.(2013).Characterisingperformanceofenvironmentalmodels.EnvironmentalModelling&Software,40,1–20.http://www.sciencedirect.com/science/article/pii/S1364815212002435.[doi:10.1016/j.envsoft.2012.09.011]

BOERO,R.,Bravo,G.,Castellani,M.&Squazzoni,F.(2010).Whybotherwithwhatotherstellyou?Anexperimentaldata-drivenagent-basedmodel.JournalofArtificialSocietiesandSocialSimulation,13(3),6.http://jasss.soc.surrey.ac.uk/13/3/6.html

BORGONOVO,E.(2007).Anewuncertaintyimportancemeasure.ReliabilityEngineering&SystemSafety,92(6),771–784.[doi:10.1016/j.ress.2006.04.015]

BOUSQUET,F.,Barreteau,O.,LePage,C.,Mullon,C.&Weber,J.(1999).Anenvironmentalmodellingapproach:Theuseofmultiagentsimulations.In:Advancesinenvironmentalandecologicalmodelling(Blasco,F.,ed.).Paris:Elsevier,pp.113–122.

BOUSQUET,F.&LePage,C.(2004).Multi-agentsimulationsandecosystemmanagement:areview.Ecologicalmodelling,176,313–332.[doi:10.1016/j.ecolmodel.2004.01.011]

BROWN,D.G.,Page,S.,Riolo,R.,Zellner,M.&Rand,W.(2005).Pathdependenceandthevalidationofagent-basedspatialmodelsoflanduse.InternationalJournalofGeographicalInformationScience,19(2),153–174..[doi:10.1080/13658810410001713399]

BROWN,D.G.&Robinson,D.T.(2006).Effectsofheterogeneityinresidentialpreferencesonanagent-basedmodelofurbansprawl.EcologyandSociety,11(1).

BRYSON,J.J.,Ando,Y.&Lehmann,H.(2007).Agent-basedmodellingasscientificmethod:Acasestudyanalysingprimatesocialbehaviour.PhilosophicalTransactionsoftheRoyalSocietyB:BiologicalSciences,362(1485),1685–1698.

CALVEZ,B.&Hutzler,G.(2006).Automatictuningofagent-basedmodelsusinggeneticalgorithms.In:Multi-AgentBasedSimulationVI,LectureNotesinComputerScience(Sichman,J.&Antunes,L.,eds.).Berlin/Heidelberg:Springer,pp.41–57.

CAMERON,A.C.&Trivedi,P.K.(2005).Microeconomics:MethodsandApplications.Cambridge:CambridgeUniversityPress.

CAMPO,P.,Bousquet,F.&Villanueva,T.(2010).Modellingwithstakeholderswithinadevelopmentproject.EnvironmentalModelling&Software,25(11),1302–1321.http://www.sciencedirect.com/science/article/pii/S1364815210000162.[doi:10.1016/j.envsoft.2010.01.005]

CAMPOLONGO,F.,Kleijnen,J.&Andres,T.(2000).Screeningmethods.In:SensitivityAnalysis(Saltelli,A.,Chan,K.&Scott,E.M.,eds.).UK:Chichester:Wiley-Interscience,pp.65–80.

CARLEY,K.,Fridsma,D.,Casman,E.,Yahja,A.,Altman,N.,Chen,L.,Kaminsky,B.&Nave,D.(2006).Biowar:Scalableagent-basedmodelofbioattacks.IEEETransactionsonSystems,ManandCybernetics,PartA,36(2),252–265.

http://jasss.soc.surrey.ac.uk/18/4/4.html 20 25/01/2016

[doi:10.1109/TSMCA.2005.851291]

CARLEY,K.M.&Lee,J.-S.(1998).Dynamicorganizations:Organizationaladaptationinachangingenvironment.In:AdvancesinStrategicManagement:DisciplinaryRootsofStrategicManagement,vol.15.Greenwich,CT:JAIPress,pp.269–297.

CHANG,M.-H.&Harrington,J.E.(2006).Agent-basedmodelsoforganizations.In:Handbookofcomputationaleconomicsvolume2:Agent-basedcomputationaleconomics(Tesfatsion,L.&Judd,K.L.,eds.).Amsterdam:ElsevierB.V.,pp.949–1011.

CHEN,S.-H.,Chang,C.-L.&Du,Y.-R.(2012).Agent-basedeconomicmodelsandeconometrics.TheKnowledgeEngineeringReview,27(2),187–219.[doi:10.1017/S0269888912000136]

COE,R.(2002).It'stheeffectsize,stupid:Whateffectsizeisandwhyitisimportant.In:AnnualConferenceoftheBritishEducationalResearchAssociation.UniversityofExeter,England.

COHEN,J.(1988).StatisticalPowerAnalysisforthebehaviouralSciences.LawrenceErlbaumAssociates.

CONTINI,B.,Leombruni,R.&Richiardi,M.(2007).Exploringanewexpace:Thecomplementaritiesbetweenexperimentaleconomicsandagent-basedcomputationaleconomics.JournalofSocialComplexity,3(1),13–22.

DEAN,J.S.,Gumerman,G.J.,Epstein,J.M.,Axtell,R.L.,Swedlund,A.C.,Parker,M.T.&McCarroll,S.(2000).UnderstandingAnasaziculturechangethroughagent-basedmodelling.In:Dynamicsinhumanandprimatesocieties:Agent-basedmodellingofsocialandspatialprocesses(Kohler,T.&Gumerman,G.,eds.).Oxford,UK:OxfordUniversityPress,pp.179–205.

DEARDEN,J.&Wilson,A.(2012).Therelationshipofdynamicentropymaximisingandagent-basedapproachesinurbanmodelling.In:Agent-BasedModelsofGeographicalSystems(Heppenstall,A.J.,Crooks,A.T.,See,L.M.&Batty,M.,eds.).SpringerNetherlands,pp.705–720..

DUFFY,J.(2006).Agent-basedmodelsandhumansubjectexperiments.In:Handbookofcomputationaleconomicsvolume2:Agent-basedcomputationaleconomics(Tesfatsion,L.&Judd,K.L.,eds.).Amsterdam:ElsevierB.V.,pp.949–1011.

EFFKEN,J.A.,Carley,K.M.,Lee,J.-S.,Brewer,B.B.&Verran,J.A.(2012).SimulatingNursingUnitPerformancewithOrgAhead:StrengthsandChallenges.ComputersInformaticsNursing,30(11),620–626.[doi:10.1097/NXN.0b013e318261f1bb]

EPSTEIN,J.M.(2002).modellingcivilviolence:Anagent-basedcomputationalapproach.ProceedingsoftheNationalAcademyofSciencesoftheUnitedStatesofAmerica,99(Suppl3),7243–7250.[doi:10.1073/pnas.092080199]

EPSTEIN,J.M.&Axtell,R.(1996).Growingartificialsocieties:Socialsciencefromthebottomup.Washington,DC:BrookingsInstitutionPress.

ÉTIENNE,M.(2014).CompanionModelling.Springer.http://link.springer.com/10.1007/978-94-017-8557-0.[doi:10.1007/978-94-017-8557-0]

FILATOVA,T.(2014).Empiricalagent-basedlandmarket:Integratingadaptiveeconomicbehaviourinurbanland-usemodels.Computers,EnvironmentandUrbanSystemshttp://www.sciencedirect.com/science/article/pii/S0198971514000714.

FILATOVA,T.,Parker,D.C.&vanderVeen,A.(2011).TheImplicationsofSkewedRiskPerceptionforaDutchCoastalLandMarket:InsightsfromanAgent-BasedComputationalEconomicsModel.AgriculturalandResourceEconomicsReview,40(3),405–423.http://ideas.repec.org/a/ags/arerjl/120639.html.

FISHER,R.A.(1971).TheDesignofExperiments.Hafner.

FONOBEROVA,M.,Fonoberov,V.A.&Mexic,I.(2013).Globalsensitivity/uncertaintyanalysisforagent-basedmodels.ReliabilityEngineering&SystemSafety,118(0),8–17.[doi:10.1016/j.ress.2013.04.004]

GHORBANI,A.,Bots,P.,Dignum,V.&Dijkema,G.(2013).Maia:Aframeworkfordevelopingagent-basedsocialsimulations.JournalofArtificialSocietiesandSocialSimulation,16(2),9.http://jasss.soc.surrey.ac.uk/16/2/9.html

GOOD,I.J.(1982).Standardizedtail-areaprobabilities.JournalofStatisticalComputationandSimulation,16,65–66.[doi:10.1080/00949658208810607]

GOOD,I.J.(1984).C191.howshouldtail-areaprobabilitiesbestandardizedforsamplesizeinunpairedcomparisons?JournalofStatisticalComputationandSimulation,19(2),174–174.

GOOD,I.J.(1992).TheBayes/non-Bayescompromise:Abriefreview.JournaloftheAmericanStatisticalAssociation,87(419),597–606.[doi:10.1080/01621459.1992.10475256]

GRAZZINI,J.(2012).Analysisoftheemergentproperties:Stationarityandergodicity.JournalofArtificialSocietiesandSocialSimulation,15(2),7.http://jasss.soc.surrey.ac.uk/15/2/7.html

GRAZZINI,J.&Richiardi,M.(2015).Estimationofergodicagent-basedmodelsbysimulatedminimumdistance.JournalofEconomicDynamicsandControl,51,148–165.[doi:10.1016/j.jedc.2014.10.006]

http://jasss.soc.surrey.ac.uk/18/4/4.html 21 25/01/2016

GRIFFITH,C.S.,Long,B.L.&Sept,J.M.(2010).HOMINIDS:Anagent-basedspatialsimulationmodeltoevaluatebehaviouralpatternsofearlyPleistocenehominids.EcologicalModelling,221,738–760.[doi:10.1016/j.ecolmodel.2009.11.009]

GRIMM,V.,Berger,U.,Bastiansen,F.,Eliassen,S.,Ginot,V.,Giske,J.,Goss-Custard,J.,Grand,T.,Heinz,S.K.,Huse,G.,Huth,A.,Jepsen,J.U.,Jørgensen,C.,Mooij,W.M.,Müller,B.,Pe'er,G.,Piou,C.,Railsback,S.F.,Robbins,A.M.,Robbins,M.M.,Rossmanith,E.,Rüger,N.,Strand,E.,Souissi,S.,Stillman,R.A.,Vabø,R.,Visser,U.&DeAngelis,D.L.(2006).Astandardprotocolfordescribingindividual-basedandagent-basedmodels.EcologicalModelling,198(1–2),115–126.[doi:10.1016/j.ecolmodel.2006.04.023]

GRIMM,V.,Berger,U.,DeAngelis,D.L.,Polhill,J.G.,Giske,J.&Railsback,S.F.(2010).Theoddprotocol:Areviewandfirstupdate.EcologicalModelling,221(23),2760–2768.[doi:10.1016/j.ecolmodel.2010.08.019]

GRIMM,V.&Railsback,S.(2005).Individual-basedmodellingandecology.PrincetonUniversityPress.[doi:10.1515/9781400850624]

HAMBY,D.M.(1994).Areviewoftechniquesforparametersensitivityanalysisofenvironmentalmodels.EnvironmentalMonitoringandAssessment,32(2),135–154.[doi:10.1007/BF00547132]

HAMILL,L.(2010).Agent-basedmodelling:Thenext15years.JournalofArtificialSocietiesandSocialSimulation,13(4),7.http://jasss.soc.surrey.ac.uk/13/4/7.html

HAPPE,K.,Kellermann,K.&Balmann,A.(2006).Agent-basedanalysisofagriculturalpolicies:AnillustrationoftheagriculturalpolicysimulatorAgriPoliS,itsadaptation,andbehaviour.EcologyandSociety,11(1).

HASSANI-MAHMOOEI,B.&Parris,B.W.(2013).Resourcescarcity,effortallocationandenvironmentalsecurity:Anagent-basedtheoreticalapproach.EconomicModelling,30,183–192.[doi:10.1016/j.econmod.2012.08.020]

HELBING,D.,Buzna,L.,Johansson,A.&Werner,T.(2005).Self-organizedpedestriancrowddynamics:Experiments,simulations,anddesignsolutions.TransportationScience,39(1),1–24.[doi:10.1287/trsc.1040.0108]

HEPPENSTALL,A.J.,Crooks,A.T.,See,L.M.&Batty,M.(eds.)(2012).Agent-BasedModelsofGeographicalSystems.SpringerNetherlands.[doi:10.1007/978-90-481-8927-4]

HEPPENSTALL,A.J.,Evans,A.J.&Birkin,M.H.(2007).Geneticalgorithmoptimisationofanagent-basedmodelforsimulatingaretailmarket.EnvironmentandPlanningB:PlanningandDesign,34,1051–1070.[doi:10.1068/b32068]

HINTZE,J.L&Nelson,R.D.(1998).Violinplots:Aboxplot-densitytracesynergism.TheAmericanStatistician,52(2),181–184.

HOLLAND,J.(1975).AdaptationinNaturalandArtificialSystems.AnnArbor,MI:UniversityofMichiganPress.

HOMMA,T.&Saltelli,A.(1996).Importancemeasuresinglobalsensitivityanalysisofnonlinearmodels.ReliabilityEngineering&SystemSafety,52(1),1–17.[doi:10.1016/0951-8320(96)00002-6]

HUANG,Q.,Parker,D.C.,Sun,S.&Filatova,T.(2013).Effectsofagentheterogeneityinthepresenceofaland-market:Asystematictestinanagent-basedlaboratory.Computers,EnvironmentandUrbanSystems,41,188–203.http://www.sciencedirect.com/science/article/pii/S0198971513000616.[doi:10.1016/j.compenvurbsys.2013.06.004]

ILACHINSKI,A.(2000).Irreduciblesemi-autonomousadaptivecombat(isaac):Anartificiallifeapproachtolandcombat.MilitaryOperationsResearch,5(3),29–46.[doi:10.5711/morj.5.3.29]

IZQUIERDO,S.S.,Izquierdo,L.R.&Gotts,N.M.(2008).Reinforcementlearningdynamicsinsocialdilemmas.JournalofArtificialSocietiesandSocialSimulation,11(2),1.http://jasss.soc.surrey.ac.uk/11/2/1.html

JANSSEN,M.A.(2009).UnderstandingartificialAnasazi.JournalofArtificialSocietiesandSocialSimulation,12(4),13.http://jasss.soc.surrey.ac.uk/12/4/13.html

KAHL,C.H.&Hansen,H.(2015).Simulatingcreativityfromasystemsperspective:CRESY,18(1),4.

KEOGH,E.&Ratanamahatana,C.A.(2005).Exactindexingofdynamictimewarping.KnowledgeandInformationSystems,7(3),358–386.[doi:10.1007/s10115-004-0154-9]

KIRMAN,A.P.(1992).Whomorwhatdoestherepresentativeindividualrepresent?JournalofEconomicPerspectives,6(117–136).

KLEIJNEN,J.P.C.,Sanchez,S.M.,Lucas,T.W.&Cioppa,T.M.(2005).Ausersguidetothebravenewworldofdesigningsimulationexperiments.INFORMSJournalonComputing,17(3),263–289.[doi:10.1287/ijoc.1050.0136]

KLEIN,F.,Bourjot,C.&Chevrier,V.(2005).Dynamicaldesignofexperimentwithmastoapproximatethebehaviourofcomplexsystems.Multi-AgentsformodellingComplexSystems(MA4CS'05)SatelliteWorkshopoftheEuropeanConferenceonComplexSystems(ECCS'05),Nov2005,Paris/France.

http://jasss.soc.surrey.ac.uk/18/4/4.html 22 25/01/2016

KLINGERT&Meyer(2012).Effectivelycombiningexperimentaleconomicsandmulti-agentsimulation:Suggestionsforaproceduralintegrationwithanexamplefrompredictionmarketsresearch.ComputationalandMathematicalOrganizationTheory,18(1),63–90.http://link.springer.com/article/10.1007\%2Fs10588-011-9098-2\#page-1.[doi:10.1007/s10588-011-9098-2]

KUCHERENKO,S.,Tarantola,S.&Annoni,P.(2012).Estimationofglobalsensitivityindicesformodelswithdependentvariables.ComputerPhysicsCommunications,183(4),937–946.[doi:10.1016/j.cpc.2011.12.020]

KUHNERT,M.,Voinov,A.&Seppelt,R.(2005).Comparingrastermapcomparisonalgorithmsforspatialmodellingandanalysis.PhotogrammetricEngineering&RemoteSensing,71(8),975–984.[doi:10.14358/PERS.71.8.975]

LAW,A.M.&Kelton,W.D.(2007).SimulationmodellingandAnalysis.Boston:McGrawHill.

LEBARON,B.,Arthur,W.&Palmer,R.(1999).Timeseriespropertiesofanartificialstockmarket.JournalofEconomicDynamicsandControl,23(9–10),1487–1516.http://www.sciencedirect.com/science/article/pii/S0165188998000815.[doi:10.1016/S0165-1889(98)00081-5]

LEE,J.-S.(2004).GeneratingFriendshipNetworksofIllegalDrugUsersUsingLargeSamplesofPartialEgo-NetworkData.In:ProceedingsoftheNorthAmericanAssociationforComputationalSocialandOrganizational(NAACSOS).Pittsburgh,PA.

LEE,J.-S.&Carley,K.M.(2004).OrgAhead:AComputationalModelofOrganizationalLearningandDecisionMaking[Version2.1.5].Tech.Rep.CMU-ISRI-04-117,CarnegieMellonUniversity.

LEE,J.-S.&Carley,K.M.(2013).Bootstrappingsimulationresultstoassessadequatenumberofreplications.Tech.Rep.TBD,CarnegieMellonUniversity.

LIGMANN-ZIELINSKA,A.(2013).Spatially-explicitsensitivityanalysisofanagent-basedmodeloflandusechange.InternationalJournalofGeographicalInformationScience,27(9),1764–1781.[doi:10.1080/13658816.2013.782613]

LIGMANN-ZIELINSKA,A.,Kramer,D.B.,Cheruvelil,K.S.&Soranno,P.A.(2014).Usinguncertaintyandsensitivityanalysesinsocioecologicalagent-basedmodelstoimprovetheiranalyticalperformanceandpolicyrelevance.PloSOne,9(10),e109779.[doi:10.1371/journal.pone.0109779]

LIGMANN-ZIELINSKA,A.&Sun,L.(2010).Applyingtimedependentvariance-basedglobalsensitivityanalysistorepresentthedynamicsofanagent-basedmodeloflandusechange.InternationalJournalofGeographicalInformationScience,24(12),1829–1850.[doi:10.1080/13658816.2010.490533]

LILBURNE,L.&Tarantola,S.(2009).Sensitivityanalysisofspatialmodels.InternationalJournalofGeographicalInformationScience,23(2),151–168.[doi:10.1080/13658810802094995]

LIU,Y.&Feng,Y.(2012).Alogisticbasedcellularautomatamodelforcontinuousurbangrowthsimulation:Acasestudyofthegoldcoastcity,Australia.In:Agent-BasedModelsofGeographicalSystems(Heppenstall,A.J.,Crooks,A.T.,See,L.M.&Batty,M.,eds.).SpringerNetherlands,pp.643–662..

LORSCHEID,I.(2014).LAMDA–Learningagentsformechanismdesignanalysis.Unpublisheddissertation.

LORSCHEID,I.,Heine,B.-O.&Meyer,M.(2012).Openingthe'blackboxofsimulations:Increasedtransparencyandeffectivecommunicationthroughthesystematicdesignofexperiments.ComputationalandMathematicalOrganizationTheory,18,22–62.[doi:10.1007/s10588-011-9097-3]

MACY,M.W.&Willer,R.(2002).Fromfactorstofactors:Computationalsociologyandagent-basedmodelling.AnnualReviewofSociology,28,143–166.[doi:10.1146/annurev.soc.28.110601.141117]

MAGLIOCCA,N.(2012).Exploringcoupledhousingandlandmarketinteractionsthroughaneconomicagent-basedmodel(chalms).In:Agent-BasedModelsofGeographicalSystems(Heppenstall,A.J.,Crooks,A.T.,See,L.M.&Batty,M.,eds.).SpringerNetherlands,pp.543–568..

MALLESON,N.,Heppenstall,A.,See,L.&Evans,A.(2013).Usinganagent-basedcrimesimulationtopredicttheeffectsofurbanregenerationonindividualhouseholdburglaryrisk.EnvironmentandPlanningB:PlanningandDesign,40(3),405–426.[doi:10.1068/b38057]

MANSON,S.M.(2005).Agent-basedmodellingandgeneticprogrammingformodellinglandchangeinthesouthernYucatanpeninsularregionofMexico.Agriculture,Ecosystems&Environment,111(1),47–62.[doi:10.1016/j.agee.2005.04.024]

MARA,T.A.&Tarantola,S.(2012).Variance-basedsensitivityindicesformodelswithdependentinputs.ReliabilityEngineering&SystemSafety,107(0),115–121.[doi:10.1016/j.ress.2011.08.008]

MARREL,A.,Iooss,B.,Jullien,M.,Laurent,B.&Volkova,E.(2011).Globalsensitivityanalysisformodelswithspatiallydependentoutputs.Environmetrics,22(3),383–397.[doi:10.1002/env.1071]

MILLER,J.(1998).Activenonlineartests(ANTs)ofcomplexsimulationmodels.ManagementScience,44(6),820–830.[doi:10.1287/mnsc.44.6.820]

http://jasss.soc.surrey.ac.uk/18/4/4.html 23 25/01/2016

MILLINGTON,J.,Romero-Calcerrada,R.,Wainwright,J.&Perry,G.(2008).Anagent-basedmodelofMediterraneanagriculturalland-use/coverchangeforexaminingwildfirerisk.JournalofArtificialSocietiesandSocialSimulation,11(4),4.http://jasss.soc.surrey.ac.uk/11/4/4.html

MORRIS,M.D.(1991).Factorialsamplingplansforpreliminarycomputationalexperiments.Technometrics,33(2),161–174.[doi:10.1080/00401706.1991.10484804]

MOSS,S.(2008).Alternativeapproachestotheempiricalvalidationofagent-basedmodels.JournalofArtificialSocietiesandSocialSimulation,11,(1),5http://jasss.soc.surrey.ac.uk/11/1/5.html.

MÜLLER,B.,Bohn,F.,Dreßler,G.,Groeneveld,J.,Klassert,C.,Martin,R.,Schlüter,M.,Schulze,J.,Weise,H.&Schwarz,N.(2013).Describinghumandecisionsinagent-basedmodels–ODD+D,anextensionofthe{ODD}protocol.EnvironmentalModelling&Software,48(0),37–48.[doi:10.1016/j.envsoft.2013.06.003]

NERI,F.(2012).Learningpredictivemodelsforfinancialtimeseriesbyusingagentbasedsimulations.In:TransactionsonComputationalCollectiveIntelligenceVI(Nguyen,N.T.,ed.),vol.7190.SpringerBerlinHeidelberg,pp.202–221.

NORTH,M.&Macal,C.(2007).Managingbusinesscomplexity:Discoveringstrategicsolutionswithagent-basedmodellingandsimulation.USA:OxfordUniversityPress.[doi:10.1093/acprof:oso/9780195172119.001.0001]

PARKER,D.C.,Berger,T.&Manson,S.M.(2002).Agent-basedmodelsofland-useandland-coverchange:Reportandreviewofaninternationalworkshop.LUCCReportSeriesBloomington,LUCC.

PARKER,D.C.&Meretsky,V.(2004).Measuringpatternoutcomesinanagent-basedmodelofedge-effectexternalitiesusingspatialmetrics.Agriculture,Ecosystems&Environment,101(2–3),233–250.[doi:10.1016/j.agee.2003.09.007]

PARRY,H.R.&Bithell,M.(2012).Largescaleagent-basedmodelling:Areviewandguidelinesformodelscaling.In:Agent-BasedModelsofGeographicalSystems(Heppenstall,A.J.,Crooks,A.T.,See,L.M.&Batty,M.,eds.).SpringerNetherlands,pp.271–308..

PATEL,A.&Hudson-Smith,A.(2012).Agenttools,techniquesandmethodsformacroandmicroscopicsimulation.In:Agent-BasedModelsofGeographicalSystems(Heppenstall,A.J.,Crooks,A.T.,See,L.M.&Batty,M.,eds.).SpringerNetherlands,pp.379–407..

PLANTINGA,A.J.&Lewis,D.J.(2014).Landscapesimulationswitheconometric-basedlandusemodels.In:TheOxfordHandbookofLandEconomics(Duke,J.M.&Wu,J.-J.,eds.),chap.15.OxfordUniversityPress.http://www.oxfordhandbooks.com/10.1093/oxfordhb/9780199763740.001.0001/oxfordhb-9780199763740-e-013.

PONTIUS,R.G.,Jr.(2002).Statisticalmethodstopartitioneffectsofquantityandlocationduringcomparisonofcategoricalmapsatmultipleresolutions.PhotogrammetricEngineeringandRemoteSensing,68(10),1041–1050.

PONTIUS,R.G.,Jr.,Boersma,W.,Castella,J.-C.,Clarke,K.,deNijs,T.,Dietzel,C.,Duan,Z.,Fotsing,E.,Goldstein,N.,Kok,K.,Koomen,E.,Lippitt,C.D.,McConnell,W.,MohdSood,A.,Pijanowski,B.,Pithadia,S.,Sweeney,S.,Trung,T.N.,Veldkamp,A.T.&Verburg,P.H.(2008).Comparingtheinput,output,andvalidationmapsforseveralmodelsoflandchange.TheAnnalsofRegionalScience,42(1),11–37..[doi:10.1007/s00168-007-0138-2]

PONTIUS,R.G.,Jr.&Millones,M.(2011).Deathtokappa:Birthofquantitydisagreementandallocationdisagreementforaccuracyassessment.InternationalJournalofRemoteSensing,32(15),4407–4429..[doi:10.1080/01431161.2011.552923]

RACZYNSKI,S.(2004).Simulationofthedynamicinteractionsbetweenterrorandanti-terrororganizationalstructures.JournalofArtificialSocietiesandSocialSimulation7(2),8http://jasss.soc.surrey.ac.uk/7/2/8.html.

RADAX,W.&Rengs,B.(2010).Prospectsandpitfallsofstatisticaltesting:Insightsfromreplicatingthedemographicprisoner'sdilemma.JournalofArtificialSocietiesandSocialSimulation,13(4),1.http://jasss.soc.surrey.ac.uk/13/4/1.html.

Rand,W.&Wilensky,U.(2007).NetLogoElFarolmodel.CenterforConnectedComputer-Basedmodelling,NorthwesternUniversity,Evanston,IL.

RICHIARDI,M.,Leombruni,R.,Saam,N.&Sonnessa,M.(2006).Acommonprotocolforagent-basedsocialsimulation.Journalofartificialsocietiesandsocialsimulation,9(1),15http://jasss.soc.surrey.ac.uk/9/1/15.html.

RICHIARDI,M.G.(2012).Agent-basedcomputationaleconomics:Ashortintroduction.TheKnowledgeEngineeringReview,27(2),137–149.[doi:10.1017/S0269888912000100]

RIKSBV(2010).MapComparisonKitUserManual.Maastricht,TheNetherlands:RIKSBV,3.2.1ed.http://mck.riks.nl/.

ROBINSON,D.T.,Brown,D.,Parker,D.C.,Schreinemachers,P.,Janssen,M.A.&Huigen,M.(2007).Comparisonofempiricalmethodsforbuildingagent-basedmodelsinlandusescience.JournalofLandUseScience,2(1),31–55.[doi:10.1080/17474230701201349]

ROUDER,J.N.,Speckman,P.L.,Sun,D.,Morey,R.D.&Iverson,G.(2009).Bayesianttestsforacceptingandrejectingthe

http://jasss.soc.surrey.ac.uk/18/4/4.html 24 25/01/2016

nullhypothesis.PsychonomicBulletin&Review,16(2),225–37.http://www.ncbi.nlm.nih.gov/pubmed/19293088.[doi:10.3758/PBR.16.2.225]

SACKS,J.,Welch,W.,Mitchell,T.&Wynn,H.(1989).Designandanalysisofcomputerexperiments.StatisticalScience,4(4),409–422.[doi:10.1214/ss/1177012413]

SALTELLI,A.&Annoni,P.(2010).Howtoavoidaperfunctorysensitivityanalysis.EnvironmentalModelling&Software,25(12),1508–1517.[doi:10.1016/j.envsoft.2010.04.012]

SALTELLI,A.,Chan,K.&Scott,E.M.(eds.)(2000).SensitivityAnalysis.UK:Chichester:Wiley-Interscience.

SALTELLI,A.,Ratto,M.,Andres,T.,Campolongo,F.,Cariboni,J.,Gatelli,D.,Saisana,M.&Tarantola,S.(2008).GlobalSensitivityAnalysis:ThePrimer.UK:Chichester:Wiley-Interscience.

SANCHEZ,S.M.&Lucas,T.W.(2002).Exploringtheworldofagent-basedsimulations:Simplemodels,complexanalyses.In:WSC2002:Proceedingsofthe34thWinterSimulationConference.

SCHELLING,T.C.(1969).Modelsofsegregation.AmericanEconomicReview,59(2),488–493.

SCHELLING,T.C.(1978).MicromotivesandMacrobehaviour.London,UK:W.W.Norton&Co.

SMAJGL,A.,Brown,D.G.,Valbuena,D.&Huigen,M.G.A.(2011).Empiricalcharacterisationofagentbehavioursinsocio-ecologicalsystems.EnvironmentalModelling&Software,26(7),837–844.[doi:10.1016/j.envsoft.2011.02.011]

SQUAZZONI,F.(2012).Agent-BasedComputationalSociology.Chichester:Wiley.[doi:10.1002/9781119954200]

SQUAZZONI,F.&Boero,R.(2002).Economicperformance,inter-firmrelationsandlocalinstitutionalengineeringinacomputationalprototypeofindustrialdistricts.JournalofArtificialSocietiesandSocialSimulation,5(1),1http://jasss.soc.surrey.ac.uk/5/1/1.html.

STONEDAHL,F.&Rand,W.(2014).Whendoessimulateddatamatchrealdata?comparingmodelcalibrationfunctionsusinggeneticalgorithms.In:AdvancesinComputationalSocialScience:TheFourthWorldCongress(Tai,C.,Chen,S.,Ternao,T.&Yamamoto,R.,eds.),vol.11ofAgent-BasedSocialSystems.Springer-Verlag,pp.297–313.

STONEDAHL,F.,Rand,W.&Wilensky,U.(2010).Evolvingviralmarketingstrategies.In:Proceedingsofthe12thAnnualConferenceonGeneticandEvolutionaryComputation(GECCO10).July7–11.Portland,OR.

STONEDAHL,F.&Wilensky,U.(2010).EvolutionaryrobustnesscheckingintheartificialAnasazimodel.In:ProceedingsoftheAAAIFallSymposiumonComplexAdaptiveSystems:Resilience,Robustness,andEvolvability.Arlington,VA.

STONEDAHL,F.&Wilensky,U.(2011).Findingformsofflocking:Evolutionarysearchinabmparameter-spaces.In:Multi-Agent-BasedSimulationXI(Bosse,T.,Geller,A.&Jonker,C.,eds.),vol.6532ofLectureNotesinComputerScience.SpringerBerlinHeidelberg,pp.61–75..

SULLIVAN,G.M.&Feinn,R.(2012).Usingeffectsize–orwhythepvalueisnotenough.JournalofGraduateMedicalEducation,4(3),279–282.[doi:10.4300/JGME-D-12-00156.1]

SUN,S.,Parker,D.C.,Huang,Q.,Filatova,T.,Robinson,D.T.,Riolo,R.L.,Hutchins,M.&Brown,D.G.(2014).Marketimpactsonland-usechange:Anagent-basedexperiment.AnnalsoftheAssociationofAmericanGeographers,104(3),460–484..[doi:10.1080/00045608.2014.892338]

TAKAHASHI,H.&Terano,T.(2003).Agent-basedapproachtoinvestors'behaviourandassetpricefluctuationinfinancialmarkets.JournalofArtificialSocietiesandSocialSimulation,6(3),3http://jasss.soc.surrey.ac.uk/6/3/3.html.

TAMENE,L.,Le,Q.B.&Vlek,P.L.(2014).Alandscapeplanningandmanagementtoolforlandandwaterresourcesmanagement:Anexampleapplicationinnorthernethiopia.WaterResourcesManagement,28(2),407–424..[doi:10.1007/s11269-013-0490-1]

TANG,W.&Jia,M.(2014).Globalsensitivityanalysisofalargeagent-basedmodelofspatialopinionexchange:Aheterogeneousmulti-GPUaccelerationapproach.AnnalsoftheAssociationofAmericanGeographers,104(3),485–509.[doi:10.1080/00045608.2014.892342]

TARANTOLA,W.B.,S.&Zeitz,D.(2012).Acomparisonoftwosamplingmethodsforglobalsensitivityanalysis.ComputerPhysicsCommunications,183(5),1061–1072.[doi:10.1016/j.cpc.2011.12.015]

TENBROEKE,G.,vanVoorn,G.&Ligtenberg,A.(2014).Sensitivityanalysisforagent-basedmodels:Alowcomplexitytest-case.In:10thAnnualMeetingoftheEuropeanSocialSimulationAssociation(ESSA).Barcelona,Catalonia,Spain.

TESFATSION,L.&Judd,K.L.(2006).HandbookofComputationalEconomicsVolumeII:Agent-BasedComputationalEconomics.ElsevierB.V.

http://jasss.soc.surrey.ac.uk/18/4/4.html 25 25/01/2016

THIELE,J.C.&Grimm,V.(2010).NetLogomeetsR:Linkingagentbasedmodelswithatoolboxfortheiranalysis.EnvironmentalModelling&Software,25(8),972–974.[doi:10.1016/j.envsoft.2010.02.008]

THIELE,J.C.,Kurth,W.&Grimm,V.(2014).Facilitatingparameterestimationandsensitivityanalysisofagent-basedmodels:AcookbookusingNetLogoand'R'.JournalofArtificialSocietiesandSocialSimulation,17(3),11http://jasss.soc.surrey.ac.uk/17/3/11.html.

TORRENS,P.M.(2006).Simulatingsprawl.AnnalsoftheAssociationofAmericanGeographers,96(2),248–275..[doi:10.1111/j.1467-8306.2006.00477.x]

TROITZSCH,K.G.(2014).SimulationExperimentsandSignificanceTests.In:ArtificialEconomicsandSelfOrganization(Leitner,S.&Wall,F.,eds.),vol.669ofLectureNotesinEconomicsandMathematicalSystems.SpringerInternationalPublishing,pp.17–29.http://link.springer.com/10.1007/978-3-319-00912-4.

VILLAMOR,G.B.,Le,Q.B.,Djanibekov,U.,vanNoordwijk,M.&Vleka,P.L.(2014).Biodiversityinrubberagroforests,carbonemissions,andrurallivelihoods:Anagent-basedmodelofland-usedynamicsinlowlandsumatra.EnvironmentalModelling&Software,61,151–165.[doi:10.1016/j.envsoft.2014.07.013]

VOINOV,A.&Bousquet,F.(2010).Modellingwithstakeholders.EnvironmentalModelling&Software,25(May),1268–1281.http://linkinghub.elsevier.com/retrieve/pii/S1364815210000538.[doi:10.1016/j.envsoft.2010.03.007]

VOINOV,A.,Seppelt,R.,Reis,S.,E.M.S.Nabel,J.&Shokravi,S.(2014).Valuesinsocio-environmentalmodelling:Persuasionforactionorexcuseforinaction.EnvironmentalModelling&Software,53(March),207–212.http://linkinghub.elsevier.com/retrieve/pii/S1364815213003083.[doi:10.1016/j.envsoft.2013.12.005]

WANG,J.,Dam,G.,Yildirim,S.,Rand,W.,Wilensky,U.&Houk,J.(2008).Reciprocitybetweenthecerebellumandthecerebralcortex:Nonlineardynamicsinmicroscopicmodulesforgeneratingvoluntarymotorcommands.Complexity,14(2).[doi:10.1002/cplx.20241]

WHEELER,B.(2011).AlgDesign:AlgorithmicExperimentalDesign.Rpackageversion1.1-7ed.

WHITE,J.W.,Rassweiler,A.,Samhouri,J.F.,Stier,A.C.&White,C.(2013).Ecologistsshouldnotusestatisticalsignificanceteststointerpretsimulationmodelresults.Oikos,123(4),385–388.[doi:10.1111/j.1600-0706.2013.01073.x]

WILENSKY,U.(1997).Netlogosimplebirthratesmodel.CenterforConnectedLearningandComputer-Basedmodelling,NorthwesternUniversity,Evanston,IL.,http://ccl.northwestern.edu/netlogo/models/SimpleBirthRates.

WILENSKY,U.(1999).Netlogo.CenterforConnectedLearningandComputer-Basedmodelling,NorthwesternUniversity,Evanston,IL,http://ccl.northwestern.edu/netlogo/.

WILENSKY,U.(2004).NetlogoRebellionmodel.CenterforConnectedLearningandComputer-Basedmodelling,NorthwesternUniversity,Evanston,IL.

WINDRUM,P.,Fagiolo,G.&Moneta,A.(2007).Empiricalvalidationofagent-basedmodels:Alternativesandprospects.JournalofArtificialSocietiesandSocialSimulation,10(2),8http://jasss.soc.surrey.ac.uk/10/2/8.html.

WORRAPIMPHONG,K.,Gajaseni,N.,Page,C.L.&Bousquet,F.(2010).Acompanionmodellingapproachappliedtofisherymanagement.EnvironmentalModelling&Software,25(11),1334–1344.http://www.sciencedirect.com/science/article/pii/S1364815210000599.[doi:10.1016/j.envsoft.2010.03.012]

WU,F.(2002).Calibrationofstochasticcellularautomata:Theapplicationtorural-urbanlandconversions. InternationalJournalofGeographicalInformationScience,16(8),795–818..[doi:10.1080/13658810210157769]

YAMADA,T.&Terano,T.(2009).Fromthesimplestpriceformationmodelstoparadigmofagent-basedcomputationalfinance:Afirststep.In:Agent-BasedApproachesinEconomicandSocialComplexSystems(Terano,V.T.,Kita,H.,Takahashi,S.&Deguchi,H.,eds.),vol.6.SpringerJapan,pp.219–230.

ZILIAK,S.T.&McCloskey,D.N.(2008).TheCultofStatisticalSignificance:HowtheStandardErrorCostsUsJobs,Justice,andLives.UniversityofMichiganPress.

ZILIAK,S.T.&McCloskey,D.N.(2009).Thecultofstatisticalsignificance.In:JointStatisticalMeetings.Washington,DC.

ZINCK,R.&Grimm,V.(2008).Morerealisticthananticipated:Aclassicalforest-firemodelfromstatisticalphysicscapturesrealfireshapes.OpenEcologyJournal,1,8–13.[doi:10.2174/1874213000801010008]

ZUNIGA,M.M.,Kucherenko,S.&Shah,N.(2013).Metamodellingwithindependentanddependentinputs.ComputerPhysicsCommunications,184(6),1570–1580.[doi:10.1016/j.cpc.2013.02.005]

Processingmath:100%

http://jasss.soc.surrey.ac.uk/18/4/4.html 26 25/01/2016