3/9/19

1

WargameAnalysis

Rex Brynen

McGill University

analysis

u How can a game design produce causal inferences?u question generation

u what sorts of issues do we need to explore further?

u plausibility probe u given this set of circumstances, this sort of behavior/actions/outcomes seems

possible

u capability and doctrine assessment

u if we had X, what might we do differently?

u How can we best do Y?

u internal (game model) and external (generalization) validity

u the problem of small (or single) iterations

3/9/19

2

are wargames experiments?

u Person-in-the-loop wargames are never “pure” experiments:u Impossibility of identical control and treatment groups because of variations

among participants.

u different participants could lead to different outcomes

u using same participants results in learning and other adaptive behaviours from game to game

u Rarely are large-n iterations possible (time, cost, participants), and hence the inability to conduct randomized trials.

u Difficulty of limiting the number of variables while keeping the situation realistic.

are wargames predictions?

u no wargame should be considered a high-confidence, specific predictionu limited models of complex realities

u known and unknown unknowns

u idiosyncratic effects

u limited number of iterations (often only one)

u even with multiple iterations, actual outcomes may not be the median or modal result

3/9/19

3

• evening, 26 May: aircraft from HMS Ark Royal accidentally attack HMS Sheffield

• c21:25m 26 May: second attack, with contact torpedoes hits starboard side of Bismarck, jamming rudder at 12o

the challenge of prediction

Weak national

leadership inhibits

effective strategy.

Superior weapons.

Flanking force

achieves tactical surprise.

Potential BLUE allies deterred

from intervention.

REDVICTORY!

White Wizard takes

command of city

defences.

King Théoden

musters the Rohirrim.

Witch King of Angmar slain by

hobbit and shieldmaiden

Creepy dead people led by missing King

arrive instead of pirates.

BLUE VICTORY!

Pelennor Fields/Battle of Minas Tirith (3018, Third Age)

3/9/19

4

Data: 16 runs of (largely similar) Brynania simulation, 1999-2017.

the challenge of prediction

the challenge of prediction

3/9/19

5

BUT

u wargames can be useful quasi-experiments

u wargames are part of the “cycle of research”

u wargames should be part of research triangulationu other methods aren’t flawless either (GIGO problems in

OR)

u an indispensable tool in our analytical toolkit

u wargames generate questions and ideas for further study (Schelling)

Peter Perla C

NA

analytical wargamesu develop clear research question(s)

u develop appropriate game design

u collect insights generated by design process

u develop data collection and analysis plan

u might require changes to design

u collect data/after during game play

u actions

u note-taking/recording of team deliberations

u staff insights (adjudicators, facilitators)

u player reflections

u analyze datahttps://usnwc.edu/Research-and-Wargaming/Wargaming/Publications-and-Journals

3/9/19

6

addressing the iteration issue*u two simultaneous games

u some players in both, enabling them to “compare and contrast”

*one example that might not work elsewhere

(Brynen 2016)

bosses. sponsors and clients

3/9/19

7

u In September 2017, Jim Wallman and I ran an opportunistic experiment in conjunction with the DIRE STRAITS megagame held at the Connections UK conference.

u Three teams of generally experienced wargamers (with c130+ years of wargame design/facilitation/play experience amongst them):

u Team USA

u Team UK

u Team Euro

u Each team observed the game, was given access to game documentation (briefings, materials produced by the teams, injects and news reporting), and was asked to separately prepare a report on game methodology and findings.

the DIRE STRAITS challenge

u …the findings were striking. The teams—despite being at the same game—largely disagreed about what they had just seen.. uh oh…

the DIRE STRAITS challenge

*Caveats to follow

3/9/19

8

DIRE STRAITS analysis results

u Not a purpose-designed experiment, but rather an opportunistic one piggy-backed on conference ice-breaker.

u Resources constraints.

u The polarizing effects of US foreign policy in the Trump era was built into the experiment.

experimental bugs and features

3/9/19

9

issue 1: confirmation biasu “tendency to search for, interpret, favor, and recall information in a way that

confirms one's pre-existing beliefs or hypotheses”

u Could arise from political views, organizational/doctrinal views, or even views on wargame methodology.

u preferences for early informationu effects on information searching, assessment, and retention

u impact of Bayesian updating

u the role of “motivated beliefs”

issue 2: framing effectsu How a game looks (name, materials, graphics) can have profound effects on

game play—and, presumably, might also have effects on how it is interpreted.

(Liberman, Samuels, and Ross 2004)

Both versions were identical games of “prisoner’s

dilemma” with identical payoffs.

Reframing the game doubled the rate of

cooperative behaviours (with no other changes).

Framing of game had more effect than anticipated

player style.

3/9/19

10

u Group assessment—especially by diverse groups—enhances analytical performance (“the wisdom of crowds”), if relevant expertise and appropriately aggregated

vs

u Groups may tend to lowest-common denominator analysis, or be excessively influenced by senior participants (“groupthink”).

u aggregation and team-design methods matter

u DIRE STRAITS experiment suggests some value in having analysts undertake “first impressions” separately, before collective discussion.

u The dangers of hierarchy.

u Equally important to collect player perceptions.

u hedgehogs vs foxes (Tetlock 2005)

groups and analysis

u There may be utility in conducting two or more game analyses independently, and then comparing results.

game

Analysisteam

#1

Analysisteam

#2

initial report

initial report

synthesis report Degree of convergence

provides indicator of confidence level.

red-teaming game analysis?

3/9/19

11

dual-purpose games

u Games should be optimized for analytical or training purposes.u However, it may be possible to piggy-back analysis on the back of

educational games, or use analytical games as educational opportunities.u Brynania (radicalization study)

u Brynania (simulation-based learning study)u DIRE STRAITS (analytical experiment)

u MEPP gaming (analysis, education, and policy influence)

u Not all NATO partners have US-levels of wargaming resources.u WARNING: This can only work if you are aware of the pitfalls, and

actively anticipate and mitigate them.