AREVIEWOFONLINEARGUMENTREPRESENTATIONPLATFORMS
AworkingpaperfortheSchoiloproject
https://www.scholio.net
December2018
MartinKing
CentrefortheStudyofDemocracy
UniversityofWestminster,UK
Introduction
Adistinction can be drawnbetween attempts to promotemore deliberative and humble
engagement on common forms of online communication and efforts to design novel
platforms(Shane2004,Davies2009).Researchintocommonformsofonlinecommunication
focusesonlightlystructuredforumsandsocialmediaplatforms,oftennotdesignedexplicitly
tosupportdeliberation(LoukisandWimmer2010).DaviesandChandler(2011)observehow
thisresearchhasrevealedthelimitationsoftheopeninternetandlightlystructuredforums
in supporting online deliberation, highlighting challenges including noise, information
overload,negativebehaviourandpolarisation(Sunstein2001).Researchershavepointedto
designandstructureasameansofaddressingsomeoftheseissuesandchallenges(Pingree
2009,ColemanandMoss2012,Manosevitch2014).
Thesecondstrandofworkononlinecommunicationfocusesonthedevelopmentofnovel
platformsthataimtosupportdeliberation,often,butnotexclusively,atscale(e.g.thereis
workontheeffectivesupportofsmallgroupdeliberationonline).Aparticularlyvibrantsub-
fieldthatwillbethefocusofthispaperistheuseofdifferentformsofargumentmapping
andvisualisationtechniques.Thedevelopersoftheseplatformsareofteninfluencedbyfields
outsideofdeliberativedemocratictheory,notablyinformallogicandcollectiveintelligence.
Althoughtheliteraturehighlightsthepromiseofdesign(Pingree2009,ColemanandMoss
2012), research into design has often been limited (Towne and Herbslep 2012), typically
neglectingmoreexperimentalplatformsandthemorenuancedandnovelformsofdesign
they utilise (Pingree 2009, Delborne et al 2011, Davies and Chandler 2011, Towne and
Herbslep2012).
This paper provides an analysis of platforms and tools that utilise novel design choices
associatedwithargumentmappingandvisualisationtosupportlargescaledeliberation.In
the first section, the paper discussesmethodological challenges in studying this field and
outlines theapproach to identifying, collectingand reviewing relevantonlinedeliberation
platforms.Thepaperidentifies12exemplarycasesandexplorestheseexamplesingreater
detail.Themainbodyofthepaperanalyseseachoftheplatformsindividually,explainingthe
background,objectives,design,existingresearchandapplicationsoftheplatform.
Methodology
TheMethodologicalChallengesofStudyingOnlineDeliberationPlatformsA number of methodological challenges need to be navigated in analysing novel online
deliberationplatforms.Thefirstissueweencounterconcernsdefinition.Inthispaper,wewill
usetheterm“experimentalonlinedeliberationplatform”.Thepaucityofsystematicanalysis
of suchexperimental platforms (TowneandHerbslep2012)means that there is a lackof
agreeddefinitionandclassificationofthisrangeofplatforms.Thischallengeisexacerbated
bythesensethattheterm“deliberation”iscontestedwithnoclearconsensusonitsbasic
characteristics (ColemanandMoss2012).Furthermore,developersof themostsuccessful
andpromisingplatformsareofteninfluencedbyfieldsdistinctfromdeliberativedemocratic
theory(Manosevitch2014).Consequently,howdeveloperstalkabouttheobjectivesofthe
platformmaydiffer fromconcepts familiar todeliberative theory. In this sense, the term
“online deliberation”, if it is used, may be used and understood differently by different
developers,producingverydifferentkindsofplatforms.Finally,manyplatformsandtoolsdo
not support all elements wemight typically associate with a process of deliberation, for
examplenotablyfewplatformsexplicitlysupportdecisionmaking.Insomecases,platforms
havebeenusedincollaborationwithofflineprocesses,ortosupportaparticulartasksuchas
ideageneration.
Sinceaprecisedefinitionforidentifyingexperimentalonlinedeliberationplatformsdoesnot
currentlyexistintheliterature,thisstudyadoptsabroadandflexibleunderstandingofthe
term “experimental online deliberation platform”. This allows the inclusion of platforms
influencedbyotherfieldsthatremainrelevanttothestudybyvirtueofthecontextoftheir
useanddesignfeatures.Thisalsoallowstheinclusionofinnovativeplatformsthatsupport
limitedelementsofthedeliberativeprocess.Asaminimaldefinition,toqualifyforinclusion
in this study a platform should allow more than one participant to express ideas and
argumentsinthecontextofpublicdebate.Thedesignandstructureoftheplatformmustalso
beabletodemonstratesomefeatureaimedatsupportingdeliberationthatdistinguishesit
fromthelightlystructuredformatoftypicalformsofonlinecommunication.
The second challenge concerns our methodological approach to studying the design of
platformsanddesign’spotentialinfluenceondeliberation.Currentresearchintodesignhas
beenlimitedandthemethodologicalapproachesadoptedencounterchallengesandareill
suited to the study of platforms utilising novel or experimental design choices. An initial
challengeisthecontesteddefinitionandconceptualcriteriaofdeliberation(Naurin2007).
Themovefromconceptualcriteriatoevaluativestandardspresentsfurtherchallengesinhow
deliberation,andthequalitiesassociatedwith it,canbeoperationalisedandmeaningfully
evaluated(seeNeblo2007).Inareviewofonlinedeliberationliterature,ColemanandMoss
(2012) raise the concern that studies focus on different deliberative criteria and
operationalise them differently, generating problems for judging the success of different
platforms and comparing performance across platforms. These concerns are particularly
relevant to the studyof design and experimental platforms. Studies haveoperationalised
deliberativecriteriausingmeasuresthatassumespecificdesignchoices.Forexample,Friess
andEilders (2015)discuss researchmeasuringhowoftenparticipants ina forumquoteor
refertoeachotherasameasureofreciprocity.Applyingthismeasuretoaplatformutilising
anargumentmapcannotproduceresultsforameaningfulcomparisonForexample,ifspecific
authors are not identifiable for citation it may measure no reciprocity, or, since every
argumentisconnectedtoanotherpointinthemap,itmaymeasureabsolutereciprocity.In
somecases, researchershave countedwordspermessage to judge,quite indirectly, how
oftenparticipantsjustifytheirbeliefs(JanssenandKies2005).Thisisadeeplyproblematic
measure of justification, that would produce very different results in platforms that
deliberately limit the word count of contributions or emphasise voting over writing. The
designchoicesofexperimentalonlinedeliberationplatformsvarysignificantly,thusexisting
approachestooperationalisingdeliberationthatassumespecificdesignchoiceswouldnotbe
helpfulforstudyingthisgroupofplatforms.
Studiesexploringdesign,suchasDaviesandChandler(2011)andBlack(2011),havediscussed
broad design variables such as differences between synchronous and asynchronous
communication,oranonymousandidentifiableparticipants.Thismaybecontrastedwiththe
morenuancedrangeofdesignchoicesdisplayedbyexperimentalplatforms.Thesevariables
includevariousapproachestointerfacedesign,aesthetics,andchoicesconcerningargument
visualisationandrepresentation.Thesevariablesmaysignificantlyimpacttheexperienceand
behaviour of participants, the quality of deliberation and the success of the platform. A
further element is the sense in which current approaches attempt to establish a causal
relationshipbetweenaspecificdesignchoiceandanoutcomeintermsofdeliberativequality.
Thisapproachrequiresfocusingononedesignchoice,controllingforconfoundingvariables
arisingfromotherdifferencesindesignaswellasexternalfactorssuchasthecontextofits
use. This is a challenging task: Karlsson’s (2010) study of deliberation in 28 identically
designedforumshighlightsthesignificanceofcontextualfactorsraisingconcernsregarding
thecapacityofcurrentapproachestoisolatetheinfluenceofdesign.It isalsoarguablyan
unhelpfulapproachwhenstudyingexperimentalonlinedeliberationplatforms,whichvary
greatlywith respect to theirdesignchoicesandthecontextof theirapplication.Although
Scheuer et al (2010) are discussing computer supported argumentation systems, their
comments on the challenges facing empirical study are pertinent to online deliberation
platforms,giventhesimilarconditionsofvarietybetweensystemsandverydifferentcontexts
ofapplication.Theywrite:
A simple explanation for the lack of studies that systematically compare different
argumentationsystemdesignsisthat it isquitedifficulttopracticallydosuchstudies;
varyingfactorsinacontrolledmannerwouldrequireeliminatingconfound[variables],
which isquitedifficultwhentwoexistingsoftwaresystemsarecomparedasawhole.
(Scheueretal2010:49)
Inviewofthemethodologicalchallengesdiscussedabovethispaperadoptsanexploratory
casestudyapproach.Anexploratoryapproachismostappropriategiventhelackofexisting
researchinthisareaandtheflexibledefinitionofthepopulationofcasesthestudyisadopting
(ShieldsandRangarajan2013,Reiter2013,Schutt2015).Thisallowsthestudytoexplorethe
widerandmorenuancedvarietyofdesignchoicesdisplayedbyexperimentalplatformsunder
thecircumstancesinwhichtheyarefound(Yin1994),allowingconsiderationoftheimpactof
thedesignchoicesofthesystemasawhole.Thestudywillfocusonasmallnumberofonline
deliberationplatformsandtoolsthatrepresentexemplarycases.Althoughthisfocuslimits
thecapacityofthestudytoexplorethefullrangeofpotentialinnovations,itallowsforin-
depth exploration of the experiences of the platforms and the potential of design. The
following section will review experimental online deliberation platforms and discuss the
criteriaforselectingexemplarycasestudies.
SelectingPlatforms
Technologydemonstrationshavedocumentedanumberofexamplesofexperimentalonline
deliberationplatforms(forexampleConklin(2008),Tauroetal(2008)Fishkin(2008),DeLiddo
andBuckingham-Schum(2010b)),andfurtherexampleshavebeenidentifiedandcollected
onthesitesParticipateDBandParticipedia(TowneandHerbsleb2012).ParticipateDBlists
around350toolsandservicesforwebbasedparticipation(ParticipateDB2017).Scheueret
al (2010)provideareviewofcomputersupportedargumentationsystems,someofwhich
relatetothecriteriaforonlinedeliberationplatformsoutlinedabove.MarkKlein(2017)has
producedanoverviewofcollectiveintelligencetools,creatingandbrieflyanalysingalistof
around90platforms, aswell as apaper seeking to categorise available large scaleonline
deliberationplatforms(Klein2015).
ParticipateDBprovidesataxonomyof24classesofplatform,thetaxonomydefinesplatforms
intermsoftheorganisationofinformationaswellastheirpurpose:
Argumentmapping
Audienceresponsesystem
Budgetsimulator
Budgetvisualization
Collaborativedocumentwriting
Contentmanagementsystem(CMS)
Crowdfunding
Discussionforum
E-learningplatform
E-voting
Electronicmailinglist
Groupdecisionsupportsystem(GDSS)
Ideation&brainstorming
Livevideostreaming
Onlineconsultationsuite
Onlinedialogueanddeliberation
Onlinesurvey
Socialmedia
Virtualtownhall
Virtualwhiteboard
Virtualworld
Webconferencing
Weblog
Wiki
(ParticipateDB2017)
Inreviewingpotentialplatforms,thisstudyhasusedParticipateDB,MarkKlein’sanalysisof
collectiveintelligencetools,aswellasasearchforonlinedeliberationplatforms.Thestudy
identifiedminimal criteria to qualify for inclusion in the study, thesewere applied to the
platformsidentifiedfromthesearch.ParticipateDBforexampleincludesmanyplatformsthat
utilise lightly structured forumsor support activitiesdistinct fromdeliberation, suchase-
votingande-learning,andwerethereforenotconsideredrelevanttothecurrentstudy.The
reviewofpotentialplatformsrevealedafurtherneedtoconsiderpracticalissuesinselecting
platforms.Klein(2017)observesinhisdescriptionsthatmanyoftheplatformsandtoolslisted
arenolongeractive(forexampleAhoona),maynothavebeenused,orhavenoevidenceof
largescaleuse(Cluxton).Insuchcases,thereislimitedmaterialavailableforin-depthstudy
andopportunityforassessingthepotentialofdesigninaddressingthechallengesofonline
deliberation.Inviewoftheseissues,theselectionofcasesconsiderspracticalissuessuchas
the availability of material and indications of substantial use of the platform. These
considerationsincludeeaseofaccess,evidencethattheplatformhassupportedlargescale
useorsustainedparticipationoveraperiodoftime,theavailabilityofmaterialandliterature
ontheplatformthatprovideevidenceoftheplatformsapplicationandinfluence.
In addition to basic criteria and practical considerations, the reviewbelow also considers
trendsindesignapproachesandnotableapplicationsoftheoreticalapproachesinfluentialin
the field (such as informal logic and collective intelligence), for example, the use of
argumentationschemestoorganiseandsupportrealworlddebate(e.gParmenides),andthe
use of gamification to support participant engagement (e.g @stake). The selection of
exemplarycasesaimstocapturearangeofapproachesandparticularlysuccessfulexamples
ofthesetrends.
ReviewingandCategorisingPlatforms
Thereislittleexistingresearchtoguideourreviewandcategorisationofexperimentalonline
deliberationplatforms.Klein’s(2015)reviewof“CrowdScaleOnlineDeliberation”isanotable
exampleofanoverviewofthisarea.Klein(2015)identifiesfivedifferentcategoriesofonline
deliberationsystems.Thesearetime-centric,question-centric,topic-centric,debate-centric
andargumentcentric.Thissystemofcategorisationisbasedonhowinformationisstructured
and visually represented. Scheuer et al (2010) provide a review of computer supported
argumentation systems. While this review is not specific to deliberation systems, the
categoriesandpatternsidentifysimilarfeatures;forexample,time-centriccorrespondswith
chatandforums,debate-centriccorrespondswithcontainers,argumentcentriccorresponds
withgraphsinScheueretal’sanalysis.Timecentricsystems,includingblogs,chatroomsand
webforums,maybeunderstoodasrepresentingthemoretypical,lightlystructuredforums
discussed in other literature (Towne and Herbsleb 2012). In question-centric platforms
(examplesprovidedinclude,stockoverflow.com,IdeaScaleandMindjet),acentralquestion
organisesinformation,butthereisnofurtherorganisingorcuratingprinciplethatensures
informationisnotrepeated,noristhereaprocesstohelpidentifypreferableideasoridentify
critiques and critical discussion of ideas. Topic-Centric systems, such as wiki’s, collect
informationarounda topic, andaredescribedas capturing consensuswhile controversial
aspectsofdebatearemovedtotalkpagesthatareorganisedaccordingtothetime-centric
principle described earlier. Debate-centric systems (examples include, whysaurus.com,
Debatepedia.com,debatewise.org(Klein2015))presentinformationintheformofprosand
cons and are often curated to ensure an effective overview of a topic with minimum
redundancyofinformation.Finally,Klein(2015)outlinesargument-centricsystems.Theseare
systemsthattypicallypresentinformationintheformofargumentmapsortrees,identifying
centralquestionsandideas,followedbyargumentsforandagainstandreasonsandevidence
supportingthosearguments.
Klein’s (2015) discussion can be understood as an argument for a particular approach to
online deliberation systems, specifically argument-centric systems ofwhich he is a noted
developer.Thediscussionprovidesausefulapproach tonavigating the rangeofpotential
online deliberation systems: time-centric and question-centric platforms describe lightly
structured approaches that are not relevant to this study. Debate-centric and argument
centricconstitutetwocommonapproachestoorganisinginformationamongstplatformsthat
areclearlyrelevant. Inadditiontotheuseofargumentmapsandtheproandcon listsof
debate-centricvisualisations,thereviewidentifiedtwofurthergeneraltrendsinapproaches
to design and visualisation techniques: annotation and group clustering. Some of the
platforms and tools reviewed use annotation of existing web pages as an approach to
deliberation (for example, Hypothesis and Rbutr). Annotation involves providing an extra
layer of meaning to a given web page by allowing users to annotate it and see other
annotationsincludinglinkstootherpages.Otherplatformsandtoolsrepresentinformation
intermsofgroupclustering,inwhichparticipantsarerepresentedinspacesandgroupedin
suchawayas to reflect their support forparticularpositions (forexampleConsiderItand
Pol.is).Groupclusteringisanotherformofvisualrepresentationwhichallowsuserstoview
the positions of other individuals in a community, for example reflecting how strongly
different users agreed or disagreed with a statement. Group clustering can give a
representation of participants relative to positions without attempting to place those
positions in the context of an argument structure relative to other positions. Such an
approach,canbeunderstoodasavisualisationtechniquethatprovidessocial information
concerningthedebateincontrasttoinformationrelatingtothecontentoftheargument.
Within these general trends, the platforms and tools reviewed display a range of more
nuancedchoicesinrelationtointerfacedesign,aesthetics,andthecombinationofothertools
andtechniquestosupportdifferentaspectsofdeliberation.Forexample,whilewecantalk
generally of the approach of argumentmapping, the platforms and tools will differ in a
numberofrespects:theontologyoftheargumentmapmayvary,withdifferentchoiceson
theelementsthatcanbeusedtocomposeamap;choicesoverwordlimitsorthevisibilityof
repliesandcounterarguments;andthegeneral lookandaestheticsoftheargumentmap.
These factorsmay greatly influence the success of theplatformand its ability to address
certain challenges; for example, aesthetic considerationsmay impact the capacity of the
platformtoattractandmaintainparticipantengagement.Ourreviewofonlinedeliberation
platforms also found that platforms may combine different approaches to argument
visualisation, and they may utilise other tools or design approaches that help address
challengesorissuesassociatedwithagivenapproachtovisualisation.Forexample,ConsiderIt
combines both debate centric visualisations in the form of pro and con tables, but also
displaysgroupclusteringandallowsforfurtheranalysisofthisdata(forexamplethelevelsof
supportforpositionsbasedonthedemographicsofthegroup).TheDeliberatoriumfeatures
bothargumentcentricandtimecentricsystemsofcommunicationintheformofanargument
mapandachatfunction.Finally,someapproachesfoundinpracticedonotfiteasilyintoany
category of visualisation or design approach already identified. For example, Parmenides
drawsonargumentationschemesbutorganisesinformationthroughthepresentationofa
dialoguewiththeparticipant.
Thestudyseekstoselectplatformsthatrepresentnotableorsuccessfulexamplesofsolutions
to the challenge of online deliberation; platforms that combine these approaches and
techniques in interestingways,aswellasnovelanduniqueapproaches.Thepurposeisto
captureasenseofgeneralandnotabletrendsindesignapproachesandexplorehowthey
may address challenges of online deliberation. The study does not seek to make strong
generalisationsfromtheexperiencesofselectedplatforms,whichgiventhediversityofthe
populationofcasesandmethodologicalapproachwouldbeproblematic. Insummary, the
selectioncriteriaareoutlinedbelow.
• Theplatformortoolmeetstheminimalcriteriaofsupportingoneormoreaspectof
online deliberation and demonstrating novel and relevant design features that
distinguishitfromlightlystructuredforums.
• Practicalconsiderations:easeofaccesstotheplatform,evidencethattheplatformis
successfulandhasreceivedlargescaleorsustainedparticipation,theavailabilityof
materialandliteratureonapplicationsoftheplatform.
• Instanceswherethedesignchoicesrepresentnotableapplicationsofcommontrends
oruniqueapproachestovisualisationandstructurethatcanbe linkedtoengaging
withthechallengesofonlinedeliberation.
Wherepossibletheplatformsandtoolsavailableweretrialledandreviewedaccordingtothe
criteria above. From this process, a selection of platforms and tools were identified as
exemplarycasestudiesforfurtherexploration.
ExemplaryCasesofOnlineDeliberationPlatforms
Twelveplatformsandtoolswereselectedasexemplarycasesforfurtherstudy,theseare:
@stake
ArvinaandOva
BCisive/Rationale
ClimateCoLab
Cohere/EvidenceHub
ConsiderIt
Debategraph
Deliberatorium
Parmenides
Pol.is
Rbutr
Truthmapping
Thissectionprovidesadescriptionofeachoftheseplatforms;providinginformationonthe
backgroundof theplatform,design features, illustrative screenshotsof theplatform, and
reviewsexistingstudiesandapplicationsoftheplatform.
@Stake
Background,ObjectivesandDesignofthePlatform
@Stakeisa“roleplayingonlinecardgamedevelopedtofosterempathyandcollaboration”
(eLab2017a:1)and“enhancedeliberationinreal-worldprocesses”(eLab2017b:3).@Stake
wasdevelopedby the Engagement Lab, anorganisation that seeks toproduce games for
socialchange,participationandengagement(eLab2017a:1).Thedevelopersdescribethe
game as relying on “rapid fire ideation, discussions facilitated through role playing,
experimentation with ideas, and collaboration among a diversity of stakeholders” (eLab
2017b:3).
@Stake isacardgamethat takesaboutonehour toplay. It requires thedivisionof large
groupsintosmallergroupsof4-5players.Participantsareassignedcharactersthroughcards.
Thecardcontainsbiographicalinformationandanagenda,visibleonlytotheparticipant.The
agendaprovidesdetailsofthecharacter’sobjectives,withpointsattributedtoeachobjective.
One person of the group is elected as Decider for the round, with which comes various
responsibilities.Tokensformacurrencyofthegame,eachplayerisgiventhreetokens,with
anadditionalthreetokensinapotandfivetokensfortheDecider.
Distributionoftokens(eLab2017b:4)
Charactercard(eLab2017b:11)
Theroundsconsistofthefollowingstages:
Introduction:Participantsintroducethemselvesincharacter
Brainstorm: TheDeciderannounces the issue,participantshaveoneminute todevelopa
proposal
Pitch:MovingclockwisefromtheDecidereachplayerhasaminutetopitchtheirproposal.
Theymayusetokenstoallowthemselvesextratime.
Deliberate:TheDeciderleadsafollowupdiscussion,participantsmayaskeachotherabout
theirproposals,offercounterargumentsandsuggestamendmentstooneanother’splans.
Tokensmaybeusedtoextendtime.
TheDecision:TheDeciderannouncesawinningproposal.Theplayerwhoproposedthiswins
allthetokensinthepot,plusbonustokensbasedonpointsontheiragenda.Allotherplayers
score points for their agenda items, if thewinning proposal satisfied these requirements
(determinedbytheDecider).ThewinnerthenbecomestheDeciderforthenextround(or
alternativelypassesthisontosomeoneelse)(eLab2017b:5-6)
@stakewasinitiallydevelopedasafacetofacecardgamein2014,thedevelopershavesince
producedadigitalmobileversion(EngagementLab2017).Thisversionsimplifiestherules,
reducestheneedforfacilitationandallowsforeasiertrackingofideasgeneratedduringthe
gameandtheexperiencesof theplayers (Gordonetal2016).Thegamewouldappear to
requirebespokecardsforeacheventitisappliedto,withcharactersandagendasrelevantto
theevent.Incontrasttomanyoftheotherplatformsandtoolsexploredinthisthesis,the
developers make explicit reference to deliberative democratic theory and the desire to
supportthegoalsandoutcomesofdeliberativedemocracy(Gordonetal2016).@Stakeand
theotherprojects developedby Engagement Lab canbeunderstoodas themost explicit
example of “gamification”, as discussed in the collective intelligence literature, available
amongst the platforms and tools reviewed. Although it aims to support deliberation and
decision making in general, the developers also stress the role it can play in supporting
empathybuildingbetweenparticipants(Gordonetal2016).
ImagesandScreenshotsofthePlatform
Anexampleofthecardsisreproducedbelow1.
LiteratureandApplicationsofthePlatform
Inadditiontoinformationontheengagementlabwebsite,thedevelopersproducedapaper
forthe2016CSCW(ComputerSupportedCooperativeWork)conferenceabouttheplatform.
Thispaperidentifiesalackofresearchinto“civicgames”,itoutlinestherulesandmechanics
1https://medium.com/engagement-lab-emerson-college/announcing-a-printable-stake-game-211f76804086#.uy36q159k
of the game and describes a pilot study comparing the game to a traditional ice breaker
(Gordonetal2016).Thepaperreportsthefindingsofthepilotstudyaspositive,findingthat
roleplayingmayencourageparticipantstobemorecomfortablewithpublicspeakingand
engagement, aswell as greater capacity toempathisewithothers and retain information
abouttheirideasandpersonalities(Gordonetal2016).Michelson(2015)alsoreportsonthe
resultsofplaytestsofthegames;theplatformwasusedasawarmupactivitybyarchitecture
andplanningdesignfirmUtileintheImagineBoston2030initiative.@Stakewasalsopiloted
inthreeParticipatoryBudgetingmeetingsinNewYorkCityinAutumn2014(eLab2017a).A
studyof thiseventusedparticipantobservation, surveydata,and followup interviews to
evaluateanumberofissuesincludingwhethergameplayincreasedempathy,efficacy,affinity
towardscivicengagement,andfutureparticipation.Oneoftheobservationsofthisstudywas
thesenseinwhichsomeparticipantsobjectedtospendingtimeonagame(eLab2017a).The
gamehasbeenusedinavarietyofsettingsincludingorganisationalplanning,UNDPpolicy
meetings on youth unemployment inMoldova, Egypt and Bhutan, educators’ curriculum
designworkshops,andseveralacademicconferences(Gordonetal2016:271,eLab2017a).
ArvinaandOva
Background,ObjectivesandDesignofthePlatform
Arvinaiswebbaseddiscussionsoftware,that“allowparticipantstodebatearangeoftopics
inreal-timeinawaythat isstructuredbutatthesametimeunobtrusive”(Lawrenceetal
2012:1).ArvinaandOVAweredevelopedbytheARG-tech,CentreforArgumentTechnology
attheUniversityofDundee.ARG-techdevelopstoolsaimedatargumentmining,argument
visualisationandanalysisandtheuseofartificialintelligenceindialogue.Thecentrehasbeen
influential in this field and has worked in collaboration with the BBC and IBM debating
technologies(Reed2017).ArvinaandOVAaretoolsdevelopedtosupportargumentation.
ArvinaandOVAareanapplicationofArgument InterchangeFormat (AIF) theory thathas
beenusedtosupportpoliticaldebate.OVA(OnlineVisualisationofArgument)isatoolfor
analysingandmappingargumentsonline.The interfaceallowsuserstohighlighttextona
webpageandextract this to apremisewhich canbeused to supportor challengeother
premises.Missingpremises(orenthymemes)canalsobeaddedbyusers.Arvinaisadialogue
tool thatusesgooglewave,anonline toolwhichallows for real timecommunicationand
collaboration(Google2010 inSnaithetal2010:7).ArvinaisaWaveapplicationwhichbuilds
on the Google API, allowing a user to choose a topic from any previously analysed AIF
resources. The AIF resource is examined to determine the participants involved in the
dialogueandanewrobotisaddedtothewaverepresentingeachoftheseparticipants.The
participantsmaybehumanorartificial,withartificialparticipantsusingknowledgeassigned
fromtheAIFresource.
ArvianandOVAarerelevanttothecurrentstudyforanumberofreasons.ArvinaandOVA
representanexampleoftheuseofartificialintelligenceinonlinedeliberationplatforms,and
anattempttoprovidesomeformalstructuretonaturallanguageargumentthatcanbeused
acrossdifferentplatforms.ArtificialintelligenceandparticularlytheworkofChrisReedand
ARG-techweredescribedbyanumberofotherdevelopersinthisareaasbeingparticularly
influential and promising as a direction for online deliberation. This is an area that is
particularlyrelevanttoissuesoffeasibilityandscaleaswellasinformationmanagement.
ImagesandScreenshotsofthePlatform
ArvinaInterface(Lawrenceetal2012:2)
PremisepropertiesinOVAShowingParticipants(Snaithetal2010:5)
LiteratureandApplicationsofthePlatform
ArvinaandOVAwereappliedinthecontextofaproposaltobuildatransmissionlineforthe
BeaulytoDennypowerlinethroughareasofoutstandingnaturalbeauty(Snaithetal2010).
Thepurposeofthestudywastotestthecapacityforinterchangebetweendifferentformats,
argumentvisualisationanddialogueformatusingAFI theory.Thestudyobservedthatthe
toolsdemonstratethatformallydescribableprocessesofdeliberationcanbelinkedtothe
formallydescribablestructuresofknowledge.Itclaimedthatmuchremainedtobedoneto
expand and refine the tools, and integrate the toolswith other parts of the deliberative
process,suchasinquiryanddecisionmaking.Lawrenceetal(2012)testedArvina’scapacity
to support human and artificial intelligence groups in mixed initiative argumentation.
Lawrence et al (2015) discuss the challenges of current argument mining techniques to
identify complex structural relationships between concepts, a lack of consistency in
formatting,andalackoflargequantitiesofappropriatelyannotatedargumentstoserveto
trainandtesttools.Ineffortstoaddressthis,researchershaveturnedtootheronlinetools
(including several discussed in this study,Debategraph, Truthmapping andRationale) and
soughttoconvertthemtoAIFformats(Lawrence2015).
BCisive/Rationale
Background,ObjectivesandDesignofthePlatform
BCisiveallowsthecreationofdecisionmapsto“capturediscussion,organiseideas,explore
options, testhypothesisandanalyse reasoning” (BCisiveOnline2017).Rationaleclaims to
allowusers tomake argumentmaps to “structure arguments, analyse reasoning, identify
assumptionsandevaluateevidence”(RationaleOnline2017).BCisiveandRationalearetools
developedbyAusthinkandrepresentcommercialsuccessorstoReason!Able(Scheueretal
2010).ThetoolsarecurrentlyrunbytheReasoningLab;bothtoolsallowforthecollaborative
constructionofargumentmaps.Thesoftwarecodeforthetoolsareidentical,howeverthe
interfaceallowsfordifferentoptionsinmapconstruction(Kunschetal2014).BCsisiveaims
atsupportingdecisionmakingwhileRationaleisusedasaneducationaltoolforsupporting
criticalthinkinganddevelopingargumentinessays.
Thedescriptionsfortheargumentvisualisationontologyaredifferentforeachtool.Thebasic
ontology of BCisive allows for the construction of maps from the following: Situations,
Options,Pros,Cons,Reason,Objection,Evidence,CounterEvidence,Questions,Challenges
andaFix.InthecaseofRationale,mapsarebasedonthefollowingontology:contention,
reason,objection,note,exampleandco-premise.Therearefurtheroptionstoidentifythe
natureofdifferent typesofevidenceorbasis including:Commonbelief,data, case study,
assertion,law,quote,statistic,personalexperience,andpublication.
BcisiveandRationalearecharacteristicof theargumentmapping toolsavailable inonline
deliberation. They allow relatively extensive options in constructing argument maps,
including task specific items. Participants are free to construct maps and use categories
however theychoose,however thereareprompts.Forexample, the“rabbit rule”prompt
alertsparticipantsifsomethingismentionedintheconclusionthatisnotmentionedinthe
reasosn(Twadry2004).ThedevelopersinvolvedinRationalearecurrentlydevelopingtheuse
of probabilistic judgements in argument maps, which would affect the way the system
organisesthemap.Thisisparticularlyrelevanttothechallengeofinformationmanagement
(howwell platforms canmanage different data types). In addition to these features the
developersarealsodevelopingwaysofallowingrealtimecollaborationonargumentmapsin
whichauthorscanseethechangestheirpartnersintendtomakeinrelationtoanargument
map.InthissenseBcisiveandRationalearegoodexamplesofargumentmappingapproaches
withmorenovelfeaturestosupportdecisionmakingandcriticalthinking.
ImagesandScreenshotsofthePlatform
BCisive
ArgumentmapinterfaceforBCisive2
2https://www.bcisiveonline.com/editor/
Rationale
Rationale argument map editor interface3
Rationaleargumentmapexample4
3https://www.rationaleonline.com/editor/4https://www.rationaleonline.com/editor/#?id=8ek8jh
LiteratureandApplicationsofthePlatforms
A literature review reveals seven papers on Rationale, Bcisive and an earlier version of
rationalecalledReason!Able.Scheueretal(2010)includeRationaleandReason!Ableintheir
reviewofCSAsystems.Kunschetal(2014)comparestheuseofBcisiveandRationaleinthe
contextofeducationforbusinessstudents.TheysuggestthatRationalewaspreferablefor
demonstratingthebasicsofargumentmapping,whileBCisivewasbettertoanalysebusiness
cases and present findings due to its capacity to conduct more complex analysis and
compatibilitywithformatssuchasPowerPoint.Lengbyer(2014)usesrationaletoexplorethe
useofargumentmappingtosupportdecisionmakinginspecificcasesratherthanforgeneral
educational purposes. There is discussion of the use of Rationale to support lawyers in
Australia (Drummond2006 inVanGelder2007)andhelping judgeswithexpert testimony
(vanDrielandPrakken2010),thelaterconcludingthatitwasmorelikelytobehelpfulfor
educationalandtrainingpurposes.Rationalehasbeentestedinothereducationalsettings
finding positive results when testing students’ critical thinking skills (Tawdry 2004) and
understandingofthematerial(Davies2009b).TheliteraturehastendedtofocusonRationale
oritsearlierversionsinaneducationalcontext.BCisiveappearstobealaterdevelopment
intendedforuseintheprivatesectorasatoolforsupportingdecisionmakinginorganisations
(ReasoningLab2017).TherehasbeenmoretakeupfortheRationaletool,whichhasbeen
usedinvariouseducationsettingsbystudentsandaspartofcourses,notablyinAmsterdam
andAustralia(ReasoningLab2017).Therearefreeonlineversionsavailableforbothtools.
ClimateCoLab
Background,ObjectivesandDesignofthePlatform
ClimateCoLabisdevelopedbytheMITCenterforCollectiveIntelligence.Itdescribestheaims
oftheprojectasattemptingtoaddresswickedproblems,specificallyclimatechange,through
collective intelligence.Theprojectappeals to the idea that it iscreatinganopenproblem
solvingplatformdrawingonthesuccessofprojectssuchasWikipedia(Maloneetal2009).
Climate CoLab involves contestswhere participants can put forward and discuss ideas to
addresstheproblemofclimatechange.Acontestconsistsofdifferentstagesandelements.
People can propose solutions to specific problems identified on the site, such as land
managementandenergysupply.Thereisafurtherstagewherebyproposalsareintegrated
towardthedevelopmentofacomprehensiveplanthatcouldbefeasiblyadoptedasnational
policy.A final stage considerswhether the comprehensiveplanmeetsestablished targets
(Maloneetal2009).Theproposalsareassessedbyexpertjudges,thoughthereareplansto
replacethisprocesswithaformofcrowd-basedassessment.ForthefirstthreeyearsClimate
CoLabinvolvedoneortwocontestsperyear,from2013theyintroducedcontestfamiliesof
17ormoreconteststhatseektobreakdowntheissuesofclimatechangewhicharethen
integrated(Maloneetal2017).
ClimateCoLabinvolvesthreedesignelementssupportingcollectivedecisionmaking:model
basedplanning,onlinedebatesandvoting.Modelbasedplanningallowsparticipantstouse
simulation models to provide information about the impact of different proposals. The
literaturedescribeshowthesystemusestheC-LEARNmodel,anonlineversionofC-ROADS,
aclimatechangepolicysimulator(Maloneetal2009).C-LEARNtakesasinputasetofregional
commitments to emission reductions and produces as outputs projections of carbon
concentration,temperaturechangeandsealevelrise.Theseoutputsarethenusedtodrive
elevenadditionalmodelsusedbyClimateCoLab,whichpredictanticipatedeconomiccosts,
qualitativeimpacttohumanandphysicalsystems(suchasagriculture,waterandhealth).
Online debates utilise a system similar to Deliberatorium and Compendium (see later),
providinggreaterstructurethantraditionalforumsandclassifyingeachcontributionas(1)a
question,(2)aposition(proposedsolutiontothequestion),(3)anargumentfor,or(4)an
argumentagainst.Somedebatescaptureargumentsandinformationthatcutacrossissues
thatunderlieanumberofdifferentplans,thusplancreatorsareencouragedtospecifywhat
positionstheirplanstakeoncrosscuttingissues(Iandolietal2008).Finallyparticipantscan
vote on debate positions and plans that they prefer, allowing users to identify promising
proposalsforthecontests.Cashprizesareawardedtotheproposalsthatarejudgedtobe
thebestoverallinthecontest.
ClimateCoLabisrelevanttothecurrentstudyforthefollowingreasons:ClimateCoLabisan
ambitiousandwellestablishedplatformthathasalargenumberofparticipants.Theplatform
providesadecisionmakingmechanisminadditiontoideagenerationanddebate.Through
theclimatechangepolicysimulatorandtheuseofjudgesandmoderators,theplatformalso
seeks to verify information provided in debate and ground claims made into a shared
understanding of the facts. These features are of particular relevance to the theme
information management, and issues concerning how platforms deal with contested
knowledge,moderationanddifferentdatatypes.Theprojectisalsowellresourcedasit is
able to award $10 000 to the best overall proposal per contest. This is significant when
thinkingaboutfeasibilityandscaleandthesustainabilityofonlinedeliberationplatforms.The
platform uses a number of different techniques to address different challenges in online
deliberation and represents an interesting approach drawing from the field of collective
intelligence.
ImagesandScreenshotsofthePlatform
Introneetal(2011:5)
SimulationModel
ClimateCoLabDebateInterface(Introneetal2011:6)
LiteratureandApplicationsofthePlatform
TheClimateCoLabhasbeenconductingcontestssince2009andtheseapplicationshavebeen
documentedindifferentresearchpapers(seeforexample,Maloneetal2009,Introneetal
2011).Theprojecthasover85000participantsincluding200expertsonclimatechangeand
relatedfields(Maloneetal2017).Duhaimeetal(2015)usedonlinesurveysandananalysis
ofwebactivitytodevelopapictureofthecharacteristicsandbehaviouroftheClimateCoLab
security. It foundthatthecommunitywasgeographicallydiverse,andtendedtobehighly
educatedandexperiencedwithclimatechangeissues.Italsofoundthatthoseoutsidethe
usual conversations about climate change are influenced by and contribute effectively to
collectiveproblemsolving.Memberswhodidnothavegraduateeducationpreviousclimate
changeexperienceordidnotliveintheUnitedStatesreportedsignificantlyhigherlevelsof
learning,beliefchangeandincreaseinclimaterelatedactivityasaresultofparticipation,and
thesemembers,andwomen,wereatleastaslikelytosubmithighqualityproposals(Duhaime
etal2015).Throughthelaterimplementationofcontestwebs,Maloneetal(2017)explored
whether participants would reuse their own and other’s work effectively, and whether
participantswould be able to exploremultiple combinations of interchangeable parts (of
solutions).Theresultsoftheirexperimentwerefoundtobepositive,observingtheirsystem
facilitated widespread knowledge sharing and reuse, and the combining of solutions at
multiplepointsofaggregation(Maloneetal2017).
Cohere/EvidenceHub
Background,ObjectivesandDesignofthePlatform
CohereisaprojectdevelopedbytheKnowledgeMediaInstituteattheOpenUniversity.Itis
a visual tool to create, connectand share ideas.DeLiddoandBuckinghamShum (2010b)
identifiescontestedcollective intelligenceasadistinctareaofcollective intelligence,with
Coheredevelopedasaprototypefortestingtheirdesignrationalefortheseideas.Evidence
Hubispartofthisprojectandaimstoprovideaplatformforcollaborativeknowledgebuilding
basedon theconceptof contestedcollective intelligence (DeLiddoandBukinghamShum
2013),allowinguserstopoolandmapknowledgearoundaspecificissueortheme.
The evidence hub organises information according to the following categories: key
challenges, potential solutions, research claims, evidence and counter evidence, and
resourcesontheweb.Inadditiontothisattempttopoolrelevantknowledgearounddifferent
individuals and researchers working on projects, this is mapped geographically and also
accordingtothemesandquestions.
EvidenceHubisanambitiousprojectthatutilisesanumberofdifferentapproaches,including
annotation,argumentvisualisation,chatfunctionsfordebatestosupportcollaborationand
deliberationaswellasprovidingaresourcetocollectevidencearoundagivendebate.Itsuse
ofvariousapproaches toargumentvisualisationand informationmanagementmake itan
interesting case for further exploration. The developers draw explicitly on collective
intelligenceliteratureaswellasinformallogictheoristssuchasWalton(EvidenceHub2017)
andthetoolscanbeunderstoodasamatureand influentialexampleofanapplicationof
thesetheoriesinpractice.
ImagesandScreenshotsofthePlatform
MappingArgumentationChainsintheEvidenceHub5
EvidenceMap6
LiteratureandApplicationofthePlatform
The E-Hub website describes examples of hubs running in collaboration with external
partners, includingtheCommunityofPractice for the InstituteofHealthVisiting (aclosed
Hub) and the Systems Learning& LeadershipHub (University of Bristol). OpenUniversity
CommunityHubs includetheOpenUniversity inScotland’sWork&LearningHub,andthe
OpenUniversity’sFacultyofEducationandLanguageStudiesdepartmentthroughtheirHubs
forReadingforPleasure,andResearchbyChildrenandYoungPeople(EvidenceHub2017).
De Liddo and Buckingham Shum (2013) describe the concept of Evidence Hub and its
developmentinresponsetoexperiencesfollowingitsuseinthecontextofhealthcareand
education.Thisworkhighlightedapervasivechallengeofatrade-offbetweentheneedfor
structuretomaximisethesignal-to-noise-ratioandpermittingpeopletomakecontributions
5http://rcyp.evidence-hub.net/?max=20&orderby=date&sort=DESC&filternodetypes=Challenge#home-list6http://rcyp.evidence-hub.net/explore.php?id=861572362500105612001342686197
withvery little indexingorstructurethatrequires less learning.Theysuggestthatthis isa
problemthatisfarfromsolved.
ConsiderIt
Background,ObjectivesandDesignofthePlatform
ConsiderIt “helps individuals make sense of complex issues through familiar deliberative
activities” (ConsiderIt 2017). It is described as a novel platform for supporting public
deliberation on difficult decisions (Kriplean et al 2012, Kriplean et al 2011). The platform
allows users to create forums and introduce questions or proposals for a community to
address.Thecommunitycanthencontributetothisforumbyidentifyingtheirpositionona
scaleofagree/disagreeorhighpriority/lowpriority,andselectingthemost importantpro
andconpointsforagivenposition.ConsiderItthenpresentsavisualrepresentationofthe
communityasawhole,itshowsopinionsalongascaleofagreetodisagree,alongwithalist
ofrankedprosandcons.
ConsiderItcombinesanumberofdifferentapproachestovisualisingargumentanddebate.It
representsprosand cons toagivenproposal,but it alsomapsgroups to show the social
contextofanindividual’sposition.Userscaninteractwiththisvisualisationtoidentifygroups
withsharedopinionsandpointsofconsensusamongstotherwisedisparateparties. Inthis
sense,thesystemallowspeopletovoicetheiropinionswhilealsogivingthemtheopportunity
to recognise areas of agreement with political opponents. ConsiderIt also allows for
potentiallymorenuancedunderstandingofdifferencesbyallowingparticipantstoarticulate
differentprosandconsandalsoplace levelsofpriorityonargumentsandproposals.The
developersarguethissupportsempathy,mutualunderstandingandareasofconsensus.For
example, if80%ofpeoplewhoopposean ideashare thesametwoconcerns thatcanbe
resolved,thissuggestsanopportunityforaddressingtheconflict (Freelonetal2012).The
developersoftheplatformhavealsoattemptedtoaddresstheproblemofverificationoffacts
inonlinedeliberation,albeitexternallyinapplicationsoftheplatform.
The featuresdescribedabovedistinguishConsiderIt from theother available examplesof
platformvisualisationsusingprosandconstables.ConsiderIthasalsobeendescribedasusing
gamificationinitsapproachtotutorialsandaesthetics.Giventhesophisticationofitsdesign
choices and thewell documented applicationsof its use, itmakes an interesting case for
furtherexploration.
ImagesandScreenshotsofthePlatform
ThescreenshotsbelowillustratethedesignofConsiderIt,andtheprocessofgeneratingand
contributingtoaforumdiscussion.
Displayingausercreatingaforumandpoliciesforacommunitytodiscuss7
7https://consider.it
Displaystherepresentationofacommunityinrelationtotheprioritythatisplacedongiven
topicsofdiscussionorpolicies8
Displayingthevisualisationofaspecifictopicorpolicy,allowinguserstoseetopprosand
consandthegeneralconsensusofthegroup9
Displayingauserintroducinganargument10
8https://consider.it9https://consider.it10https://consider.it
LiteratureandApplicationsofthePlatform
TheliteratureonConsiderItshowsithasbeenappliedandtestedinanumberofsituations.
Itbeganaspartofaprojectcalledthe“LivingVotersGuide”,thatincludedanexperimentin
aU.Sstateelectionallowingresidentstodebatenineballotmeasures(Kripleanetal2012).
AlthoughtheLivingVotersGuideisnolongeractivetheplatformcontinuestobeappliedin
othercitizenengagementprojectsanditisalsoavailableforfreeforpublicuseandthrough
a paid planwith additional features (ConsiderIt 2017). Research has tested howwell the
platform encourages engagement with different views (Freelon et al 2012), participants’
perceptionsofdifferentstandpointsandtheirownknowledgeofthesubject(Stieglerandde
Jong2015).Thesestudieshavegenerallyfoundpositiveresultsfortheplatform,additionally
differentvariationsoftheplatformhavebeentestedinthecontextofadebateonGreece
andtheEuropeanUnion(StiegleranddeJong2015).Thedevelopersoftheplatformhave
alsoengagedwiththeproblemofverificationandthetrustworthinessofsourcesandclaims
appealedtoinonlinedeliberation(Freelonetal2012andKripleanetal(2014)).Kripleanet
al (2014)trialledtheuseof librariansasfactcheckers inoneexampleoftheLivingVoters
Guide.Thetrialusedaquantitativeanalysisoftheuseofthefactcheckingservices,finding
14.2%ofclaimsweresubjecttofactcheckingrequests,andhalfoftheseconcernedclaimsof
factwhile theothers involved claimsof principleor other claims thatwerenot verifiable
(Kripleanetal2014).Thetrialalsoevaluatedtheexperienceofparticipantsandlibrarians.It
foundthattwothirdsofthosewhohadhadtheirsubmissionsfactcheckedfelttheprocess
hadbeenfair(noneclaimedthatithadbeenunfair),whilemanyusersexpresseddesirefor
bettercommunicationwiththefactcheckers(forexampletheabilitytorespondtotheresults
ofthefactcheck).Thelibrariansreportedpositiveexperiencesoftheprocess,theyfeltthey
wereabletoconductfactcheckinginaneutralmanner,althoughsomehighlightedthefact
thattheyfelttheylackedthelegalexpertisetocorrectlyrespondtosomeissues.Whilethe
studyfoundtheprocesswasbroadlysuccessful,theyacknowledgedproblemswithapplying
thisapproachtolargescaledeliberation(Kripleanetal2014).TravisKriplean(thedeveloper
oftheplatform)alsodescribeshowtheplatformhasbeenusedwiththebitcoincommunity
and other open source communities, in large organisations to support strategic planning
effortsand inschoolsasaneducationaltooltosupportcritical thinking.Healsodescribes
future plans to use the tool in join initiatives project in Hawaii, involving the Hawaiian
languageinanumberofschools.
Debategraph
Background,ObjectivesandDesignofthePlatform
Debategraphisdescribedasaservicewhichallowsindividualsand“communitiesofanysize
toexternalize, visualize,questionandevaluateallof theconsiderations thatanymember
thinksmayberelevanttothetopicathand”(Debategraph2017).Thevisualisationspresent
colourcodedmapsbasedonthefollowingcriteria:issues(orquestions),positions,arguments
fororagainst.Themapsareopentoeditingbythegeneralpublic,andtheideassubmittedin
themapcanberatedbyothers.Thestrongestargumentsareindicatedbythewidthofthe
arrowsconnectingtheideasinthemap.Theontologyoftheargumentmapisnotexplicitly
groundedinaparticulartheoryofargumentation;thebasicbuildingblocksofthemapshare
similaritieswithotherargumentmaps,howeveritdevelopsamuchmorecomplexrangeof
connections between ideas and relationships betweenmaps. In addition to an extensive
rangeofconnections,themapsalsoflowintooneanother,allowingparticipantstonavigate
fromoneissuetoanother.TheontologyoftheDebategraphisoutlinedinthevisualisations
below:
Illustrationofthebasicontologyoftheargumentmap11
Following this basic ontology the system develops a more elaborate set of connections
betweenideas,identifiedwithdifferentcolouredarrows.Thetablebelowdetailstherange
ofconnectionsavailableforideas.
11http://debategraph.org/Stream.aspx?nid=400384&vt=bubble&dc=focus
Tabledetailingrelationshipsbetweenitemsonamap12
Inadditiontothehierarchicalstructureformingthebasisofthemaps,therearealsoaseries
ofconnectionscalled“crosslinks”whichprovidelinksbetweenmapsorinformationabout
themapitself(forexampleinstancesofinconsistenciesorequivalence).Thesearedetailedin
thetablebelow.
12http://debategraph.org/Stream.aspx?nid=400384&vt=bubble&dc=focus
Tableshowing“crosslinks”,relationshipsbetweenitemsonthemapandothermaps13
Debategraph provides a side menu offering further help in construction of maps and
additional details; there are alternative ways of viewing the maps which allow for the
inclusionof imagesandvideos. Themaps canbeembeddedonother sites,with changes
madetoamaponagivensitesharedacrosstheothermaps.Themapscanbenavigatedsuch
that when a user clicks on a particular element of themap, the perspective of themap
changes revealing further connections. In this sense Debategraph hopes to capture the
interconnectednatureofmanyissuesthataresubjecttoargumentmapping.
Incomparisontoplatformstakingasimilarapproachtoargumentmappingandvisualisation,
Debategraphcouldbesaidtoofferaricherexperience inrelationtothekindsofmedia it
supports, the options it allows for visualisation and navigation, and the aesthetic of the
platform.Theplatformincludesanumberofnovelfeaturesnotfoundinmorebasicargument
mappingplatforms,notably themaps’capacity to rearrangethemselvesaroundparticular
pointsasusersnavigateandtheextensiveontologyofthemaps.Thisprovidesaninteresting
approach to issues of framing and structuring debate. Of the platforms reviewed,
Debategraphisoneofthemostsuccessfulinrelationtoapplicationsbyotherorganisations
13http://debategraph.org/Stream.aspx?nid=400384&vt=bubble&dc=focus
andexistingresearchandliterature.Therelativesuccessoftheplatformandnoveldesign
choicesmakeitveryrelevantforthepurposesofthecurrentstudy.
ScreenshotsandImagesofthePlatform
Thescreenshotsbelowprovideillustrationsofthedesignofdebategraph.
OpeningpageofDebategraph14
ExampleofDebategraphmap15
14http://debategraph.org/home
15http://debategraph.org/Stream.aspx?nid=11474&vt=bubble&dc=focus
(Tambourisetal2011:4)
LiteratureandApplicationsofthePlatform
TheDebategraphsiteclaimsthattheplatformisbeingusedinover100countriesandlists
applications inareas including “education,health, governance,media, conferences, group
facilitation,conflictresolutionandpublicconsultationandplanning”(Debategraph2017).It
hasbeenusedbyorganisationssuchasCNN,theWhiteHouse(onopengovernment),theUK
Prime Minister’s Office (on media policy), The Independent, Bill and Melinda Gates
Foundation(onglobalhealth),andtheForeignOffice(BullenandPrice2015,Debategraph
2017).Tambourisetal(2011)observethatDebategraphisoneofthemostmatureandstable
examplesofargumentvisualisation tools,and therehavebeenstudiesusing the tool ina
number of different contexts. Bullen and Price (2015) explore the use of Debategraph in
supporting analysis of complex policy problems, specifically obesity. One policy maker
involved in the studyemphasised thepotentialofDebategraphasamethodof collecting
variousdifferenttypesofdataonanissueandpresentingitclearly,withlessinterestinits
capacity to support debate (Capehorn in Bullen and Price 2015). Crossley-Frolick (2017)
explores theuse of debategraph in educating undergraduate political science students. A
classuseddebategraphtoengage indebatesconcerningcomplex issuessuchaspolicyon
climatechangeandsextrafficking.Crossley-Frolickfoundthatthetooldidimprovestudents’
understandingofthetopic,yetthestudentsreportedissueswiththenavigationsystemand
easeofuseoftheplatform.Afurtherissuehighlightedwasprivacy,asuserswhowerenot
partoftheclassbeganeditingdebates.Tambourisetal (2011)studiedtheexperiencesof
policymakersandexpertsusingDebategraphinthecontextofEuropeanlegislation.Inthis
studyDebategraphwas used in conjunctionwith other software calledWAVE. The study
foundmixedresultsagainwiththeusabilityoftheplatform,thoughparticipantsnotedthat
easeofuseimprovedafterashortlearningperiod.Participantsnotedthattheplatformwas
attractiveandwouldbemostappropriateforanalysisanddraftingandevaluationofpolicy,
as well as consultation on policy, while it would be less useful for formulation or
implementationofpolicy(Tambourisetal2011).Scheueretal (2010)noteDebategraph’s
support for large scale argumentation and large community use. They highlight potential
problemswithgraphicalrepresentationbeingusedfordebates,notablythattheymightfeel
unnaturalandunintuitiveanddependingonthetopicandnumberofparticipants,theboxes
andarrowsmaybesubstantialleadingtothemapsbecomingclutteredandhardtoread.
Deliberatorium
Background,ObjectivesandDesignofthePlatform
TheDeliberatorium(formerlyknownastheCollaboratorium)isdescribedasan“innovative
internet tool whose goal is to enable better collaborative deliberation” (Deliberatorium
2017). The project is developed and led byMark Klein atMIT and draws on the field of
collectiveintelligenceandspecificallytheIBIStomaparguments.Discussionisorganisedby
topicandbrokendownintothefollowingcomponents:
Issue:Aproblemthatneedstobesolved
Idea:Anapproachforaddressingthatissue
Argument:Apointfor(pro)oragainst(con)anidea(Deliberatorium2017)
The literatureon thedeliberatoriumdescribes the followingexpectationsofauthors, that
authorssubmitasingleissue,ideaorargument,thatitnotreplicateapointthathasalready
beenmade,andshouldbeattachedtotheappropriatepartofthemap.Postsshouldonlybe
editedtostrengthenthem,ifonedisagreesoneshouldcreateanewpostthatcountersthe
idea(the liveand let liverule) (Klein2011).Toguideargumentation,moderatorsevaluate
postsforcorrectstructureandvalidity(Scheueretal2010:8)
(Klein2011:5)
It is estimated that 1 moderator for every 20 contributors is required for sufficient
maintenanceoftheDeliberatorium(Klein2011).Inadditiontothemapthereisachatroom
areaforlessformalisedconversation.
Theplatformprovidesagoodexampleoftheargumentmappingapproach,basedontheIBIS
approach common in collective intelligence literature. It is supported by applications in
practicethatdemonstrateitsuseinlargescalediscussionsleadingtoadecision(discussedin
greaterdetailbelow).Thedevelopersdirectlydescribetheintentiontoaddresschallenges
relating to feasibility and scale, participant behaviour and citizen capacity as well as
informationmanagement. The platform is also a notable example of the combination of
synchronousandasynchronouscommunicationtosupportdeliberation(Delborneetal2011).
ImagesandScreenshotsofthePlatform
Thescreenshotsbelowprovideanillustrationofthevisualrepresentationandaestheticsof
the delibratorium. The Scholio project is currently working on the aesthetics of
Deliberatoirum.
Displayinghowuserscanintroduceanideaandvoteonit16
Displayingthegeneralargumentmap17
16http://franc2.mit.edu:8000/ci/show-top
17http://deliberatorium.mit.edu/
Displayingauserclickingonanindividualideainanargumentmaptogetfurtherdetail18
LiteratureandApplicationsofthePlatform
TheDeliberatoriumhasbeentestedandappliedinanumberofsettings.Initially“TheCarbon
Offsetting Thought Experiment” attempted to translate a 13pagediscussion into amuch
moresuccinct8itemdeliberationmap(Kleinetal2012).Thefirstmajorevaluationofthe
Deliberatoiruminvolved220mastersstudentsattheUniversityofNaplesinadebateabout
bio fuels. The experiment focused on whether the demands of structure would put off
participants, andwhethermoderatorswere able todealwith thedemandsof large scale
participation.Theresearchreported thatneitherof theseconcernswereproblematicand
theyobservedveryhighlevelsofuserparticipation(Klein2011).Thestudentscreatedamap
that was judged by content experts to represent a remarkably comprehensive and well
organisedreviewofthekeyissuesandoptionsaroundbiofueladoption(Klein2011).Further
evaluationsoftheDeliberatoriumhavetakenplacewithIntelCorporation,USBureauofLand
Management,theUniversityofZurichandHMCInc(Klein2011).TheDeliberatoriumhasbeen
usedbytheItalianDemocraticPartyinaninternalpartydebateoverelectoralreform.This
experimentinvolved400people,withtwogroupsof160participantsassignedtodiscussthe
topic through either the Deliberatorium or through a standard forum. It found that the
restrictedstructureofdiscussiondidnotaffectusers’retentionratenortheiraveragedaily
18http://deliberatorium.mit.edu/
activity,whiletheargumentmapreducesthequantityof ideaspostedbyusers, the ideas
postedtendedtobemoredevelopedintermsofsupportingarguments(Kleinetal2012).This
suggestscertainadvantagesanddisadvantagestothetoolthatmaydeterminewhere it is
best applied. Scheuer et al (2010) discuss Collaboratorium, an earlier version of
Deliberatorium,describinghowtheplatformsupportsthenotionofcollectiveintelligenceby
allowing participants to rate contributions, the highest rated being considered the
community’sdecisions.
Parmenides
BackgroundandObjectivesofthePlatform
Parmenidesisdescribedasan“e-participationforum...asystemfordeliberativedemocracy
that allows the government to present policy proposals to the public and lets the public
submittheiropiniononthepolicyandits justification”(Parmenides2017,Cartwrightetal
2009). In other literature Parmenides has been described as a platform that collects
argumentsforandagainstagivenproposal(Atkinsonetal2004).
Parmenides is informedby informal logic,specificallyamodificationof theargumentation
schemesofWaltonandtheBeliefDesireIntentionarchitecture(Atkinson2006).Thisguides
the platforms heavily structured interface design. The platform webpage explains that
“Parmenidesexploitstwomethodsofargumentrepresentation:Argumentationschemesto
structurepolicyproposalsandargumentationframeworkstodiagrammaticallyanalysethe
opinions submitted by users” (Parmenides 2017). It consists of fourmain components: a
debate creator (administrators can create a debate by instantiating elements of the
argumentationscheme);theParmenidesinterface(allowingpeopletoparticipateandsubmit
their opinions); administration tools (allowing argumentation schemes to be added); and
analysis tools (allowing information submitted to be analysed using argumentation
frameworksandvalue-basedargumentationframeworks)(Cartwrightetal2009).
Parmenidesdialoguestructureseesthejustificationforanactionasinvolvingthefollowing
argumentationscheme:anunderstandingof thecurrent situation;aviewof thesituation
whichwillresultfromtheperformanceoftheaction;featuresofthenewsituationwhichare
considereddesirable(theaspectswhichtheactionwasperformedinordertorealise);the
social goals which are promoted by these features (the reasons why they are desirable)
(Atkinsonetal2004).Fromthisscheme,aseriesofpotentialwaysof‘attacking’aproposal
areidentified.
(Atkinsonetal2004:314)
Attacks12,13,14and3(detailedinthetableabove)areneglectedonthegroundsthatthe
developersfeltthatanargumentproposedthroughtheirsystemcouldbepresumedtobe
sound and describe actions that were possible. The remaining attacks are used for the
structuringofaninterfacethatguidestheuserthroughajustificationforaproposal,giving
userstheopportunitytodisagreeatselectedpointsandcollectinginformationonwheremost
usersdisagreewithaproposal.Attacks7,8,9and11areusedinthestructureasthebasisto
allowuserstosubmitalternativeproposalsforaction(Atkinsonetal2004).Inadditiontothe
platform itself there are further tools to support information gathering with Parmenides
includingadebatecreator,profilinginformationofthosewhoparticipateandanalysistools
(Parmenides2017).
Parmenidesisuniqueamongstonlinedeliberationplatformsinitsapproachoftakingusers
through a heavily structured dialogue process. It represents an interesting application of
informal logic theory to the problem of online deliberation. Applications of the platform
howeverarelimitedandthecurrentpubliclyavailableversionoftheplatformisrestrictedto
fourspecificdebates.
ImagesandScreenshotsofthePlatform
ThedesignofParmenidesisillustratedbelowwiththeexampleofafoxhuntingdebate.
DisplayinghomepageoftheParmenidesplatform19
Displayingintroductiontospecifictopic,inwhichausergivestheirinitialresponse20
19http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/20http://cgi.csc.liv.ac.uk/~parmenides/
Displayinganexampleofthestructuredinterface,inwhichauserindicatestheircommitment
toparticularvalues,andisgiventheoptionofincludinganyadditionalvaluesrelevanttothe
topic21
21http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/
Parmenides(2017)Displaysasummaryofauser’sresponsestoaparticularargument22
LiteratureandApplicationsofthePlatform
AversionofParmenidesispubliclyavailableandallowsfortheexplorationoffourcurrent
debates, including fox hunting and speed cameras, and a record of one previous debate.
Although the literature discusses Parmenides use in the context of e-participation and e-
democracy(Atkinsonetal2004,Cartwrightetal2009),theapplicationofthetoolinthisarea
hasbeenlimitedandthefocusofitscurrentusehasmovedtowardsprivateapplicationin
the field of law. Atkinson et al (2004) argued that Parmenides was usable by its target
audienceandcanbeusedtoidentifypointsofdisagreement,andrecordthemsothatthe
weightofopiniononvariousissuescanbegauged.CartwrightandAtkinson(2008)document
22http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/
thedevelopmentoftoolsfore-democracy,includingallowingthesystemtocollectopinions
ondifferenttopics,analysedata,anddemographicprofilingofusers.Thepaperdescribesthe
intentiontoconductlargescalefieldteststovalidatetheeffectivenessofthesystem.
Pol.is
Background,ObjectivesandDesign
Pol.is“helpsorganisationsunderstandthemselvesbyvisualizingwhattheirmembersthink
(Pol.is2017). Itaims toallowthegatheringofopenended feedback from largegroupsof
people. Users click “agree”, “disagree” or “pass” in response to statements others have
contributed.Usersareabletosubmittheirowncomments;however,theyarenotallowedto
replydirectlytoacomment.Pol.isrunsstatisticalanalysisonthesevotingpatterns,surfacing
opiniongroups,comments thatbroughtgroupstogetherandcomments that foundbroad
consensus(Pol.is2017).Itthenprovidesavisualrepresentationofthesegroupsandclusters
ofopinion.Thedecisiontorepresentclustersofopinionratherthanplacingthoseopinionsin
thecontextofanargumentmapandnotallowingdirectresponsestomessagesdistinguishes
Pol.is from many other approaches to online deliberation and online communication in
general.
Pol.iswasconceivedduringthetimeoftheArabSpringandOccupyWallStreetmovement,
andsoughttodevelopacommentsystemthatcouldscaleupandretaincoherencewithlarge
groupsofpeople.ThedevelopersofPol.isclaimthatthedesignchoiceofnotallowingreplies
iskeytomakingitpossibletomakesenseoflargegroups;itisclaimedthatargumentsdonot
scale,andthemomentonebeginstotrackaconversationbetween individuals,andother
people’sresponsestospecificcomments,thenthatsenseofscalebreaksdown(Megill2016).
Thedevelopersfurtherstatetheaimsofensuringpeoplefeelsafe,listenedto,thatpeople
mayparticipateatanytimeinthelifecycleoftheconversation,thattheyhaveasenseof
what others felt and minority opinions are preserved and represented (Pol.is 2017). In
allowingstructuredresponses(agree,disagree,pass)andnodirectrepliesitisclaimedthat
problemsoftrollingandothernegativebehaviourareaddressed.Thevisualrepresentation
aims to ensure that participants can see all voices represented as well as any points of
consensus,particularlyamongstotherwisedisparategroups.Inthissense,itisclaimedthe
problemsofechochambersandfilterbubblesareaddressedthroughthesechoicesaround
visualrepresentation(Pol.is2017).InorderforPol.istobemoreeffectiveinrepresentingthe
different views of a collective group it encourages participants to respond rather than
submittingoriginalcomments.Oneofthewaysitdoesthisisbyusingafuzzysearchtoshow
usersothercommentsthatmaybesimilartotheonetheyaretyping.
Pol.isrepresentsaninterestingexampleoftheuseofAIinonlinedeliberation.Ithasbeen
successfullyappliedinanumberofcontexts,notablyindecisionmakinginTaiwan.Anumber
ofdesignchoicesmake itunusualamongstonlinedeliberationplatforms;notably the fact
thattheplatformdoesnotattempttodirectlysupportthecriticalworkofplacingopinions
withinthecontextofanargumentstructure,andthedecisionnottoallowdirectrepliesto
messages.Theplatformisnotableforitsuniquedesignanditssuccessfulapplications.
ImagesandScreenshotsofthePlatform
Thescreenshotbelowillustratesthevisualisationprovidedbypolisofcommentsandgroups.
Conversationinterface(Megill2017)
Representationofopiniongroups,highlightingthelargestgroup,groupB(Megill2017)
Metadataonthegroup,includinginformationonareasofuncertainty(Megil2017)
LiteratureandApplicationsofthePlatform
Pol.ishasbeenusedinanumberofsettings,notablyinTaiwan,wheretheprojectvTaiwan
usedPol.istoaddressparticularissuessuchasliquorsales,crowdfunding,Uber,andAirbnb
(Megill2016,Berman2017,Barry2016,Tang2016).Inthiscase,decisionmakingtookthe
formoffourstages.First,Pol.iswasdistributedthroughFacebookadsandnetworkstargeting
participants.Publicmeetingswerethenbroadcastwherescholarsandofficialsrespondedto
the comments emerging through Pol.is. This was followed by face to face stakeholder
meetingsbroadcasttootherparticipants.Inthefinalstage,Barry(2016)describeshowPol.is
wasusedinabindingway,withthegovernmentcommittingtoeithertransformconsensus
intonationallegislatureorprovideapointbypointexplanationofwhythisisnotpossible.
Theseapplicationshavebeenreportedtobesuccessful, forexample it isclaimedthatthe
issueofonlineliquorsaleshadbeenindeadlockforfiveorsixyears,butthroughvTaiwanand
theuseofPolisadecisionhadbeenreachedinthreetofivemonths.AsofFebruary2018,26
cases have been discussed through vTaiwan, 80% have led to decisive action from the
government(vTaiwan2018).Tang(2016)describeshowthecaseconcerningUberandTaxi
servicesresultedintheadministrationpledgingtoratifyallPol.isconsensusitemsintonew
regulation.Asanindicationofthesuccessoftheprocess,Taiwan’spremierisquotedassaying
“allsubstantialnationalissuesshouldgothroughavTaiwan-likeprocess”(Barry2016).
Rbutr
Background,ObjectivesandDesign
Rbutr“isacommunity-drivenappwhichconnectswebpagestogetheronthebasisthatone
pagearguesagainsttheother”(Rbutr2017). Itutilisescrowdsourcingto identifyrebuttals
andcriticalresponsestoagivenwebpageorarticle.Itwasinitiallydevelopedasaplug-in
alertinguserstorebuttalsofarguments.Thereisanadditionalframeoptiontoviewrebuttals,
andanaccompanyingwebsiteofferingfurtherfunctions.ThedevelopersofRbutrdescribeits
aimsasbeingtopromotecriticalthinkingforfuturegenerations,tacklingtheproblemoffilter
bubblesandensuringthat“misinformationiscorrected,scamsareexposed...andcontextis
providedtoaclaimthatallowsreadersaccesstothefullstory”(Rbutr2017).
UsersofRbutrcanuse theplug-in tosubmit rebuttals.Usersareable to linkonepageto
anotherpagethatcontainsarebuttalofthefirst.Usersarealsoabletoaddrelevanttags.
Rbutralsoutilisessocialmedia,forexampleitidentifieswhereanarticlehasbeenlinkedto
byTwitterusersandallowsforanautomaticresponsealertingthatusertorebuttalsofthat
article.Agivenarticle canhaveanumberof rebuttals linked to it, these rebuttals canbe
rankedbyusersinthehopethatpeoplewillbedirectedtowhatisconsideredtobethebest
exampleofarebuttalofagivenarticle.
TheRbutrwebsitestatesthattheprojectisrunbyasmallteamwithverylittlefunding.It
receivedseedfundingandsupportfromtheStartUpChileProgramme,thedevelopersmade
Rbutropensourceandnon-profitandarecurrentlyengagedinfundraisingwitheducational
organisations, fact checking organisations and publishing platforms aswell as looking for
volunteersanddonors(Rbutr2017).Rbutrisagoodexampleofanannotationapproachto
argument representation and online deliberation. It aims to address a specific set of
challenges inonlinedeliberation,thefilterbubbleandpolarisationofdebate,andaimsto
fostercriticalthinkingandengagementwithalternativeviewpoints.
ImagesandScreenshotsofthePlatform
ThescreenshotsbelowillustrateRbutr’sdesign.
DisplayinghowRbutrappearswhenvisitingawebsite23
23https://www.youtube.com/watch?v=V9qi3vaLKyU
Interface
foraddingrebuttals24
DisplayingRbutr’swebsite,displayingtherebuttedarticle,2rebuttals,anddetailsoftweets
sharingtherebuttedarticle25
LiteratureandApplicationsofthePlatform
There is currently no literature on the use of Rbutr, however a study into its use in an
educational context is being undertaken currently (Rbutr 2017b). The developer, Shane
24http://www.rbutr.com25http://www.rbutr.com
Greenup,describeshowRbutrhasreachedapeakof20,000usersandhasbeensurprisingly
robust. It has not experienced any problems with spam and every rebuttal has been
appropriate.
Truthmapping
Background,ObjectivesandDesignofthePlatform
TruthMappingisawebsitethatallowspeopletoconstructargumentmaps.Thedevelopers
describethedesignofTruthmappingasanattempttoresolvetheproblemof“noise”,which
isassociatedwithforumsorganisedbytime.Theproblemof“noise”isattributedtoanumber
offactors:Digressionfromthetopic,the“soapboxproblem”andparticipantsspeakingpast
oneanother.The“soapboxproblem”,itisargued,isaresultofparticipantsbeingincentivised
tobethelastpersontotalkorthemostvocal.Itisarguedthatthisisaprobleminforums
organisedbytimewhereapersonismostlikelytobeheardiftheircommentislistedfirstor
iftheymakethecommentrepeatedly.Similarly,whenparticipantsdigressorspeakpastone
anothertheyproduceinformationthatisredundantanddetractsfromtheinformationmore
relevant to the discussion (Truthmapping 2017a). Truthmapping is an attempt to resolve
theseissuesthroughtheuseofargumentvisualisation.Thisapproachtoargumentmapping
wascreatedbythedevelopersofTruthmappingandisnotgroundedinaspecifictheoretical
accountofargument.Theontologyoftheargumentmapsconsistsofconclusionssupported
by premises; these premises can then be critiqued and those critiques rebutted. This is
illustratedbelow.
Truthmapping2017
Onlyonerebuttalcanbeaddedtoacritique.Theseelementscanbeedited;thefinaldraft
being visible to users,while the previous drafts are archived. This process is intended to
ensurethatthebestexamplesofthecritiquesandrebuttalsarepreserved.Itisclaimedthis
mechanism removes the incentive of the “soapbox problem” and discourages digression
(Truthmapping2017a).Truthmapping isagoodexampleofargumentmappingtechniques
beingusedforgeneralpublicdiscussionandpoliticaldebate,theattempttolimitrebuttals
throughaneditingprocessalsorepresentsaninterestingapproachtotheissueofinformation
managementthatisnotfoundinmostotherargumentmappingtools.
ImagesandScreenshotsofthePlatform
ThescreenshotsbelowillustratethedesignandaestheticsofTruthmapping.
Displayinggeneralviewofargumentmap26
26https://www.truthmapping.com/map/806/#s5361
Displayingauserclickingonindividualargumentswithinthemaptoviewfurtherdetails27
LiteratureandApplicationsofthePlatform
AliteraturesearchfoundnodetailsonapplicationsorstudiesofTruthmapping,beyondthe
material available on the Truthmapping site. There is a publicly available version of
Truthmapping which limits the number of maps one can publish, in addition there are
subscriptionplans forsmallgroupsand largergroupsaimedatnon-profitandeducational
organisationswithadditionalfeatures(Truthmapping2017b).Thepubliclyavailableversion
ofTruthmappinghasbeenrunningforover10yearsandcontinuestohaveregularvisitors
andpublishedmaps.TodatethemostpopularcategoriesoftopicconcernPhilosophy(89
maps),Politics(73maps)andSocialIssues(59maps).
Conclusion
What is clear from this review is that there is a diversity of novel designs for the
representationofideasandargumentsandthepromotionofelementsofdeliberationonline.
Thecloseanalysisoftwelveofthemostwellrespecteddesignscurrentlyinoperationgives
usasenseofthebreadthofthefieldandthediversityofapproachestosupportingaspects
of deliberation. However, it is hard to capture the full range of activity because of
methodological challenges in defining the population and its characteristics.We are also
victims of the speed of change in the field: new designs emerge rapidly to improve the
27https://www.truthmapping.com/map/806/#t5363-c5484
interface,aestheticsandargumentrepresentation.Practiceismovingfasterthanourcapacity
toconceptualiseandcompare.
BibliographyAtkinson,K,BenchCapon,T,McBurney,P(2004)“Parmenides:FacilitatingDemocraticDebate”inInternationalconferenceonElectronicGovernment313-316Atkinson,K(2006)Whatshouldwedo?:Computationalrepresentationofpersuasiveargumentinpracticalreasoning,Ph.D.thesis,Dept.ofComp.Sci.,Univ.ofLiverpoolBarry,L(2016)VTaiwan:PublicParticipationMethodsOntheCyberpunkFrontierofDemocracy[ONLINE]Availableat;https://civichall.org/civicist/vtaiwan-democracy-frontier/[Accessed9/7/2017)
BCisiveOnline(2017)BCisiveOnline[ONLINE]Availableat:https://www.bcisiveonline.com[Accessed9/7/2017]BermanP(2017)“Hackingideology:pol.isandvTaiwan”inDemocracyEarth[ONLINE]Availableat:https://words.democracy.earth/hacking-ideology-pol-is-and-vtaiwan-570d36442ee5[Accessed:2.12.17]Black,LW.(2011)“Thepromiseandproblemsofonlinedeliberation”inCharlesF.KetteringFoundation,IncBullen,V,Price,D(2015)“Debategraph:ANewwaytoaddressthecomplexityofobesity”inBritishJournalofObseity1:54-59Cartwright,DandAtkinson,K(2008)“PoliticalEngagementthroughToolsforArgumentation”inCOMMAconferenceproceedingsCartwright,D,Atkinson,K,Bench-Capon,T(2009)“SupportingArgumentinE-Democracy”inElectronicGovernment:thirdinternationalelectronicdemocracy(EDEM2009)(Vienna,Austria)151-160Coleman,SandGoetze,J(2001)BowlingTogether:Onlinepublicengagementinpolicydeliberation,report,London:HansardSocietyColeman,S.andMoss,G.(2012)‘Underconstruction:Thefieldofonlinedeliberationresearch’inJournalofInformationTechnology&Politics,9(1),1–15Conklin,J(2008)“GrowingaGlobalIssueBase:AnIssue-BasedApproachtoPolicyDeliberation”inToolsforParticipation:Collaboration,DeliberationandDecisionSupport,Proceedings,26–29June2008
ConsiderIt(2017)ConsiderIt[ONLINE]Availableat:https://consider.it[Accessed9/7/2017]
Crossley-Frolick,K(2017)“Visualising“wickedproblems”byusingdebategraphandDialogueMapping”[ONLINE]Availableat:http://glcateachlearn.org/2017/04/10/visualizing-wicked-problems-by-using-debategraph-and-dialogue-mapping-katy-crossley-frolick-denison-university/[Accessed09/7/2017]
Davies,T.(2009)“TheblossomingfieldofOnlineDeliberation”inTDaviesandSPGangadharan(eds)OnlineDeliberation:Design,ResearchandPractice,SanRancisco,CA:CSLIPublication
Davies,W.M.,(2009b)“Computer-AssistedArgumentMapping:ARationaleApproach”2009.HigherEducationDavies,TandChandlerR(2011)“OnlineDeliberationdesign:choices,criteriaandevidence”inTNabatachi,MWeiksner,JGastilandMLeighninger(eds)DemocracyinMotion:EvaluatingthePracticeandImpactofDeliberativeCivicEngagement,Oxford:OxfordUniversityPressDebategraph(2017)Debategraph[ONLINE]Availableat:http://debategraph.org/Stream.aspx?nid=65027&vt=bubble&dc=focus[Accessed9/7/2017]Delborne,JA,Anderson,AA,Kleinman,DL,Colin,M,Powell,M(2011)“VirtualDeliberaton?ProspectsandChallengesforintegratingtheInternetinConsensusConferences”inPublicUnderstandingofScience20(3)367-384Deliberatorium(2017)Deliberatorium[ONLINE]Availableat:http://cci.mit.edu/klein/deliberatorium.html[Accessedon9/7/2017]DeLiddo,AandBuckinghamShum,S(2010a)“CapturingandRepresentingDeliberationinParticipatoryPlanningPractices”inFDeCincio,AMachintoshandCPeraboni(eds)Onlinedeliberation,proceedings,FourthInternationalConference,Leeds,UKDeLiddo,A.,BuckinghamShum,S.(2010b)“Cohere:Aprototypeforcontestedcollectiveintelligence”InACMComputerSupportedCooperativeWorkConference(CSCW2010):WorkshoponCollectiveIntelligenceinOrganizations,Feb6--10,2010DeLiddo,AandBuckinghamShum,S(2013).“TheEvidenceHub:harnessingthecollectiveintelligenceofcommunitiestobuildevidence-basedknowledge”inLargeScaleIdeationandDeliberationWorkshop,29June–02July2013,Munich,GermanyDriel,S.van,&Prakken,H,(2010)“VisualisingtheargumentationstructureofanexpertwitnessreportwithRationale”InA.Z.Wyner(ed.),ProceedingsoftheWorkshoponModellingLegalCasesandLegalRules,inconjunctionwithJURIX-10,Liverpool2010,pp.1-8
Duhaime,E,Olson,GThomasW.Malone,T(2015)“BroadParticipationinOnlineProblemSolvingCanInfluenceParticipantsandLeadtoBetterSolutions:EvidencefromtheMITClimateCoLab”inMITCCIWorkingPaperNo.2015-001,June2015.eLab(2017a)“ParticipatoryBudgetingWhitePaper”inEngagementLabatEmersonCollege[ONLINE]Availableat:elabhome.blob.core.windows.net/resources/stake-pb-white-paper.docx[Accessed09/07/2017]eLab(2017b)“@Stake:GameMaterialsandDirections”inEngagementLabatEmersonCollege[ONLINE]Availableat:http://elabhome.blob.core.windows.net/resources/stake-rules-and-role-cards.pdf[Accessed30/07/17]EngagementLab(2017)EngagementLab[ONLINE]Availableat:https://elab.emerson.edu/projects/participation-and-engagement/atstake[Accessed9/7/2017]EvidenceHub(2017)EvidenceHub[ONLINE]Availableat:http://evidence-hub.net/about/[Accessed9/7/2017]
Fishkin,J(1995)“TheVoiceofthePeople:PublicOpinionandDemocracy”NewHaven:YaleUniversityPressFreelon,DG,Kriplean,T,Morgan,J,Bennett,W,Borning,A(2011)“Facilitatingencounterswithpoliticaldifference:Engagingvoterswiththelivingvotersguide”inProceedingsoftheJournalofInformationTechnologyandPoliticsAnnualConference,2011Freelon,DG,Kriplean,T,Morgan,J,Bennett,WL,Borning,A(2012)“Facilitatingdiversepoliticalengagementwiththelivingvotersguide”inJournalofinformationTechnologyandPolitics9(3)279-297Friess,DandEilders,C(2014)“ASystematicReviewofOnlineDeliberationResearch”inPolicyandInternet7(3)Gelder,T.van,(2007)“TherationaleforRationale”inLaw,ProbabilityandRisk,(2007)6,23−42
Gordon,E.,Michelson,B.,&Haas,J.(2016).“@Stake:AGametoFacilitatetheProcessofDeliberativeDemocracy”InProceedingsofthe19thACMConferenceonComputerSupportedCooperativeWorkandSocialComputingCompanion(pp.269-272).ACM
Iandoli,L,Klein,M,Zollo,G(2008)“Canweexploitcollectiveintelligenceforcollaborativedeliberation?Thecaseoftheclimatechangecollaboratoirum”inMITSloanResearchPaperNo.4675-08Introne,J,Laubacher,R,Olson,G,andMalone,T,(2011)“TheClimateCoLab:LargeScaleModel-BasedCollaborativePlanning”InInternationalConferenceonCollaborationTechnologiesandSystems2011
JanssenD,KiesR.(2005).“Onlineforumsanddeliberativedemocracy”inActaPolit.40:317-35Karlsson,M(2010)“Whatdoesittaketomakeonlinedeliberationhappen?Acomparativeanalysisof28onlinediscussionforums”inFDeCincio,AMachintoshandCPeraboni(eds)Onlinedeliberation,proceedings,FourthInternationalConference,Leeds,UKKlein,MandIandoli,L(2008)“SupportingCollaborativeDeliberationUsingaLargeScaleArgumentationSystem:TheMITCollaboratorium”inToolsforParticipation:Collaboration,DeliberationandDecisionSupport,Proceedings,26–29June2008Klein,M(2011)“HowtoHarvestCollectiveWisdomonComplexProblems:AnintroductiontotheMITDeliberatorium”inCCIworkingpaper2011Klein,M,Spada,P,Calabretta,R(2012)“Enablingdeliberationsinapoliticalpartyusinglargescaleargumentation:Apreliminaryreport”in10thInternationalConferenceontheDesignofCooperativeSystemsfromresearchtopractice:Resultsandopenchallenges,atMarseille,May29,2012Klein,M(2015)“ACriticalReviewofCrowd-ScaleOnlineDeliberationTechnologies”inMITSloanResearchPaperNo.5143-15Klein,M(2017)“CollectiveIntelligenceWorkingNotes”,unpublishedKriplean,T,Morgan,JT,Freelon,D,Borning,A,Bennett,L(2011)“ConsiderIt:Improvingstructureddeliberation”inCH’11ExtendedabstractsonHumanFactorsinComputingSystems,1831-1836Kriplean,T,Morgan,J,Freelon,D,Borning,A,Bennett,L(2012)“SupportingReflectivePublicThoughtwithConsiderIt”inCSCW12ProceedingsoftheACM2012ConferenceonComputerSupportedCooperativeWork265-274Kriplean,T,Bonnar,C,Borning,A,Kinney,B,Gill,B(2014)“Integratingon-demandfact-checkwithpublicdialogue”inProceedingsofthe17thACMconferenceonComputerKunsch,D,W,Schnarr,K,vanTyle,R(2014)TheUseofArgumentMappingtoEnhanceCriticalThinkingSkillsinBusinessEducationinJournalofEducationforBusiness89403-410Lawrence,JBex,FandReed.,C(2012)“DialoguesontheArgumentWeb:MixedinitiativeargumentationwithArvina”InB.Verheij,S.Szeider,andS.Woltran,editors,ProceedingsoftheFourthInternationalConferenceonComputationalModelsofArgument(COMMA2012),pages513–514,Vienna,Austria,2012.IOSPress.Lawrence,J,Janier,MandReed.C,(2015)“WorkingwithOpenArgumentCorpora”InEuropeanConferenceonArgumentation(ECA),2015
Lengbyer,L.,(2014)“CriticalThinkingintheIntelligenceCommunity:ThePromiseofArgumentMapping”inInquiry:CriticalThinkingacrossthedisciplines.Summer2014,Vol.29,No.2Loukis,EandWimmer,M(2010)“AnalysingDifferentModelsofStructuredElectronicConsultationonLegislationunderFormation”inFDeCincio,AMachintoshandCPeraboni(eds)Onlinedeliberation,proceedings,FourthInternationalConference,Leeds,UKMalone,T,Laubacher,R,Introne,J,Klein,M,Ableson,H,Sterman,J,Olson,G(2009)”TheClimateCollaboratoirum:ProjectOverview”inMITCentreforCollectiveIntelligenceWorkingPaperNo.2009-03Malone,T,Nickerson,J,Laubacher,R,HesseFisher,L,DeBoer,P,Han,Y,Towne,WB(2017)“PuttingthePiecesBackTogetherAgain:ContestWebsforLargeScaleProblemSolving”inProceedingsoftheACMConferenceonComputerSupportedCoopertativeWorkandSocialComputing,PortlandORFebruary25March1,2017Manosevitch,I(2014)“TheDesignofOnlineDeliberation:ImplicationsforPracticeTheoryandDemocraticCitizenship”inJournalofPublicDeliberation10(1)
MegillC(2016)Pol.isinTaiwan[Online]Availableat:ttps://blog.pol.is/pol-is-in-taiwan-da7570d372b5#.fq6w6zf4p[Accessed9/7/2017]
MegillC(2017)Pol.isCaseStudy[Online]Availableat:https://blog.pol.is/pol-is-case-study-temperature-check-a02dff7cc838[Accessed9/8/2017]
Michelson,B(2015)“@Stake:ASeriesofFieldNotesinGameDevelopment”inEngagementLabatEmersonCollege[ONLINE]Availableat:https://medium.com/engagement-lab-emerson-college/stake-a-series-of-field-notes-in-game-development-570078fe2e74[Accessed09/07/2017]Naurin,D(2007)“WhyGiveReason?MeasuringArguingandBargaininginSurveyResearch”inSwissPoliticalScienceReview13(4):559–75
Neblo,MA.(2007).Familydisputes:diversityindefiningandmeasuringdeliberation.SwissPoliticalScienceReview13:527–57
Parmenides(2017)Parmenides[ONLINE]Availableat:http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/[Accessedon9/7/2017]ParticipateDB(2017)PartitipateDB[ONLINE]Availableat:http://www.participatedb.com/categorizations[Accessed9/7/2017]Pingree,RJ(2009)“DecisionStructure:ANewApproachtoThreeProblemsinDeliberation”inT.DaviesandS.P.Gangadharan(Eds.),Onlinedeliberation:Design,research,andpractice(pp.309-316).Stanford,CA:CSLIPublications
Pol.is(2017)Pol.isOverview[ONLINE]Availableat:https://docs.pol.is/welcome/Overview.htmlRationaleOnline(2017)RationaleOnline[ONLINE]Availableat:https://www.rationaleonline.com[Accessed9/7/2017]Rbutr(2017)AboutUs[ONLINE]Availableat:http://blog.rbutr.com/about-us/[Accessed9/7/2017]Rbutr(2017b)AcademicResearchandrbutr–AnnouncingacollaborationwiththeDelftUniversityofTechnology[Online]Availableat:http://blog.rbutr.com/2017/05/academic-research-and-rbutr-announcing-a-collaboration-with-the-delft-university-of-technology/[Accessed9/7/2017]ReasoningLab(2017)ReasoningLab[ONLINE]Availableat:http://www.reasoninglab.com[Accessed9/7/2017]
Reed,C(2017)“AgreementbetweenBBCandARG-Tech”inARG-tech:centreforArgumentTechnology[ONLINE]Availableat:http://www.arg-tech.org/index.php/agreement-between-bbc-and-arg-tech/[Accessed09/10/17]
Reiter,B.(2013).“Theepistemologyandmethodologyofexploratorysocialscienceresearch:CrossingPopperwithMarcuse”inThedialecticsofcitizenship:exploringprivilege,exclusionandracialization(pp.1-22).EastLansing,MI:MichiganStateUniversityPress
Scheuer,O.,Loll,F.,Pinkwart,N.,&McLaren,B.M.(2010).Computer-supportedargumentation:Areviewofthestateoftheart.InternationalJournalofComputer-SupportedCollaborativeArgumentation,5(1),43-102.
Schutt,R.K(2015)InvesitgatingtheSocialWorld:TheProcessandPracticeofResearch8thEditionSingapore:SagePublications
Shane,P(2004)“IntroductiontheProspectsforElectronicDemocracy”inDemocracyOnline:ProspectsforPoliticalRenewalThroughtheInternet(ed)Shane,PNewYork:Routledge
Shields,PandRangarajan,N(2013)APlaybookforResearchMethods:IntegratingConceptualFrameworksandProjectManagementStillwater,OK:NewForumsPress
Snaith,M.;Devereux,J.;Lawrence,J.;andReed,C.(2010)“Pipeliningargumentationtechnologies”inCOMMA2010,447–453
Steigler,H,DeJong,M(2015)“Facilitatingpersonaldeliberationonline:Effectsoftwoconsider.itvariations”inComputersinHumanBehaviour51(A)461-469
Sunstein,C(2001)Republic.com,Princeton,NJPrincetonUniversityPress
TambourisE.,DalakiouridouE.,PanopoulouE.,andTarabanisK.(2011)EvaluationofanArgumentVisualisationPlatformbyExpertsandPolicyMakers.InE.Tambouris,A.MacintoshandH.deBruijn(Eds.):ePart2011,LNCS6847,pp.73–84
Tang,A(2016)“DigitalToolsOpenUpTaiwan’sDemocraticImaginations”inpol.isblog[ONLINE]Availableat:https://blog.pol.is/digital-tools-open-up-taiwans-democratic-imaginations-d8f80432305c[Accessed9/7/2017]Truthmapping(2017a)Truthmapping[ONLINE]https://www.truthmapping.com/#[Accessed9/7/2017]Truthmapping(2017b)TruthmappingPlans[ONLINE]https://www.truthmapping.com/plans/[Accessed9/7/2017]Twardy,C(2004)“ArgumentMapsImproveCriticalThinking”,2004.TeachingPhilosophy,27:2.Tauro,C,Ahuja,M,Perez-Quinones,A,Kavanaugh,AandIsenhour,P(2008)“VizBlog:AVisualizationToolforBlogDiscover”inToolsforParticipation:Collaboration,DeliberationandDecisionSupport,Proceedings,26–29June2008Towne,WBandHerbslebJD(2012)“Designconsiderationsforonlinedeliberationsystems”inJournalofInformationTechnologyandPolitics9:97-115vTaiwan(2017)“Afewaccomplishments”invTaiwan[ONLINE]Availableat:http://info.vtaiwan.tw/#three[Accessed09/07/2017]
Yin,RK(1994)CaseStudyResearch:DesignandmethodsSecondEditionThousandOaks:SagePublications