I N S T I T U T E

Methodology

Copyright © University of Gothenburg, V-Dem Institute,

University of Notre Dame, Kellogg Institute. All rights reserved.

Version 7 - May 2017

1

Suggestedcitation:Coppedge, Michael, John Gerring, Staffan I. Lindberg, Svend-Erik Skaaning, Jan Teorell, JoshuaKrusell,KyleL.Marquardt,ValeriyaMechkova,DanielPemstein,JosefinePernes,LauraSaxer,NataliaStepanova,EitanTzelgov,Yi-tingWangandStevenWilson.2017.“V-DemMethodologyv7.”VarietiesofDemocracy(V-Dem)Project.

Authors Collaborators• MichaelCoppedge–U.ofNotreDame • DavidAltman–PontificiaU.CatólicadeChile• JohnGerring–BostonUniversity • MichaelBernhard–UniversityofFlorida• StaffanI.Lindberg–U.ofGothenburg • M.StevenFish–UCBerkeley• Svend-ErikSkaaning–AarhusUniversity • AdamGlynn–EmoryUniversity• JanTeorell–LundUniversity • AllenHicken–UniversityofMichigan

• JoshuaKrusell–U.ofGothenburg• KyleL.Marquardt–U.ofGothenburg• ValeriyaMechkova–U.ofGothenburg• DanielPemstein–NorthDakotaStateU.• JosefinePernes–U.ofGothenburg• LauraSaxer-U.ofGothenburg• NataliaStepanova–U.ofGothenburg• EitanTzelgov–U.ofEastAnglia• Yi-tingWang–NationalChengKungU• StevenWilson–UniversityofGothenburg

• CarlHenrikKnutsen–UniversityofOslo• KellyMcMann–CaseWesternReserve

PamelaPaxton–U.ofTexas• BrigitteSeim–U.ofNorthCarolina• JeffreyStaton–EmoryUniversity

2

TableofContents

1. CONCEPTUALSCHEME 4PRINCIPLES–MEASUREDBYV-DEM’SDEMOCRACYINDICES 4COMPONENTS 10INDICATORS 11SUMMARY 12

2. DATACOLLECTION 14

HISTORYOFPOLITIES 14CODINGTYPES 17EXPERTRECRUITMENT 18EXPERTCODINGPROCESS 21BRIDGE-ANDLATERALCODING 25PHASES 26

3. MEASUREMENT 27

THEQUESTIONNAIRE 27IDENTIFYING,CORRECTING,ANDQUANTIFYINGMEASUREMENTERROR 28

MeasurementModels 28CorrectingErrors 35

VERSIONSOFC-VARIABLES 37ADDITIONALPOSSIBILITIESFORIDENTIFYINGSOURCESOFMEASUREMENTERRORINTHEFUTURE 40

REFERENCES 43

APPENDIXA:V-DEMINDICES,COMPONENTS,ANDINDICATORS 47

3

Thisdocumentoutlinesthemethodologicalconsiderations,choices,andproceduresguiding

the development of the Varieties of Democracy (V-Dem) project. Part I sets forth the

conceptual scheme.Part IIdiscusses theprocessofdatacollection.Part IIIdescribes the

measurementmodelalongwitheffortstoidentifyandcorrecterrors.

Wecontinuallyreviewourmethodology—andoccasionallyadjustit—withthegoal

ofimprovingthequalityofV-Demindicatorsandindices.Wethereforeissueanewversion

ofthisdocumentwitheachnewversionofthedataset.

Additional project documents complement this one.V-Dem Codebook includes a

comprehensive list of indicators, response-categories, sources, and brief information

regarding theconstructionof indices.V-DemCountryCodingUnits explainshowcountry

unitsaredefinedandlistseachcountryincludedinthedataset,withnotespertainingtothe

yearscoveredandspecialcircumstancesthatmayapply.V-Dem:ComparisonsandContrastssurveysthefieldofdemocracyindicatorsandsituatestheV-Demprojectinrelationtothese

efforts.V-DemOrganizationandManagement introducestheprojectteam,thewebsite,

outreachtotheinternationalcommunity,funding,progresstodate,andsustainability.

Versioningofthecurrentdocument,V-DemCodebook,V-DemCountryCodingUnits

andV-DemOrganization&Managementdocumentsaresynchronizedwiththereleaseof

eachnewdataset,whileV-Dem:ComparisonsandContrastsisnot.

Severalconfigurationsof theV-Demdatasetareavailable, includingcountry-year,

country-date,andcoder-leveldatasets.Foradditionaldocumentationandguidance,users

shouldrefertotheHowtoCitefilethatisappendedtoeachdatadownload.

IntheV-DemWorkingPaperSeries,userscanfindamoretechnicaldiscussionofthe

measurementmodelweusetoaggregatecoder-leveldatatopointestimatesforcountry-

years(Pemsteinetal.2015,WP#21,seealsoWP#41onIRTmodels),whileworkingPaper

#6 introduces the democracy indices, their components, indicators, and rules for

aggregation.WorkingPaper#25discussesthedetailsoftheElectoralDemocracy/Polyarchy

index.WorkingPaper#22describestheindexofEgalitarianDemocracy.Additionalworking

papers provide in-depth treatments of more specialized indices such as the Female

Empowerment Index (#19), the Core Civil Society Index (#13), Measuring Subnational

Democracy (#26), theCorruption Index (#23), andordinal versionsof theV-Dem indices

(#20).TheV-DemWorkingPaperSeriesisavailablefordownloadontheV-Demwebsite(v-

4

dem.net).

V-Dem isamassive,global collaborativeeffort.Anup-to-date listingofourmany

collaborators,withoutwhomthisprojectwouldnotbepossible,isalsoavailableontheweb

site.Collaborators includeProgramManagers,RegionalManagers, InternationalAdvisory

Boardmembers,theV-DemInstitutestaff(Director,Program-,Operations-,DataProcessing

and Data Managers, Assistant Researchers, and Post-Doctoral Fellows and Associate

Researchers), Research Assistants, and Country Coordinators. We are also especially

indebted to some 2,800 Country Experts, whose identitiesmust remain anonymous for

ethicalreasons.

The website serves as the repository for other information about the project,

includingCountryandThematicReports,BriefingPapers,publications,grantandfellowship

opportunities, and the data itself. Data for all 173 countries included in the first public

release (V-Dem Dataset v5) is also available for exploration with online analysis tools

(countryandvariablegraphs,motioncharts,and–soon–globalmaps).

1. ConceptualScheme

Anymeasurementschemerestsonconcepts. Inthissection,wesetforththeconceptual

schemethatinformstheV-Demproject–beginningwith“democracy”andproceedingto

thepropertiesandsub-propertiesofthatfar-flungconcept.Bywayofconclusion,weissue

severalclarificationsandcaveatsconcerningtheconceptualscheme.V-Dem:Comparisons

andContrastsprovidesamoredetaileddiscussion,butwerecaptheessentialpointshere.

Principles–MeasuredbyV-Dem’sDemocracyIndices

Thereisnoconsensusonwhatdemocracywrit-largemeansbeyondavaguenotionofrule

bythepeople.Politicaltheoristshaveemphasizedthispointforsometime,andempiricists

woulddowelltotakethelessontoheart(Gallie1956;Held2006;Shapiro2003:10–34).At

thesametime,interpretationsofdemocracydonothaveanunlimitedscope.

A thorough search of the literature on this protean concept reveals seven key

principlesthatinformmuchofourthinkingaboutdemocracy:electoral,liberal,majoritarian,

consensual,participatory,deliberative,andegalitarian.Eachoftheseprinciplesrepresents

5

adifferentwayofunderstanding“rulebythepeople.”Theheartofthedifferencesbetween

theseprinciplesisinthefactthatalternateschoolsofthoughtprioritizedifferentdemocratic

values.Thus,whilenosingleprincipleembodiesallthemeaningsofdemocracy,theseseven

principles, taken together, offer a fairly comprehensive accounting of the concept as

employedtoday.1

TheV-Demprojecthassetouttomeasuretheseprinciples,andthecorevalueswhich

underliethem.Wesummarizetheprinciplesbelow.

• Theelectoralprincipleofdemocracyembodiesthecorevalueofmaking

rulers responsive to citizens through periodic elections, as captured by

Dahl’s (1971, 1989) conceptualization of “polyarchy.” Our measure for

electoraldemocracyiscalledthe“V-DemElectoralDemocracyIndex.”We

considerthismeasurefundamentaltoallothermeasuresofdemocracy:

wewouldnotcallaregimewithoutelections“democratic”inanysense.

• The liberal principle of democracy embodies the intrinsic value of

protectingindividualandminorityrightsagainstapotential“tyrannyofthe

majority” and state repression. This principle is achieved through

constitutionally-protectedcivil liberties,strongruleof law,andeffective

checksandbalancesthatlimittheuseofexecutivepower.

• Theparticipatoryprincipleembodiesthevaluesofdirectruleandactive

participation by citizens in all political processes.While participation in

elections counts toward this principle, it also emphasizes nonelectoral

formsofpoliticalparticipation,suchascivilsocietyorganizationsandother

formsofbothnonelectoralandelectoralmechanismsofdirectdemocracy.

• Thedeliberativeprincipleenshrinesthecorevaluethatpoliticaldecisions

inpursuitofthepublicgoodshouldbeinformedbyaprocesscharacterized

1Thisconsensusonlyholds insofarasmostscholarswouldagreethatsomepermutationoraggregationoftheseprinciplesunderlieconceptionsofdemocracy.Forexample,scholarscanreasonablyarguethatthelistcould consist of seven, six, or five principles; our “principles”may be “properties” or “dimensions;” and“majoritarian”and“consensual”areactuallyoppositepolesofasingledimension.Asaresult,weintendforthisdiscussiontoassureconsumersofthedataofthecomprehensivenatureofourinventoryofcorevaluesofdemocracy:namely,thatitincludesalmostalltheattributesthatanyuserwouldwanttohavemeasured.

6

by respectful and reason-based dialogue at all levels, rather than by

emotionalappeals,solidaryattachments,parochialinterests,orcoercion.

• Theegalitarianprincipleholdsthatmaterialand immaterial inequalities

inhibit the actual use of formal political (electoral) rights and liberties.

Ideally,allgroupsshouldenjoyequaldejureanddefactocapabilitiesto

participate;toserve inpositionsofpoliticalpower;toput issuesonthe

agenda; and to influence policymaking. Following the literature in this

tradition, gross inequalities of health, education, or income are

understood to inhibit the exercise of political power and the de facto

enjoymentofpoliticalrights.

Theconceptualschemepresentedabovedoesnotcaptureallthetheoreticaldistinctionsat

playinthecomplexconceptofdemocracy.Wehavechosentofocusonthecorevaluesand

institutionsthattheotherprinciplesemphasizeintheircritiqueoftheelectoralconception

asastand-alonesystem.Eachoftheseprinciplesislogicallydistinctand—atleastforsome

theorists—independently valuable. Moreover, we suspect that there is a considerable

divergenceintherealizationofthepropertiesassociatedwiththesesevenprinciplesamong

the world’s polities. Some countries will be particularly strong on electoral democracy;

otherswillbestrongontheegalitarianproperty,andsoforth.

AggregationProcedures

Atthispoint,V-Demoffersseparateindicesoffivevarietiesofdemocracy:electoral,liberal,

participatory, deliberative, and egalitarian. We anticipate providing indices for the

remaining two principles – majoritarian and consensual – in the near future.2 V-Dem

CodebookcontainstheaggregationrulesforeachindexandseveralV-DemWorkingPapers

(presentandforthcoming) layoutjustificationsforthechoicesmadeineachaggregation

scheme.Thehigh-levelindices,measuringcoreprinciplesofdemocracy,arereferredtoas

democracyindices.

2 Themajoritarian principle of democracy (reflecting the belief that a majority of the people must becapacitatedtoruleandimplementtheirwillintermsofpolicy);andtheconsensualprincipleofdemocracy(emphasizingthatamajoritymustnotdisregardpoliticalminoritiesandthatthereisaninherentvalueintherepresentationofgroupswithdivergentinterestsandview).

7

Sartori held that every defining attribute is necessary for the concept. This logic

requiresmultiplyingtheattributessothateachofthemaffectstheindexonlytothedegree

that theothersarepresent. Family resemblancedefinitionsallowsubstitutability: ahigh

valueononeattributecancompensateforalowvalueonanother.Thislogiccorresponds

toanadditiveaggregationformula.Therearesoundjustificationsfortreatingallofthese

attributesasnecessary,ormutuallyreinforcing.Forexample,ifoppositioncandidatesare

notallowedtorunforelectionortheelectionsarefraudulent,thefactthatalladultshave

votingrightsdoesnotmattermuchforthelevelofelectoraldemocracy.Buttherearealso

goodreasonstoregardtheseattributesassubstitutable.Wherethesuffrageisrestricted,

the situation is less undemocratic if the disenfranchised are still free to participate in

associations, tostrikeandprotest,andtoaccess independentmedia (Switzerlandbefore

1971)thaniftheylacktheseopportunities(ItalyunderMussolini).Evenwheretheexecutive

isnotelected,citizenscanfeelthattheyliveinafairlydemocraticenvironmentaslongas

theyarefreetoorganizeandexpressthemselves,asinLiechtensteinbefore2003.

Becausewebelieveboththenecessaryconditionsandfamilyresemblancelogicsare

validforconceptsofdemocracy,ouraggregationformulasincludeboth;becausewehave

nostrongreasontoprefertheadditivetermstothemultiplicativeterm,wegivethemequal

weight.Theindexisformedbytakingtheaverageof,ontheonehand,theweightedaverage

oftheindicesmeasuringfreedomofassociation(thick)(v2x_frassoc_thick),cleanelections

(v2xel_frefair), freedom of expression (v2x_freexp_thick), elected executive (v2x_accex),

andsuffrage(v2x_suffr)and,ontheother,thefive-waymultiplicativeinteractionbetween

thoseindices.Thisishalfwaybetweenastraightaverageandstrictmultiplication,meaning

the average of the two. It is thus a compromise between the two most well-known

aggregationformulasintheliterature,bothallowing(partial)"compensation"inonesub-

componentforlackofpolyarchyintheothers,butalsopunishingcountriesnotstronginone

sub-componentaccordingtothe"weakestlink"argument.Theaggregationisdoneatthe

levelofDahl’ssub-components(withtheoneexceptionofthenon-electoralcomponent).

Theindexisaggregatedusingthisformula:

v2x_polyarchy=.5MPI+0.5API

=.5(v2x_accex*v2xel_frefair*v2x_frassoc_thick*v2x_suffr*v2x_freexp_thick)

8

+.5(1/8v2x_accex+1/4v2xel_frefair+1/4v2x_frassoc_thick+1/8v2x_suffr+

1/4v2x_freexp_thick)

Thesumof theweightsof theadditive termsequals theweightof the interaction

term.Theadditivepartoftheformulaletsthetwocomponentsthatcanachievehighscores

based on the fulfillment of formal-institutional criteria (elected officials and suffrage)

togetherweighhalfasmuchastheothercomponentsthatenjoyastrongerindependent

standingintermsofrespectfordemocraticrights(cleanelections,freedomoforganization

and expression).3 In any event, because most of the variables are strongly correlated,

differentaggregationformulasyieldverysimilarindexvalues.Theofficialformulapresented

herecorrelatesat.94to.99withapurelymultiplicativeformula,apurelyadditiveformula,

one thatweights the additive terms twice asmuch as themultiplicative term, one that

weightsthemultiplicativetermtwiceasmuchastheadditiveterms,andonethatweights

suffragesixtimesasmuchastheotheradditiveterms.Themaindifferenceacrossthese

formulasisintheirmeanvalues,withsomebeingclosertooneandothers(i.e.themore

multiplicativeformulas)beingclosertozero.

The Electoral Democracy Index also serves as the foundation for the other four

indices.Therecanbenodemocracywithoutelectionsbut,followingthecanonineachof

thetraditionsthatarguesthatelectoraldemocracy is insufficientforatruerealizationof

“rulebythepeople,”thereismoretodemocracythanjustelections.Wethereforecombine

the scores for our Electoral Democracy Index (v2x_polyarchy) with the scores for the

components measuring deliberation, equalitarianism, participation, and liberal

constitutionalism, respectively. This is not an easy task. Imagine two components,

P=Polyarchy and HPC=High Principle Component (liberal, egalitarian, participatory, or

deliberative),4thatwewanttoaggregateintomoregeneraldemocracyindices,whichwe

will call DI (DeliberativeDemocracy Index, EgalitarianDemocracy Index, and so on). For

convenience,bothPandHPCarescaledtoacontinuous0-1interval.Basedonextensive

3OnecouldarguethatthesuffragedeservesgreaterweightbecauseitliesonadifferentdimensionthantheothersandisthekeycomponentofoneofDahl’stwodimensionsofpolyarchy(Dahl1971;Coppedgeetal.2008).However,ourformulaallowsarestrictedsuffragetolowertheElectoralDemocracyIndexconsiderablybecauseitdiscountsalltheothervariablesinthemultiplicativeterm.

4TheHPCsare indicesbasedon theaggregationofa largenumberof indicators (liberal=23,egalitarian=8,participatory=21,deliberative=5).

9

deliberations among the authors and othermembers of the V-Dem research group,we

tentativelyarrivedatthefollowingaggregationformula:

DI=.25*P1.6+.25*HPC+.5*P1.6*HPC

TheunderlyingrationaleforthisformulaforallfourDIsisthesameasthatforthe

ElectoralDemocracy Index: equalweighting of the additive terms and themultiplicative

term in order to respect both the Sartorian necessary conditions logic and a family

resemblance logic. For example, the degree of deliberation stillmatters for deliberative

democracy even when there is no electoral democracy, and electoral democracy still

mattersevenwhenthereisnodeliberation;butthehighestlevelofdeliberativedemocracy

canbeattainedonlywhenthereisahighlevelofbothelectoraldemocracyanddeliberation.

Themoreacountryapproximatespolyarchy,themoreitscombinedDIscoreshould

reflect the unique component. This perspective is a continuous version of theoretical

arguments presented in the literature saying that polyarchy or electoral democracy

conditions should be satisfied to a reasonable extent before the other democracy

componentgreatlycontributestothehighlevelindexvalues.Atthesametime,itreflects

theviewintheliteraturethat,whenacertainlevelofpolyarchyisreached,whatmattersin

termsof,say,participatorydemocracyishowmuchoftheparticipatorypropertyisrealized.

This argument also resembles the widespread perspective in the quality of democracy

literatureemphasizingthatthefulfillmentofsomebaselinedemocracycriteriaisnecessary

beforeitmakessensetoassessthequalityofdemocracy.5Giventhisbodyofliterature,it

becomesnecessarytospecifytherateatwhichacomponentshouldinfluenceaDIscore.

Wedosobyraisingthevalueofacomponentby1.6.Weidentifythisnumericvalueby

definingananchorpoint:whenacountryhasapolyarchyscoreof.5(inpractice,thisisa

thresholdontheElectoralDemocracyIndexbeyondwhichcountriestendtobeconsidered

electoraldemocraciesinaminimalsense)anditsHPCisatitsmaximum(1),thehighlevel

indexscoreshouldbe.5.6

Takentogether,theseindicesofferafairlycomprehensiveaccountingof“varieties

of democracy.” The five (soon to be seven) democracy indices constitute a first step in

5Foranoverview,seeMunck(2016).6Define theexponentasp. SettingPolyarchy=.5,HPC=1,andHLI=.5,andsolving forDI=.25*Polyarchy^p+.25*HPC+.5*Polyarchy^p*HPC,p=log(base0.5)of.25/.75≈1.6.

10

disaggregatingtheconceptofdemocracy.Thenextstepisthecomponents.

Components

Themain democracy components, already included in the discussion above, specify the

distinct properties associatedwith the principles. TheV-DemElectoralDemocracy Index

consistsoffivesub-components(eachofthesesub-componentsbeingindicesthemselves

built from a number of indicators) that together capture Dahl’s seven institutions of

polyarchy:freedomofassociation,suffrage,cleanelections,electedexecutive,andfreedom

ofexpression.Thecomponentindicesmeasuringtheliberal,deliberative,participatory,and

egalitarianpropertiesof democracy (majoritarian and consensualwill be released in the

near future) follow the principles of democracy described in the previous section – but

withoutthecoreunifyingelementofelectoraldemocracy.Theycaptureonlywhatisunique

foreachoftheprinciples.Assuch,thesecomponentsaremutuallyexclusive,ororthogonal

toeachother.

These main democracy components typically have several sub-components. For

example, the liberal democracy component consists of three sub-components, each

capturedwithitsownindex:theEqualitybeforethelawandindividuallibertyindex;the

Judicialconstraintsontheexecutiveindex;andtheLegislativeconstraintsontheexecutive

index.

Inadditiontothecomponentandsub-componentindicesthatarepartoftheV-Dem

democracy indicesconceptual scheme,membersof theV-Demteamhaveconstructeda

series of indices of lower-level concepts such as civil society, party institutionalization,

corruption,andwomen’spoliticalempowerment.Wealsolisttheseindicesintheappendix.

In total, V-Dem offers 5 indices of components, 14 sub- components, and 12 related

conceptswith19sub-concepts.TheV-Demdatasetincludesalloftheseindices.Published

V-Demworkingpapersalreadydetailmanyoftheseindices(e.g.papers#6,#13,#17-20).

Additionalworkingpaperswillprovidefurtherdetailsonotherindices.

Weusetwotechniqueswhenaggregatingintodemocracyindices,components,and

sub- components, as well as related concepts’ indices. For the first step, going from

indicatorsto(sub-)components,weaggregatethelatentfactorscoresfrommeasurement

model (MM) output. More specifically, we use relevant theoretical distinctions in the

11

literature togroup interval-levelMMoutput into setsof variables that sharea common

underlying concept.We then randomly select 100 draws from each variable’s posterior

distribution(seedetailsunder“MeasurementModels”below),anduseaunidimensional

Bayesian factor analysis (BFA) to measure this latent concept sequentially for each

randomly-selected draw in each grouping of variables. We then combine the posterior

distributions of the latent factor scores in each variable group to yield the latent factor

scores.Inallanalyses,thevariablesgenerallyloadhighlyontheunderlyingfactor.

For the next level in the hierarchy –another subcomponent, a component, or a

democracyindexdependingonthecomplexityoftheconceptualstructure(seeAppendixA)

– we take the latent factor scores from the separate BFAs and use in combination in

constructingthe“HigherLevelIndices”(HLIs).HLIsarethuscompositemeasuresthatallow

thestructureoftheunderlyingdatatopromulgatethroughthehierarchyinthesameway

astheBFAsdo–andcriticallycarryoverthefullinformationaboutuncertaintytothenext

levelinordertoavoidallowingtheaggregationtechniqueartificiallyincreasetheestimated

confidence – while being faithful to the theoretically informed aggregation formula.

FollowingtheformulaofeachHLI(seetheV-DemCodebook),wetakeaveragesorproducts

ofeachoftherelevantBFAfactorscoreposteriordistributions,andthencalculatethepoint

estimates(means)andconfidenceintervalsacrosstheresultingmatrixtogeneratetheHLI

estimates. For example, the liberal component of democracy index comprises three

elements: equality before the law and individual liberties, judicial constraints on the

executive,andlegislativeconstraintsontheexecutive.Webelievethesethreeelementsare

substitutiveandthereforetaketheaverageofthesethreeelementstoconstructtheliberal

componentindex.FortheDIs,weusetheequationsdiscussedabovetoassignweightsto

thecombinations.

Indicators

Thefinalstepindisaggregationistheidentificationofindicators.Inidentifyingindicatorswe

lookforfeaturesthat(a)arerelatedtoat leastonepropertyofdemocracy;(b)bringthe

political process into closer alignmentwith the coremeaningof democracy (rule by the

people);and(c)aremeasurableacrosspolitiesandtime.

Indicatorstaketheformofnominal(classifications,text,dates),ordinal(e.g.,Likert-

12

stylescales),orintervalscales.Somerefertodejureaspectsofapolity–rulesthatstatute

orconstitutionallaw(includingtheunwrittenconstitutionofstatesliketheUnitedKingdom)

stipulate.Othersrefertodefactoaspectsofapolity–thewaythingsareinpractice.

Thereareover350uniquedemocracyindicatorsintheV-Demdataset.Welisteach

indicator, along with its response-type, in the V-Dem Codebook. We discuss coding

proceduresingreaterdetailinthenextsection.TheV-Demdatasetcontainsmanyindicators

thatwedonotincludeinthecomponentanddemocracyindicesdiscussedabove,though

theyarerelatedtodemocracy.Theirabsencereflectsthefactthatwehavesoughttomake

thecomponent-anddemocracyindicesasorthogonalaspossibletoeachother,andalsoas

parsimoniousaspossible.Furthermore,wheneverwehavemeasuresofboththede jure

andthedefactosituationinastate,ourindicesbuildprimarilyonthedefactoindicators

becausewewant themeasures to portray the “real situation on the ground” as far as

possible.

Summary

Tosummarize,theV-Demconceptualschemerecognizesseverallevelsofaggregation:

●Coreconcept(1)●DemocracyIndices(5,soontobe7)

●DemocracyComponents(5)●Subcomponents,andrelatedconcepts(46)

●Indicators(≈350)

As anappendix to this document,weattacha tablewith a completehierarchyof

democracyindices,democracycomponentindices,democracysub-componentindices,and

indicators,aswellasthehierarchyofrelatedconceptindices.

Several important clarifications apply to this taxonomy. First, our attempt to

operationalizedemocracydoesnotattempttoincorporatethecausesofdemocracy(except

insofarassomeattributesofourfar-flungconceptmightaffectotherattributes).Regime-

typesmaybeaffectedbyeconomicdevelopment(Epsteinetal.2006),colonialexperiences

(Bernhard et al. 2004), or attitudes and political cultures (Almond & Verba 1963/1989;

Hadenius& Teorell 2005;Welzel 2007). However,we do not regard these attributes as

constitutiveofdemocracy.

13

Second,ourquesttoconceptualizeandmeasuredemocracyshouldnotbeconfused

with the quest to conceptualize and measure governance.7 Of course, there is overlap

betweenthesetwoconcepts,sincescholarsmayconsidermanyattributesofdemocracyto

beattributesofgoodgovernance.

Third,werecognizethatsomeindicatorsandcomponents(listedintheCodebook)are

moreimportantinguaranteeingapolity’soverall levelofdemocracythanothers,though

thepreciseweightingparametersdependuponone’smodelofdemocracy.

Fourth,aspectsofdifferentideasofdemocracysometimesconflictwithoneanother.

Atthelevelofprinciples,thereisanobviousconflictbetweenmajoritarianandconsensual

norms,whichadoptcontraryperspectivesonmostinstitutionalcomponents.Forexample,

protectingindividuallibertiescanimposelimitsonthewillofthemajority.Likewise,strong

civilsocietyorganizationscanhavetheeffectofpressuringgovernmenttorestrictthecivil

libertiesenjoyedbymarginalgroups(Isaacn.d.).Furthermore,thesameinstitutionmaybe

differentlyviewedaccordingtodifferentprinciplesofdemocracy.Forexample,thecommon

practiceofmandatoryvotingisclearlycontrarytotheliberalmodel(whereindividualrights

aresacrosanctandincludetherightnottovote),buttheparticipatorymodelsupportsthis

practice,sinceithasademonstratedeffectinboostingturnoutwhereversanctionsaremore

thannominal.

Suchcontradictionsareimplicitindemocracy’smultidimensionalcharacter.Nowide-

rangingempirical investigationcanavoidconflictsamongdemocracy’sdiverseattributes.

However, with separate indicators representing these different facets of democracy it

shouldbepossibletoexaminepotentialtradeoffsempirically.

Fifth,ourproposedsetofdemocracyindices,components,andindicators,whilefairly

comprehensive,isbynomeansexhaustive.Theproteannatureofdemocracyresistsclosure;

there are always potentially new properties/components/indicators that, from one

perspectiveoranother,maybeassociatedwiththisessentiallycontestedterm.Moreover,

7SeeRose-Ackerman(1999)andThomas(2010).Inglehart&Welzel(2005)arguethateffectivedemocracy–as opposed to purely formal or institutional democracy – is linked to rule of law: a formally democraticcountrythatisnotcharacterizedbytheruleoflawisnotdemocraticinthefullsenseoftheterm.InordertorepresentthisthickconceptofdemocracytheymultiplytheFreedomHouseindicesbyindicesofcorruption(drawnfromTransparencyInternationalortheWorldBank),producinganindexofeffectivedemocracy.SeeHadenius&Teorell(2005)andKnutsen(2010)forcriticaldiscussions.

14

someconceptionsofdemocracyaredifficulttocaptureempirically;thisdifficultyincreases

whenanalyzingtheseconceptionsovertimeandacrosscountriesonaglobalscale.Thisfact

limitsthescopeofanyempiricalendeavor.

Sixth,principlesandcomponents,whilemucheasiertodefinethandemocracy (at-

large), are still resistant to authoritative conceptualization. Our objective has been to

identifythemostessentialanddistinctiveattributesassociatedwiththeseconcepts.Even

so,wearekeenlyawarethatothersmightmakedifferentchoices,andthatdifferenttasks

require different choices. The goal of the proposed conceptual framework is to provide

guidance,notto legislate inanauthoritativefashion.Theschemademonstrateshowthe

various elements of V-Dem hang together, according to a particular set of inter-

relationships. We expect other writers will assemble and dis-assemble these parts in

whateverfashionsuitstheirneedsandobjectives.Inthisrespect,V-Demhasthemodular

qualitiesofaLegoset.

Finally,asshouldbeobvious,thissectionapproachesthesubjectfromaconceptual

angle.Elsewhere(e.g.,intheV-DemCodebookandinV-DemComparisonsandContrasts,as

wellasinworkingpapersfoundontheV-Demwebsite),wedescribetechnicalaspectsof

indexconstructioninmoredetail.

2. DataCollection

Theviabilityofanydatasethingescriticallyonitsmethodofdatacollection.V-Demaimsto

achievetransparency,precision,andrealisticestimatesofuncertaintywithrespecttoeach

(evaluativeandindex)datapoint.

HistoryofPolities

Ourprincipalconcerniswiththeoperationofpoliticalinstitutionsthatexistwithinlargeand

fairlywell-defined political units andwhich enjoy amodicumof sovereignty or serve as

operationalunitsofgovernance(e.g.,coloniesofoverseasempires).Werefertotheseunits

aspolitiesorcountries.8

8 We are not measuring democracy within very small communities (e.g., neighborhoods, school boards,

15

Wearenotconcernedmerelywiththepresentandrecentpastofthesepolities.In

our view, understanding the present – not tomention the future – requires a rigorous

analysisofhistory.Theregimesthatexisttoday,andthosethatwillemergetomorrow,are

theproductofcomplexprocessesthatunfoldoverdecades,perhapscenturies.Although

regimechangesaresometimessudden,likeearthquakes,thesedramaticeventsareperhaps

sometimes tobeunderstoodasacombinationofpent-up forces thatbuildupover long

spansoftime,notsimplytheprecipitatingfactorsthatreleasethem.Likewise,recentwork

has raised the possibility that democracy’s impact on policies and policy outcomes take

effect over a very long period of time (Gerring et al., 2005) and that there are indeed

sequencesintermsofnecessaryconditionsindemocratization(Wangetal.2015).Arguably,

short-termandlong-termeffectsarequitedifferent,whetherdemocracyisviewedasthe

cause or outcome of theoretical interest. For all these reasons, we believe that a full

understandingofdemocratizationdependsuponhistoricaldata.9

Theadvantageofourtopic–incontrastwithotherhistoricalmeasurementtaskssuch

asnational incomeaccounts– is thatmuchof theevidenceneeded to code featuresof

democracy is preserved in books, articles, newspapers archives, and living memory.

Democracyis,afterall,ahigh-profilephenomenon.Althoughasecretiveregimemayhide

thetruevalueofgoodsandservicesinthecountry,itcannotdisguisetheexistenceofan

election; those features of an election that might prejudice the outcome toward the

incumbent are difficult to obscure completely. Virtually everyone living in that country,

studyingthatcountry,orcoveringthatcountryforsomeforeignnewsorganizationoraid

organizationhasaninterestintrackingthisresult.

Thus,weregardthegoalofhistoricaldatagatheringasessentialandalsorealistic,

even if it cannot be implemented for every possible indicator of democracy. V-Dem

thereforeaimstogatherdata,wheneverpossible,backto1900forallterritoriesthatcan

municipalities, corporations), in contexts where the political community is vaguely defined (e.g.,transnationalmovements),oronagloballevel(e.g.,theUnitedNations).Thisisnottosaythattheconceptofdemocracyshouldberestrictedtoformalandwell-definedpolities.Itissimplytoclarifyourapproach,andtoacknowledgethatdifferentstrategiesofconceptualizationandmeasurementmayberequiredfordifferentsubjectareas.

9 This echoes a persistent theme presented in Capoccia and Ziblatt (2010), Knutsen, Møller & Skaaning(forthcoming), Teorell (2011), and in other historically grounded work (Nunn 2009; Mahoney &Rueschemeyer2003;Pierson2004;Steinmo,Thelen,&Longstreth1992).

16

claimasovereignorsemi-sovereignexistence(i.e.theyenjoyedadegreeofautonomyat

leastwithrespecttodomesticaffairs)andserveastheoperationalunitofgovernance.The

lattercriterionmeansthattheyaregoverneddifferentlyfromotherterritoriesandwemight

reasonablyexpectmanyofour indicators to varyacross theseunits. Thus, in identifying

political units we look for those that have the highest levels of autonomy and/or are

operationalunitsofgovernance.Thesesortsofunitsarereferredtoas“countries,”evenif

theyarenot fullysovereign.Thismeans, forexample, thatV-Demprovidesacontinuous

time-seriesforEritreacodedasanItaliancolony(1900-41),aprovinceofItalianEastAfrica

(1936-41),aBritishholdingadministeredunder the termsofaUNmandate (1941-51),a

federation with Ethiopia (1952-62), a territory within Ethiopia (1962-93), and an

independent state (1993-). For further details, see V-Dem Country Coding Units. In the

future,weplantoaddinformationinthedatasetanddocumentationtolinkpredecessor

andsuccessorstates,facilitatingpanelanalysiswithcontinuouscountry-levelunits.

V-Dem provides time-series ratings that reflect historical changes as precisely as

possible. Election-specific indicators are coded as events occurring on the date of the

election.Wecodeotherindicatorscontinuously,withanoption(thatsomecodersutilize)

tospecifyexactdates(day/month/year)correspondingtochangesinaninstitution.

Date-specificdatacanbeaggregatedat12-monthintervals,whichmaybeessential

for time-series where country-years form the relevant units of analysis. The V-Dem

“standard”dataset is inthecountry-year format,wheredate-specificchangeshavebeen

aggregatedtogetherattheyearlevel.However,wealsoprovideacountry-datedatasetfor

userswhowantgreaterprecision.Inthedataarchiveaccessibleviathedatadownloadpage

onourwebsite,wealsoprovidetherawcoder-leveldata.Doingsoallowsuserstoinspect

thedatadirectlyoruseitforalternateanalyses.Finally,inthesamearchivewealsoprovide

theposteriordistributionsfromtheBayesianordinalIRTmodelforeachvariabletofacilitate

theirdirectuseinanalyses.

Currently,weareworkingtoextendV-Demcodingbackfurtherinhistoricaltime,i.e.,

to1789,for85sovereigncountriesandforaselectionofindicators.Thiscodingwillenhance

ourknowledgeofdemocraticdevelopmentforcountrieswhoseprocessofdemocratization

began prior to the twentieth century. It will also enhance our knowledge of the pre-

democratic history of all countries, a history thatmay exert an enduring influence over

17

subsequentdevelopmentsinthe20thand21stcenturies.

CodingTypes

The350+V-DemspecificindicatorslistedinV-DemCodebookfallintofourmaintypes:(A*)

factualindicatorspre-codedbymembersoftheV-Demteamandprovidedinthesurveys

forCountryCoordinatorsand–Expertstoindicatetheirconfidenceregardingthepre-coded

data, (A) factual indicators codedbymembersof theV-Demteam, (B) factual indicators

coded by Country Coordinators and/or members of the V-Dem team, (C) evaluative

indicatorsbasedonmultipleratingsprovidedbyexperts,and(D)compositeindices.PartI

of V-Dem Codebook describes these indicators Parts II and III provide a fifth type of

indicators:(E)extantdata(bothfactualandsubjective).

We gather Type (A*) and (A) data from extant sources, e.g., other datasets or

secondarysources,aslistedintheCodebook.Thesedataarelargelyfactualinnature,though

somecoderjudgmentmayberequiredininterpretinghistoricaldata.PrincipalInvestigators

andProjectManagerssupervisethecollectionof thesedata,whichassistantresearchers

connectedtotheprojectcarryoutusingmultiplesources,withinputfromV-Dem’sCountry

Coordinators.

CountryCoordinators,underthesupervisionofRegionalManagers,gatherType(B)

datafromcountry-specificsourcesby.Foranumberofcountries,assistantresearchersat

theV-DemInstitutehavecodedtheseindicatorsduringtheupdateswhentheoriginalseries

goingfrom1900to2012havebeenextendedto2016.AswithType(A*)and(A)data,this

sortofcodingislargelyfactualinnature.

Type(C)datarequiresagreaterdegreeof judgmentaboutthestateofaffairs ina

particular country at a particular point in time. Country Experts code these data. These

experts are generally academics (about 80%) or professionals working media, or public

affairs(e.g.,senioranalysts,editors,judges);about2/3arealsonationalsofand/orresidents

in a country and have documented knowledge of both that country and a specific

substantive area. Generally, each Country Experts code only a selection of indicators

followingtheirparticularbackgroundandexpertise(e.g.thelegislature).

Type(D)dataconsistsofindicescomposedfrom(A),(B),or(C)variables.Theyinclude

18

cumulativeindicatorssuchas“numberofpresidentialelectionssince1900”aswellasmore

highlyaggregatedvariablessuchasthecomponentsanddemocracyindicesdescribedinthe

previoussectionanddetailedinAppendixA.

WedrawType(E)datadirectlyfromothersources.TheyarethereforenotaV-Dem

product.TherearetwogenresofE-data.Thefirstgenreconsistsofalternativeindicesand

indicators of democracy found in Part II of V-Dem Codebook, which may be useful to

compare and contrast with V-Dem indices and indicators. This genre also includes

alternativeversionsoftheV-Demindicesthatareordinalinsteadofinterval(Lindberg2015).

ThesecondtypeofE-indicatorsconsistoffrequentlyusedcorrelatesofdemocracysuchas

GDP.TheyarefoundinPartIII.

CountryExpertRecruitment

Type (C)coding–byCountryExperts– involvesevaluative judgmentson thepartof the

coder.Asaresult,wetakeanumberofprecautionstominimizeerrorinthedataandto

gaugethedegreeofimprecisionthatremains.10

An important aspect of these precautions is the fact thatwe endeavor to find a

minimumoffiveCountryExpertstocodeeachcountry-yearforeveryindicator.Thequality

and impartiality of C-data naturally depends on the quality of the Country Experts that

provide the coding. Consequently, we pay a great deal of care and attention to the

recruitmentofthesescholars,whichfollowsanexactingprotocol.

First,weidentifyalistofpotentialcodersforacountry(typically100-200namesper

country).RegionalManagers,inconsultationwithCountryCoordinators,usetheirintimate

knowledgeofacountrytocompilethebulkoftheexpertsonthislist.AssistantResearchers

locatedattheV-DemInstitute(UniversityofGothenburg)alsocontributetothislist,using

readilyavailableinformationdrawnfromtheInternet.11Othermembersoftheprojectteam

(PIs,PMs,andassociates)mayalsosuggestcandidates.Atpresent,ourdatabaseofpotential

CountryExpertscontainssome18,000names.

10ForaperceptivediscussionoftheroleofjudgmentincodingseeSchedler(2012).11ResearchAssistantsattheUniversityofNotreDamealsosuppliedmorethan3,000namesforallregionsin2011-2013,usinginformationfromtheInternet.

19

RegionalManagers and Country Coordinators thus play a critical role in the data

collectionprocess.V-Dem’sapproachistorecruitRegionalManagerswhoarenationalsor

residentsofoneofthecountriesineachregionwheneverpossible.TheRegionalManagers

aretypicallyprominentscholars inthefieldwhoareactiveasprofessors intheregion in

question.Insomecases,RegionalManagersarelocatedoutsideoftheregion,iftheyare

currentlyactiveinwell-respectedinternationalthinktanksorsimilarinstitutions.Country

Coordinatorsarealmostalwaysnationalsandresidentsofthecountrytobecoded.Theyare

alsoscholars,althoughtheyaretypicallymorejuniorthanRegionalManagers.

Using short biographical sketches, publications, website information, or similar

material we compile basic information for each Country Expert: their country of origin,

current location, highest educational degree, current position, and area of documented

expertise(relevantfortheselectionofsurveystheexpertmightbecompetenttocode)to

makesureweadheretothefiverecruitmentcriteria.

RegionalManagers,CountryCoordinators,andotherprojectteammembersreferto

five criteriawhen drawing up the list of potential Country Experts. Themost important

selectioncriterionisanindividual’sexpertiseinthecountry(ies)andsurveystheymaybe

assigned to code. This expertise is usually signified by an advanced degree in the social

sciences,law,orhistory;arecordofpublications;orpositionsinoutsidepoliticalsocietythat

establishtheirexpertiseinthechosenarea(e.g.awell-knownandrespectedjournalist;a

respectedformerhighcourtjudge).RegionalManagersandCountryCoordinatorsmayalso

indicatewhichsurveysapotential coderhasexpertise in.Naturally,potential codersare

drawntoareasofthesurveythattheyaremostfamiliarwith,andareunlikelytoagreeto

code topics they know little about. As a result, self-selection alsoworks to achieve our

primarygoalofmatchingquestionsinthesurveywithcoderexpertise.

Thesecondcriterionisconnectiontothecountrytobecoded.Bydesign,threeoutof

five(60%)oftheCountryExpertsrecruitedtocodeaparticularcountry-surveyshouldbe

nationalsorpermanentresidentsofthatcountry.Exceptionsaremadeforasmallnumber

of countries where it is difficult to find in-country coders who are both qualified and

independentofthegoverningregime,orwherein-countrycodersmightbeplacedatrisk.

ThiscriterionhelpsusavoidpotentialWesternorNorthernbiasesincoding.

The third criterion is the prospective coder’s seriousness of purpose, i.e. her

20

willingnesstodevotetimetotheprojectandtodeliberatecarefullyoverthequestionsasked

in the survey. Sometimes, personal acquaintanceship is enough to convince a Regional

ManagerandaCountryCoordinatorthatapersonisfit,orunfit,forthejob.Sometimes,this

featurebecomes apparent in communicationswithProgramManagers that precede the

offer to work on V-Dem. This communication is quite intensive, with an average of 13

interactionsbeforecodingisconcluded,andinvolvesrequiringthepotentialcodertoread

andworkwithseverallengthy,detaileddocuments.Thisprocessreadilyidentifiespotential

coderswhoarenotseriousenough.

The fourth criterion is impartiality. V-Demaims to recruit coderswhowill answer

surveyquestionsinanimpartialmanner.Wethereforeavoidthoseindividualswhomight

bebeholdentopowerfulactors–byreasonofcoercivethreatsormaterialincentives–or

whoserveasspokespersonsforapoliticalpartyorideologicaltendency.Closeassociation

(currentorpast)withpoliticalparties,seniorgovernmentofficials,politicallyaffiliatedthink-

tanksorinstitutesisgroundsfordisqualification.Incaseswherefindingimpartialcodersis

difficult,weaimtoincludeavarietyofcoderswho,collectively,representanarrayofviews

andpoliticalperspectivesonthecountryinquestion.

Thefinalcriterionisobtainingdiversityinprofessionalbackgroundamongthecoders

chosenforaparticularcountry.Forcertainareas(e.g.,themedia,judiciary,andcivilsociety

surveys)suchdiversityentailsamixtureofacademicsandprofessionalswhostudythese

topics.Italsomeansfindingexpertswhoarelocatedatavarietyofinstitutions,universities

andresearchinstitutes.

Afterweighingthesefivecriteria,wegivethe100-200potentialexpertsonourlistof

candidatesarankfrom“1”to“3,”indicatingtheorderofprioritywegivetorecruitingan

Expert.TheRegionalManagersandCountryCoordinatorsareprimarilyresponsibleforthe

ranking,butProgramManagersandoneof thePrincipal Investigatorsmay review these

choices.

Using this process,wehave recruitedover 2,800 scholars and experts fromevery

corneroftheworld.About26percentoftheCountryExpertsarewomen,12andover68

12ThenumberofwomenamongtheranksofourCountryExpertsislowerthanwewouldhaveliked,anditoccurreddespiteourstrenuousefforts.However,itreflectsgenderinequalitieswithregardtoeducationanduniversitycareersintheworld.

21

percenthavePhDsorMAsandareaffiliatedwithresearchinstitutions,thinktanks,orsimilar

organizations.

Inordertopreserveconfidentiality,V-Demhasadoptedapolicyofneitherconfirming

nordenyingtheidentitiesofCountryExperts.Wealsofollowaregulationofkeepingthe

Personal Identifiable Information (PII)protected, all this sensitive informationassociated

with Country Experts is handled and stored with care. During update 2017, only three

ProgramManagerswereactively involved in interactionswithCountryExpertsandwere

awareoftheidentitiesofthefinalchosenCountryExperts.Theseindividualsalsohandleall

correspondencewithCountryExperts,sothisconfidentialityisnotinadvertentlyrevealed

throughcommunication.

Thus,whiletheidentityofothermembersoftheV-Dementerpriseispublicizedon

ourwebsite,wepreservetheconfidentialityofCountryExperts.Severalreasonsliebehind

thisdecision.First,thereareanumberofcountriesintheworldwhereauthoritiesmight

sanctionCountryExperts,ortheirfamiliesorfriends,fortheirinvolvementintheproject.

Second,thereisnowaytopredictwhichcountrymayinthefuturebecomerepressiveand

thereforesanctiontheCountryExperts.Third,weanticipatethatV-Demdatamaybecome

used inevaluationsandassessments internationally inwaysthatcouldaffectacountry’s

status. Thus, one may foresee incentives for certain countries’ governments and other

actorstotrytoaffecttheirratings.Forallthesereasons,weconsideritessentialtopreserve

CountryExpertanonymity.

ExpertCodingProcess

TheProgramManagersattheV-DemInstitute(UniversityofGothenburg)issueinvitations

until thequotaof fiveCountryExpertspersurvey (country-year indicator).13 Weusually

recruit6-7expertstobeabletoreplacethosewhofailtobeginorcompletethesurveyin

time.Codersreceiveamodesthonorariumfortheirworkthatisproportionaltothenumber

ofsurveystheyhavecompleted.

C-indicatorsareorganizedintofourclustersandelevensurveys:

13BeforeJuly2014,therewasathirdProgramManagerattheKelloggInstituteoftheUniversityofNotreDamewhomanagedmostcountryexpertsinLatinAmericaandafewintheMiddleEastandNorthAfrica.

22

1. Elections Politicalparties/electoralsystems

2. Executive Legislature Deliberation

3. Judiciary Civilliberty Sovereignty

4. CivilsocietyorganizationsMedia

Politicalequality

Wesuggest(butdonotrequire)thateachCountryExpertcodeatleastonecluster.

In consultation with the Country Coordinators and Principal Investigators, Regional

ManagerssuggestwhichCountryExpertmightbemostcompetenttocodewhichsurveys.

WethenconsultwiththeCountryExpertaboutwhichcluster(s)theyfeelmostcomfortable

coding.Mostcodeonetotwoclustersofsurveys.Thismeansthat,inpractice,adozenor

more Country Experts provide ratings for each country (with a target of five for each

country/indicator/year,asstated).14

AllCountryExpertscarryouttheircodingusingaspeciallydesignedonlinesurvey.

Theweb-basedcodinginterfacesaredirectlyconnectedwithapostgresdatasetwherewe

storetheoriginalcoder-leveldata.Figure4providesanexampleofthecodinginterface.

ThecodinginterfaceisanessentialelementofV-Dem’sinfrastructure.Itconsistsof

aseriesofweb-basedfunctionsthatallowCountryExpertsandCountryCoordinatorsto(1)

logintothesystemusingtheirindividual,randomizedusernameandself-assigned,secret

password;(2)accesstheseriesofsurveysassignedtothemforaparticularcountry(orset

ofcountries);and(3)submitratingsforeachquestionoveraselectedseriesofyears.

Thecoding interfaceallows formanytypesofquestions (binary,ordinal,multiple

selection, numeric, range, text, date, and country-list selection), country and question-

specificyearmasks(e.g.,allowingthecodingofelectionsonlyinyearstheyoccurredforthat

14Insomerarecases---mainlysmallandunder-studiedcountries---weaskindividualexpertstocodethewholesetofsurveys,simplybecauseexpertsonthevariousspecificpartsofthesurveyarenotavailable.Similarly,itisalsonotalwayspossibletoreachthegoalofhavingfivecountryexpertscodeeachindicatorforthesecountries.

23

country),auto-filleddefaultdata(suchasnamesofheadsofstateforparticularcountry-

years),andquestion-specificinstructionsandclarifications.

Theinterfacealsorequiresthat,foreachrating,expertsassignalevelofconfidence,

indicatinghowconfidenttheyarethattheirratingiscorrect(onascaleof0-100,whereeach

5-percentintervalhasasubstantiveanchorpoint,inadditiondescriptivetextsareprovided

at0%,20%,40%,60%,80%,90%,and100% intervals),providinganother instrument for

measuringuncertaintyassociatedwiththeV-Demdata.Weincorporatethisconfidenceinto

themeasurementmodel.CountryExpertsalsohaveanopportunitytoregisteruncertainty

inthe“Remarks”fieldthatliesattheendofeachsectionofthesurvey.Here,expertscan

comment(inprose)onanyaspectoftheindicatorsorratingsthatshefoundproblematicor

difficulttointerpret.

Fig.4ExampleofCodingInterface

Finally, in order to ensure wide recruitment of potential experts, and minimize

confusiondue tounfamiliaritywith English,we translate all type-Cquestions, aswell as

24

coder-instructionsanddocumentationforthem,intofiveotherlanguages:Arabic,French,

Portuguese,Russian,andSpanish.Approximately82%ofcoderscodeintheEnglishversion

ofthequestionnairewhile18percentofthecoderscodeinanon-English(7%-French,6%

- Spanish, 3% - Russian, 1% - Arabic and 1% - Portuguese). Country Experts get a small

remunerationasatokenofappreciationfortheirtime.15

Wetakeanumberofstepstoassureinformedconsentandconfidentialityamong

participants.Theonlinesurveyprovides full informationabouttheproject (includingthis

document)andtheuseofthedata,sothatcodersarefullyinformed.Italsorequiresthat

prospectivecoderscertify that theyaccept the termsof theagreement.Theyaccess the

surveysonlywitharandomizedusernamethatweassignandasecretpasswordthatthey

createthemselves.Westorethedatatheysupplyonafirewall-protectedserver.Anydata

we release to thepublic excludes information thatmightbeused to identify coders.All

personally identifying information is kept in a separate database in order to ensure

confidentiality.

Aspeciallydesignedprogramminginterfaceisemployedtomanagethedatabaseof

potentialcountryexperts.Itincludesmanytoolsthatenableustohandleover2,800Country

Experts while guaranteeing their safety and confidentiality. These tools also ensure

consistency in instructions and information sent to Country Experts, quality control and

cleaningofdata,followupandevaluationofthecodingprocess.Itisdirectlylinkedtothe

postgresdatabasewhereratingsarestored.

Thecodermanagementtoolisjustoneofover50sophisticatedtoolsamongtheV-

Demmanagementinterfacesinthesoftware.Therearetoolsformanagementofcountries,

rounds of surveys, surveys and questions, country coordinators, regional managers, for

logging activities, analyses of progress on recruitment as well as coding, planning, and

generalmanagement.Aweb-interfaceportal isconnectedto themanagementsoftware,

allowing Regional Managers to securely upload Country Expert rosters to the database

15Fromwhatwecantell,thisisnotasignificantthreattocodingvalidity.Fewindividualsseemtohavebeenmotivatedtoconductthisarduouscodingassignmentforpurelymonetaryreasons:V-Dempaysverylittlerelativetowhathighlyqualifiedexpertscouldearnforthesameamountofworkfromotherpursuits.Furtherstrengtheningthispoint,thereseemstobenorelationshipbetweenthewealthofthecountryandourabilitytorecruitcoders:wehavefacedchallengesgettingexpertstoagreetoconductcodingforthepoorestaswellastherichestcountriesintheworld.

25

withouthavingtoshareconfidentialinformationviaemail.

Bridge-andlateralcoding

In addition to regular ratings by multiple Country Experts for C-type indicators, we

encourage Country Experts to conduct bridge coding (coding ofmore than one country

throughtime)andlateralcoding(codinglimitedtoasingleyear–2012).Thepurposeofthis

additionalcodingistoassurecross-countryequivalencebyforcingcoderstomakeexplicit

comparisonsacrosscountries.Thishelpsthemeasurementmodelestimate,andcorrectfor,

systematic biases across coders and across countries thatmay result if Country Experts

employvaryingstandards intheirunderstandingofaquestion,e.g.,aboutwhata“high”

levelofrepressionmightconsistof.

Throughoutimplementationoftheproject,wehaveencouragedCountryExpertsto

codemultiplecountriesover time -bridge coding.Anexpertwhoagrees tocodeoneor

moreadditionalcountriesreceivesthesamesetofsurveysforthesametimeperiodasthe

originalcountrytheycoded;bridgecodingthereforetypicallycoverstwotimeperiods:1900

topresent,or2005tothepresent.BridgecodinghelpsusbettermodelhowCountryExperts

makejudgmentsbetweendifferentresponsecategories,andallowsustoincorporatethis

informationintotheestimatedscoreforeachcountry-indicator-year/date.

Bridge coding is most useful when the chosen countries have different regime

histories.ThisgeneratesvarianceacrossaCountryExpert’sratings,whichinturnprovides

information about the coder’s judgments that can be used to inform themeasurement

model.Inordertomaximizevariance,andthereforegainasmuchinformationaspossible

abouteachexpert’s thresholdsand reliability,weencourageCountryExperts to select–

fromamongcountriestheyarefamiliarwith–thosethathavethemostdistinctivehistorical

trajectories.

AsofMarch2017,wehaveover530bridgecoders–about19percentofallCountry

Experts.Onaverage,theseexpertscode2.2countries.

ConstraintsoftimeorexpertisesometimespreventCountryExpertsfromconducting

bridgecoding. In thesesituations,weencourageCountryExperts toperformthesimpler

typeofcross-countrycomparisoncalledlateralcoding.Thatis,inadditiontotheiroriginal

26

codingofonecountryovertime(e.g.,from1900tothepresent),theycodeanumberof

countriesforasinglepointintime–January1,2012–focusingonthesamesetofquestions.

SomeCountryExpertshavecodedupto14countries.Moretypically,lateralcoding

extendstoafewcountries.Todate,350CountryExperts(about12%)haveperformedlateral

coding,coveringonaverageof5.5countriesand6.3surveys.Asaresult,lateralcodingby

regular Country Experts has provided linkages equivalent to over 1,100 “fully covered”

countries–inotherwords,countriesthathavebeen“cross-coded”bylateral/bridgecoding

acrossallindicatorsinthedataset.

Phasesofthedatacollection

ThefirstphaseofV-Dem,comprisingofdatacollectionfortheentireworldfrom1900to

2012,beganinMarch2012andwasconcludedinfall2013.167countries/territoriesexisting

todaywereincluded,thisrequiredtheinvolvementofsome2,000coders.

ByDecember2014,V-Dementeredintoanewphasebytheveryfirstupdateofthe

data.Goingforwardwehopetobeabletoupdatethedataonanannualorbiannualbasis

forall/asmanycountriesaspossible.

Firstdataupdate:Datawascollectedfor2013-2014for54countriesandcollected

dataforthefull-timeperiodfrom1900to2014forsixnewcountries.

Seconddataupdate:ByMarch2016thesecondupdatewasconcluded.Thistime

datafor76countriesfor2013-2015hadbeenupdated.

Third data update: Is launched in April 2017 and includes updated data for 174

countries,includingfournewcountries.Thisistheveryfirstcompletedataupdate

thattheV-Deminstituteisconducting.TheV-Demdatabasenowcontainsmorethan

17milliondatarecords.

Duetocoderattrition,codingfortheupdatesareconductedbyamixofreturningCountry

ExpertsandnewCountryExperts.Astoencourageconsistencyinratingsovertime,Country

Expertsareabletoseeandchangealltheirpreviousratings.WeasknewCountryExpertsto

codeform2005anonwardssoastoensurethattheirscoresoverlapbyanumberofyears

withreturningCountryExperts’ratings.

Forthesecondandthirdupdateofthedataweimplementedaseriesofvignettes

27

foreachsurvey togiveusadditional leverageonmeasurementerror.The fourthupdate

takesplaceOctober2017-March2018,withthereleaseofdatabyMarch2018.

3. Measurement

Havingdiscussedtheprocessofdatacollection,weproceedtothetaskofmeasurement.

Underthisrubric,weinclude(a)thequestionnaire,(b)ourmeasurementmodel,(c)methods

ofidentifyingerrorinmeasurement,(d)studiesofmeasurementerror,and(e)methodsof

correctingerror.Inprinciple,thediscussionsarerelevantfordifferenttypesofdata(A,B,

andCintheV-Demscheme)butmostifnotallofthemaremuchmoreacutewhenitcomes

toexpert-basedcodingofevaluative,non-factualyetcriticalindicators.Hence,mostofthe

followingfocusesontheC-typeindicators.

TheQuestionnaire

Themost important featureofasurvey is theconstructionofthequestionnaire itself. In

craftingindicatorstomeasuretheC-typedata,wehavesoughttoconstructquestionswith

bothspecificandclearmeanings,andwhichdonotsuffer fromtemporalorspatialnon-

equivalence.Todesignthesequestions,weenlistedleadingscholarsondifferentaspectsof

democracyanddemocratizationasProjectManagers.

We enrolled each Project Manager because of her record of scholarly

accomplishment in a particular area related to issues of democracy (e.g. legislatures,

executives, elections, and civil society), with the goal of creating a team that also had

substantiveexperiencesandexpertiseonallregionsoftheworld.ProjectManagersbegan

designingsurvey-questionsintheirareaofexpertisein2009,andwecollectivelyreviewed

andrefinedtheirquestionsoverthecourseoftwoyears.

WeimplementedapilotoftheV-Demsurveyin2011,whichservedasaninitialtest

ofourquestionnaire.Itwasimplementedfor12countries,two(one“easy”andone“hard”)

fromeachofthesixmajorregionsoftheworldenlistingover120pilot-CountryExpertsand

resultedinsome450,000ratingsonpreliminaryindicators.Theresultspromptedrevisions

inthenextroundofsurveys.Anotherroundofcollectivedeliberationfollowed, involving

consultationswithscholarsoutsideoftheprojectteam.TherevisedquestionsforC-coding

28

thusenduredseveralroundsofreviewwithProjectManagersandoutsideexpertsoverthe

courseoftwoyearsbeforeemergingintheirfinalform,asdescribedintheCodebook.

Identifying,Correcting,andQuantifyingMeasurementError

Evenwithcarefulquestiondesign,aprojectofthisnaturewillencountererror.Sucherror

maybetheproductoflinguisticmisunderstandings(mostofourcodersdonotspeakEnglish

astheirfirstlanguage,andsometakethesurveyinatranslatedform),misunderstandings

aboutthewayaquestionappliestoaparticularcontext,factualerrors,errorsduetothe

scarcityorambiguityofthehistoricalrecord,differinginterpretationsabouttherealityofa

situation,variationinstandards,coderinattention,errorsintroducedbythecoderinterface

orthehandlingofdataonceithasbeenenteredintothedatabase,orrandommistakes.

Someof theseerrorsare stochastic in the senseofaffecting theprecisionofour

estimatesbutnottheirvalidity.Othererrorsaresystematic,potentiallyintroducingbiasinto

theestimatesthatweproduce.Inthissection,wefirstdescribethemethodologicaltools

weusetomodelandcorrectforsystematicbiasincoders’answerstoourquestions,aswell

astoprovideestimatesofthereliabilityofthesecodings.Wethendescribetheprocedures

weusetoassessthevalidityofourestimates.Finally,weexplainhowweidentifythemost

serioussourcesofmeasurementerror,inordertocontinuouslyimprovehowwegatherand

synthesizedata.

MeasurementModels

ThemostdifficultmeasurementproblemsconcerntheC-typequestions,allofwhichrequire

substantialcaseknowledgeandgenerallysomedegreeofsubjectiveevaluation.Havingfive

codersforeachofthesequestionsisimmenselyuseful,asitallowsustoconductinter-coder

reliabilitytests.Thesesortsoftests–standardinmostsocialsciencestudies–areonlyrarely

ifeveremployedinextantdemocracyindices.

Whileweselectexpertscarefully,theyexhibitvaryinglevelsofreliabilityandbias,

and may not interpret questions consistently. In such circumstances, the literature

recommends that researchers usemeasurementmodels to aggregate diversemeasures

wherepossible,incorporatinginformationcharacterizedbyawidevarietyofperspectives,

29

biases,and levelsof reliability (Bollen&Paxton2000,Clinton&Lapinski2006,Clinton&

Lewis2008, Jackman2004,Treier& Jackman2008,Pemstein,Meserve&Melton2010).

Therefore,tocombineexpertratingsforaparticularcountry-indicator-yeartogeneratea

single“bestestimate”foreachquestion,weemploymethodsinspiredbythepsychometric

and educational testing literature (see, e.g., Lord&Novick 1968, Jonson&Albert 1999,

Junker1999,Patz& Junker1999).Theunderpinningsof thesemeasurementmodelsare

straightforward:theyusepatternsofcross-rater(dis)agreementtoestimatevariations in

reliabilityandsystematicbias.Inturn,thesetechniquesmakeuseofthebiasandreliability

estimatestoadjustestimatesofthelatent—thatis,onlyindirectlyobserved—concept(e.g.,

executiverespectfortheconstitution,judicialindependence,orpropertyrights)inquestion.

Thesestatisticaltoolsallowustoleverageourmulti-coderapproachtobothidentifyand

correctformeasurementerror,andtoquantifyconfidenceinthereliabilityofourestimates.

Variationintheseconfidenceestimatesreflectsituationswhereexpertsdisagree,orwhere

little information is available because few raters have coded a case. These confidence

estimatesare tremendouslyuseful. Indeed, to treat thequalityofmeasuresof complex,

unobservableconceptsasequalacrossspaceandtime,ignoringdramaticdifferencesinease

ofaccessandmeasurementacrosscases,isfundamentallymisguided,andconstitutesakey

threattoinference.

ThemajorityoftheC-typequestionsareordinal: theyrequireCountryExpertsto

rankcasesonadiscretescale. Take,forexample,thefollowingquestionaboutelectoral

violence:

Question:Inthisnationalelection,wasthecampaignperiod,electionday,andpostelectionprocessfreefromothertypes(notbythegovernment,therulingparty,ortheiragents)ofviolencerelatedtotheconductoftheelectionandthecampaigns(butnotconductedbythegovernmentanditsagents)?Responses:

0. No. There was widespread violence between civilians occurring throughout the electionperiod,orinanintenseperiodofmorethanaweekandinlargeswathsofthecountry.Itresultedinalargenumberofdeathsordisplacedrefugees.

1. Notreally.Thereweresignificantlevelsofviolencebutnotthroughouttheelectionperiodorbeyondlimitedpartsofthecountry.Afewpeoplemayhavediedasaresult,andsomepeoplemayhavebeenforcedtomovetemporarily.

2. Somewhat.Thereweresomeoutburstsoflimitedviolenceforadayortwo,andonlyinasmallpartofthecountry.Thenumberofinjuredandotherwiseaffectedwasrelativelysmall.

3. Almost.Therewereonlyafewinstancesofisolatedviolentacts,involvingonlyafewpeople;noonediedandveryfewwereinjured.

4. Peaceful.Noelection-relatedviolencebetweenciviliansoccurred.

30

Note, inparticular, thattheserankingsdonot followan interval-levelscale. Onecannot

subtractalmostfrompeacefulandgetnotreally.Furthermore,itneednotbethecasethat

thedifferencebetweennotreallyandsomewhatisthesameasthatbetweenalmostand

peaceful.Perhapsmostimportantly,althoughwestrivetowritequestionsandresponses

that are not overly open to interpretation, we cannot ensure that two coders look at

descriptionslikesomewhatinauniformway—evenwhensomewhatisaccompaniedbya

carefully formulated description—especially because coders have widely varying

backgroundsandreferences.Inotherwords,onecoder’ssomewhatmaybeanothercoder’s

not really; a problem known as scale inconsistency. Therefore, we use Bayesian item

response theory (IRT) modeling techniques (Fox 2010) to estimate latent polity

characteristicsfromourcollectionofexpertratingsforeachordinal(C)question.

Specifically, we fit ordinal IRTmodels to each of our ordinal (C) questions. (See

Johnson&Albert1999foratechnicaldescriptionofthesemodels.)Thesemodelsachieve

threegoals.First,theyworkbytreatingcoders’ordinalratingsasimperfectreflectionsof

interval-levellatentconcepts.Withrespecttotheexamplequestionabove,ourIRTmodels

assumethatelectionviolencerangesfromnon-existenttoendemicalongasmoothscale,

andcodersobservethislatentcharacteristicwitherror.Therefore,whileanIRTmodeltakes

ordinalvaluesasinput,itsoutputisaninterval-levelestimateofthegivenlatenttrait(e.g.

election violence). Interval-valued estimates are valuable for a variety of reasons; in

particular,theyareespeciallyamenabletostatisticalanalysis.Second,IRTmodelsallowfor

the possibility that coders have different thresholds for their ratings (e.g. one coder’s

somewhatmight fall above another coder’salmost on the latent scale), estimate those

thresholds from patterns in the data, and adjust latent trait estimates accordingly.

Therefore, they allow us to correct for this potentially serious source of bias, known as

differential itemfunctioning (DIF).16This isvery important inamulti-raterproject likeV-

Dem,wherecodersfromdifferentgeographic,cultural,andotherbackgroundsmayapply

differingstandardstotheirratings.Finally,IRTmodelsassumethatcoderreliabilityvaries,

16Givencurrentlyavailabledata,wemustbuild inassumptions—formally, theseareknownashierarchicalpriors—thatrestricttheextenttowhichcoders’thresholdestimatesmayvary.Informally,whileweallowcoderstolookatordinalrankingslikesomewhatandalmostdifferently,weassumethattheirconceptionsarenottoodifferent.Weareworkingtorelaxtheseassumptionsbycollectingmoredata.TechnicaldetailsareavailableinV-DemWorkingpaperno.21,andfullcodeisreleasedwiththedataset.

31

produceestimatesof raterprecision, anduse theseestimates—in combinationwith the

amountofavailabledataandtheextenttowhichcodersagree—toquantifyconfidencein

reportedscores.

Sinceourcodersgenerallyrateonecountrybasedontheirexpertise,itisnecessary

toutilizelateralcoders.Aspreviouslydescribed,thesecodersratemultiplecountriesfora

limitedtimeperiod(mostlyoneyear,butinsomecasesten).Wehaveatpresentsome350

lateralcoders.Inaddition,wehaveover390bridgecoders,asdiscussedabove.Theseare

coderswho code the full time series (generally 1900-2016) formore than one country,

coveringoneormoreareas (“surveys”).17 Essentially, this codingprocedureallowsus to

mitigate the incomparability of coders’ thresholds and the problem of cross-national

estimates’calibration(Pemsteinetal.2017).Whilehelpfulinthisregard,ourtestsindicate

that,giventhesparsityofourdata,eventhisextensivebridge-codingisnotsufficienttofully

solvecross-nationalcomparabilityissues.Wethereforeemployadata-collapsingprocedure.

At its core, this procedure relies on the assumption that as long as noneof the experts

changetheirratings(ortheirconfidenceabouttheirratings)foragiventimeperiod,wecan

treat the country-years in this period as one year. The results of our statistical models

indicate that this technique isextremelyhelpful in increasing theweightgiven tobridge

coders,andthusfurtheramelioratescross-nationalcomparabilityproblems.

Asafinalnote,ourmodeldivergesfrommorestandardIRTmodelsinthatitemploys

empiricalpriors.Specifically,wemodelacountry-year’slatentscoreforagivenvariableas

beingdistributedaccordingtoanormaldistributionwithanappropriatelywidestandard

deviationparameterandameanequaltotherawmeanofthecountry’sscores,weighted

bycoderconfidenceandnormalizedacrossallcountry-years.Moreformally,𝑍"~𝑁(𝜇", 1),

where𝑍isthelatentscoreforcountry-year𝑖,and𝜇isthenormalizedconfidence-weighted

average from the raw data.18 In contrast,most standardmodels employ a vaguemean

17Thuswehave lateral/bridgecodingcovering theequivalentofover1,100“full coverage”ofall country-questions.

18Therearetwosetsofexceptionstoouruseofthenormalizedconfidence-weightedaverageofcoderscoresasempiricalpriors. First,wedonot includedata from lateral coders in thecomputationof theempiricalpriors. We exclude these data from this procedure because the purpose of lateral codings is to betterestimatethresholdsofexperts,notprovidedataregardingthespecificcountryyeartheyarelateralcoding.Inprinciple,excludingthesedatawillassistintheestimationoflateralcoders'thresholds,sinceitanchorstheir thresholds tocountry-yearvalues forwhichwehaveagreatdealofdata (i.e. lateral-codedcountry

32

estimate,i.e.𝑍"~𝑁(0,1).Ourapproachofusingempiricalpriorsissimilartothestandard

approach: our wide standard deviation parameter still allows for the model output to

divergefrompriorasthedatawarrant.However,ourapproachincorporatesouractualprior

beliefsaboutacountry’sscoreandthusyieldsmoreaccuratemeasures.Especially inthe

caseofcountrieswithextremevalues,atraditionalapproachrisksbiasingoutputtoward

themean.

FutureversionsofourordinalIRTmodelswillimproveoncurrentestimatesintwo

primaryways.First,hierarchicalIRTmodelingtechniques(Patzetal.2002,Mariano&Junker

2007)wouldallowustoborrowstrengthfromdifferentvariableestimates,yieldingmore

precisemeasuresofeachvariable.Second,allraterscompleteapost-surveyquestionnaire

thatasksdemographicandattitudinalquestions.Codersalsoreportpersonalassessments

of confidence in their responses to each question. At present, of these data we only

incorporateconfidenceintothemodel,usingittoweightourpriormeanestimates;further

useoftheseformsofdatainourmodelswillallowustoteaseoutpatternsconcerningbiases

andreliabilityacrossdifferent typesofexperts,andgenerally improve thequalityofour

estimates.

Wealsouseconceptually-similarIRTtechniqueswhensufficientvariationexiststo

identifyraterthresholdsfornominalandsomedichotomousexpert-codedvariables.Forthe

remainingvariablesweprovidetheunweightedmean.

AnchoringVignettes

V-Dem’sthree-prongedapproachtodealingwithDIF—usingIRTmodels,recruitingbridge

years).Second,weoffsetthecontributionofhistoricalcoders(i.e.coderswhocodeyearsbefore1900)andnewcoders (i.e. coderswhoonlycodeyearsafter2005) to theempiricalpriorby theaveragedifferencebetweenthesecodersandthosecoderswhocodedtheyears1900-2012inoverlapyears(i.e.thoseyearsboth these sets of coders and the full time period coders coded). More specifically, we determine theconfidence-weightedaveragescoreofthefull-timeperiodcodersforaspecificcountryintheoverlapyears,andsubtracttheequivalentaveragefornewcodersofthesamecountryfromthisvalue.Wethenaddthisdifference to the new coders' scores for a given country for when computing the prior (restricting theresultingvaluessuchthattheycannotexceedtherangeoftheordinaldata).Weusethesameprocedureforhistoricalcoders (i.e.wecomputeoffsets fornewandhistoricalcodersseparately).Thepurposeof theseoffsetsisasfollows.Expertswhocodedifferenttimeperiodsmayhavedifferentcognitivereferencepointsforlevelsoftheordinalscale,andthusprovidedifferentvaluesforthesamelatentconstructduetoDIF.Theoffsetsamelioratethisproblembyfixingthepriorforagivencountry-yeartoaconsistentreferencepoint,i.e. thescoresof thosecoders forwhomwehavethemostdata (thoseexpertswhocodedthe full timeperiod).

33

andlateralcoders,andemployingempiricalpriors—hadhelpedtoproduceadatasetthat

standsupwelltotestsofvalidity(McMann2016,McMannetal2016,Sigman&Lindberg

2015,Teorell,Coppedge,Skaaning&Lindberg2016).Nonetheless,thereremainsroomfor

improvement.ThereforeV-Demhas includedanchoringvignettes (King&Wand2007) in

surveywavesstarting in2015/2016.Anchoringvignettesaredescriptionsofhypothetical

casesthatprovideinformationnecessarytoansweragivensurveyquestion.Weaskcoders

toratevignettesforV-Demquestionsbecausepatternsofvariationinhowratersevaluate

these synthetic cases provides information about difference how coders translate their

perceptions about cases into ordinal ratings, providing another tool formeasuring, and

adjustingforDIF.

Wefieldedourfirstwaveofvignettesduringthe2015/2016update,presenting116

vignettes for 31V-Demquestions to 599 coders from94 countries.We followedwith a

secondwaveofvignettesduringthe2016/2017update,presenting224vignettesfor66V-

Demquestions to1400coders from174countries.Codersarenot required tocomplete

vignettes,butwererequested.

Vignettesprovidebridgingdatathatrequiresnospecificcaseknowledge,enabling

ustoobtainbridginginformationacrossraterswhoarenotqualifiedtocodethesameset

of real-world cases. They also ensure that coders are considering the same information

when evaluating cases, helping us to isolate the effect of DIF on raters’ codes.We are

studying a variety of methods for incorporating information from anchoring vignette

responsesintoourmodelingstrategy.Currentlywetreatthemlikeanyotherobservation

whenfittingmeasurementmodels,therebyusingthebridginginformationthattheyprovide

toimprovetheDIFadjustmentsproducedbyourIRTmodels.

IdentifyingRemainingErrors

Toevaluatepossibleerrorsweemployanumberoftests,someofwhichareincorporated

into themeasurementmodels and others of which are applied ex post to examine the

validityofmodeloutput.

First,wehaveuseddatafromthepost-surveyquestionnairethateveryV-Demcoder

completestoidentifypotentialsourcesofbias.Thissurveydelvesintofactorsofpossible

relevance to coder judgments, such as personal characteristics like sex, age, country-of-

34

origin,educationandemployment.ItalsoinquiresintoopinionsthatCountryExpertshold

about thecountry theyarecoding,asking themtoassignapoint scoreona0-100scale

summarizing the overall level of democracy in the country on January 1, 2012, using

whateverunderstandingofdemocracy they choose to apply.Weask the samequestion

about several prominent countries from around the world that embody varying

characteristics of democracy/autocracy. Finally, the questionnaire contains several

questionsintendedtoelicitthecoder’sviewsabouttheconceptofdemocracy.Wehaverun

extensivetestsonhowwellsuchindividual-levelfactorspredictscountry-ratingsbuthave

foundthattheonlyfactorconsistentlyassociatedwithcountry-ratingsiscountryoforigin

(with“domestic”codersbeingharsherintheirjudgments).Thisisalsotheindividual-level

characteristicincludedinthemeasurementmodelestimates.

Inthefuture,weneverthelessplantouseeachpieceofinformationfromthispost-

surveyquestionnairetohelpinformthemeasurementmodel,i.e.,toenhanceprecisionand

limit possible undetected biases. The measurement model will also take into account

information we can glean from the performance of the coders that might serve as an

indicationoftheirlevelofattentiveness,effort,andknowledge.Thisinformationincludes

inter-coder reliability (assessed at the coder level across all codings), self-reported

confidence(ineachcoding),numberofcountry-yearscoded(alltogether),codingchanges

(thenumberoftimesthatacoderchangestheircodingfromT-1toTrelativetoothercoders

forthatcountry/indicator,aggregatedacrossallcodings),timeontask(thenumberofhours

a coder is logged into the on-line system, discounted by the number of

country/indicator/yearss/hehascoded),accesses(thenumberoftimestheon-linesurvey

isaccessed),contacts(writingcommentsoraskingquestionsoftheV-Demteamthatare

non-logistical in nature), and response rate (assessed at the country level). (With the

exceptionofinter-coderreliability,theseelementshavenotyetbeenincludedinthemodel.)

Eachoftheaforementionedfeatureswillalsobetestedindependently.Thus,wewill

beabletoreportonwhether,andtowhatextent,eachoftheobservedandself-reported

featuresofthecodersaffectstheirratings.Inparticular,byincludinghierarchicalpriorsthat

depend on observed rater characteristics and behavior in our latent variable model

specifications—anapproachoftenreferredtoas“empiricalBayes”—wecanevaluatethe

extenttowhichsuchfeatureshelptoexplainraterbiasandreliability,whilesimultaneously

35

incorporatingthatinformationintoindicatorestimates.

Inaddition,wewillapplyseveralexpost tests toevaluatethequalityof thedata

emanatingfromthemeasurementmodel.Onesortoftestreliesonthedistributionofthe

data.Ifthedistributionofresponsesforaparticularcountry/indicator/yearisbi-modalwe

haveanobviousproblem:codersdisagreewildly.Thisalsomeansthatthepointestimate

fromthemeasurementmodelisunstable:achangeofcodingforanysinglecoder,orthe

additionofanewcoder,islikelytohaveabigimpactonthepointestimate.Disagreement

as registeredbyabi-modaldistributioncould representa situation inwhich the truth is

recalcitrant – presumably because available information about a topic is scarce and/or

contradictory.Oritcouldrepresenterrorsthatarecorrigible.

A secondapproach tovalidationcomparesV-Dem indiceswithother indices that

purport to measure similar concepts, i.e., convergent validity. For example, a set of

regressionsusingallavailabledataoftheV-DemElectoralDemocracy/PolyarchyIndex–and

some of its constituent indicators – against Polity2 indicates relatively high correlations

(Pearson’s r= .85) and (separately) against FH Political rights (Pearson’s r= .90).

Unfortunately,techniquesofconvergentvalidityarelimitedintheirutility.First,wehave

some doubts about the validity of standard indices (see Comparisons and Contrasts).

Second, standard indices tend to hover at a higher level of aggregation, thus impairing

comparability betweenV-Dem indices andalternative indices. Indeed, only a fewextant

indicesarecloseenoughinconceptionandconstructiontoprovideanopportunityfordirect

corroborationwithV-Demindices.

A third approach to validation focuses on face validity. Once data collection is

completeforagroupofcountries,RegionalManagersandothermembersoftheV-Dem

teamlookcloselyatpointestimatesinanattempttodeterminewhethersystematicbias

mayexist.OnemajorsuchreviewwasconductedinOctober2013whenalmostallRegional

Managers,allProjectManagers,ResearchFellows,PIsandstaff,spentfourdayscollectively

reviewingalldatacollatedatthatpointtovalidatetheapproachandaggregationmethods.

Theprocessoffacevaliditycheckshassincethenbeenrecurrent.

CorrectingErrors

We correct problems with factual questions (B-type indicators) whenever the Principal

36

Investigators,inconsultationwiththerelevantProjectManagers,becomeconvincedthata

better (i.e., more correct) answer is available. Based on analysis of submitted data by

CountryCoordinators,certainvariablesweredesignatedasB+A.UsingtheoriginalB-data

as a point of departure and cross-checking with external resources, we designed and

implemented a coding scheme to re-code these indicators, as the Codebook describes.

Indicatorsaffectedincludeallindicatorsfromthedirectdemocracysurvey,fourindicators

ontheexecutive,fouronelectionsandnineonlegislature.Thedecisiontore-assignthese

indicators was also due to the interaction between question formulation and coder

interpretation, e.g. in some instances the meaning of “plebiscite” was interpreted in a

differentwaythanwhattheProjectManagerenvisaged,leadingtodiscrepanciesincoding.

Wehandleproblemswithevaluativequestions(C-typeindicators)withrestraint.We

fullyexpectthatanyquestionrequiringjudgmentwillelicitarangeofanswers,evenwhen

allcodersarehighlyknowledgeableaboutasubject.AkeyelementoftheV-Demproject–

settingitapartfrommostotherindicesthatrelyonexpertcoding–iscoderindependence:

eachcoderdoesherworkinisolationfromothercodersandmembersoftheV-Demteam

(apart fromclarifyingquestions about theprocess). Thedistributionof responses across

questions, countries, and years thus provides vital insight into the relative

certainty/uncertaintyofeachdatapoint.SinceaprincipalgoaloftheV-Demprojectisto

produceinformativeestimatesofuncertaintywedonotwishtotamperwithevidencethat

contributestothoseestimates.Arguably,thenoiseinthedataisasinformativeasthesignal.

Moreover,waywardcoders(i.e.,coderswhodivergefromothercoders)areunlikelytohave

a strong influence on the point estimates that result from the measurement model’s

aggregationacrossfiveormorecoders.Thisisespeciallythecaseifthewaywardcodersare

consistentlyoff-center (acrossall their codings); in this case, theirweight indetermining

measurementmodelscoresisreduced.

That said, there have been instances in which we have altered C-data. A few

questions were largely of factual nature (e.g. number of legislative chambers; if a local

governmentexists,whichofficeswereelectedinaparticularelection,etc.).Sincewelater

acquiredenoughfundingtohaveassistantsconductthefactualcodingbasedonsystematic

consultationofcrediblesources,wedischargedthedatasubmittedbyCountryExpertsfor

theseparticularquestionsandany“downstream”data.Forexample, ifaCountryExpert

37

indicatedthat thereweretwochambers in the legislature foraparticularyear,shethen

coded“downstream”inthequestionnaireaseriesofquestionsregardingboththelower

andupperchamber.Ifourresearchestablishedthatanupperchamberdidnotinfactexist

inthatparticularyear,wecleanedtherecordsofdataprovidedbytheexpertfortheupper

chamber.Thiscleaningaffected19%ofallexecutivedatasubmittedforthosedownstream

variables,7.7%ofthedataintheelectionsurveyand11%inthelegislativesurvey,forthe

initial data collection effort covering 1900 to 2012. These numbers reflect placeswhere

coders unnecessarily coded due either to a) problem with the skipping function in the

surveys,b)coders’ability tochangethepre-coded, factualdata,orc)an initialdecision,

subsequently reversed, to have Country Experts to answer some of the A-coded (more

factual) questions. After improving the coding interfaces and making it impossible for

country experts to change such factual pre-coded data during the coding during later

updates,theneedforsuch“downstream”cleaninghasbeenreducedtoclosetonil.

Inafinalcase,weremovedoriginalcodingbysomeCountryExpertsbecauseofa

factual misunderstanding (or misunderstanding about response-categories) about the

existenceoftheinternetineraspriortoitsinvention.

Inallthesesituations,wemaintaintheoriginalcoder-leveldatainarchivedfilesthat

mayberetrievedbyspecialrequestofthePIs.

VersionsofC-Variables

TheV-DemdatasetthencontainsA,B,C,andDindicatorsthatareallunique.Inaddition,to

facilitateeaseofuseforvariouspurposes, theC-variablesaresupplied inthreedifferent

versions(alsonotedintheV-DemCodebook):

1.“RelativeScale”-MeasurementModelOutput–hasnospecialsuffix(e.g.v2elmulpar).

This versionof thevariablesprovides country-year (country-date in thealternative

dataset)pointestimatesfromtheV-Demmeasurementmodeldescribedabove.The

pointestimatesarethemedianvaluesofthesedistributionsforeachcountry-year.

The scale of ameasurementmodel variable is similar to a normal (“Z”) score (i.e.

typically between -5 and 5, with 0 approximately representing the mean for all

country-years in the sample) though it does not necessarily follow a normal

distribution.Formostpurposes,thesearethepreferredversionsofthevariablesfor

38

time-seriesregressionandotherestimationstrategies.

“Measure of Uncertainty” –MeasurementModel Highest Posterior Density (HPD)

Intervals–havethesuffixes–"codelow"and"codehigh"(e.g.,v2elmulpar_codelow

and v2elmulpar_codehigh). These two variables demarcate one standard deviation

upperandlowerboundsoftheintervalinwhichthemeasurementmodelplaces68

percent of the probabilitymass for each country-year score. The spread between

"codelow" and "codehigh" is equivalent to a traditional one standard deviation

confidence interval; a larger range indicates greater uncertainty around the point

estimate.

2.“OriginalScale”–LinearizedOriginalScalePosteriorPrediction–hasthesuffix“_osp,”

(e.g.v2elmulpar_osp).Inthisversionofthevariables,wehavelinearlytranslatedthe

measurementmodelpointestimatesbacktotheoriginalordinalscaleofeachvariable

(e.g. 0-4 for v2elmulpar_osp) as an interval measure.19 The decimals in the _osp

version indicate the distance between the point estimate from the linearized

measurementmodelposteriorpredictionandthethresholdforreachingthenextlevel

on the original ordinal scale. Thus, a_osp value of 1.25 indicates that themedian

measurementmodelposteriorpredictedvaluewascloser to theordinalvalueof1

than2ontheoriginalscale.Sincethereisnoconventionaltheoreticaljustificationfor

linearlymappingordinalposteriorpredictionsontoan intervalscale,20 thesescores

shouldprimarilybeusedforheuristicpurposes.However,sincethe_ospversionmaps

ontothecodingcriteriafoundintheV-DemCodebook,andisstronglycorrelatedwith

theMeasurementModeloutput(typicallyat.98orhigher),someusersmayfindthe

_osp version useful in estimating quantities such as marginal effects with a clear

substantiveinterpretation.Usingthe“OrdinalScale”estimates---orincorporatingthe

19 More specifically, we use the measurement model to estimate the posterior distribution around thepredictedprobabilitythatatypicalcoderwouldplaceacountry-yearestimateateachleveloftheoriginalcodebook scale. We then linearly map these predicted probability distributions onto the original scale,producingadistributionofinterval-valuedscoresontheoriginalcodebookscaleforeachcountry-year.

20ThemaintheoreticalandpragmaticconcernwiththesedataisthatthetransformationdistortsthedistancebetweenpointestimatesintheMeasurementModeloutput.Forexample,thedistancebetween1.0and1.5inthe_ospdataisnotnecessarilythesameasthedistancebetweena1.5and2.0.

39

properties of ordinal probit models into the estimation procedure---is generally

preferabletousingthe_ospestimatesinstatisticalanalyses.Thatsaid,ifauseruses

_ospdatainstatisticalanalysesitisimperativethatshefirstconfirmthattheresults

arecompatiblewithestimationsusingMeasurementModeloutput.

“MeasureofUncertainty”–LinearizedOriginalScaleHPDIntervals–havethesuffixes

– "codelow" and "codehigh" (e.g., v2elmulpar_osp_codelow and

v2elmulpar_osp_codehigh).Weestimatethesequantitiesinasimilarmannerasthe

Measurement Model Highest Posterior Density Intervals. They demarcate one

standarddeviationupperandlowerboundsoftheintervalinwhichthemeasurement

model places 68 percent of the probabilitymass for each country-year score. The

spreadbetween"codelow"and"codehigh"isequivalenttoatraditionalonestandard

deviationconfidenceinterval;alargerrangeindicatesgreateruncertaintyaroundthe

pointestimate.

3.“OrdinalScale”-MeasurementModelEstimatesofOriginalScaleValue–hasthesuffix

"_ord" (e.g. v2elmulpar_ord). This method translates the measurement model

estimates back to the original ordinal scale of a variable (as represented in the

Codebook) after taking coder disagreement andmeasurement error into account.

Moreprecisely, itrepresentsthemost likelyordinalvalueontheoriginalcodebook

scale intowhichacountry-yearwould fall, given theaveragecoder’susageof that

scale. Specifically, we assign each country-year a value that corresponds to its

integerizedmedianordinalhighestposteriorprobabilitycategoryoverMeasurement

Modeloutput.

“MeasureofUncertainty”–OriginalScaleValueHPDIntervals–havethesuffixes–

"codelow" and "codehigh" (e.g., v2elmulpar_ord_codelow and

v2elmulpar_ord_codehigh). We estimate these values in a similar manner as the

Measurement Model Highest Posterior Density Intervals. They demarcate one

standarddeviationupperandlowerboundsoftheintervalinwhichthemeasurement

model places 68 percent of the probabilitymass for each country-year score. The

40

spreadbetween"codelow"and"codehigh"isequivalenttoatraditionalonestandard

deviationconfidenceinterval;alargerrangeindicatesgreateruncertaintyaroundthe

pointestimate.

AdditionalPossibilitiesforIdentifyingSourcesofMeasurementErrorintheFuture

A final approach to validationanalyzes various featuresof thedatagatheringprocess in

ordertogaugepossiblesourcesoferror.Thisanalysistakestheformofvariousstudiesin

whichaparticularissueisprobedinanintensivefashion.Thefollowingstudiesareunderway

oronthedrawingboard–thoughwecannotsayforsurehowlongitwilltakeustocomplete

them.

Onesuchstudywillfocusoncodertypes.Akeychallengetothevalidityisthatdata

maybesubjecttothesubjectiveperceptionsandopinionsofthechosencoders. Is itthe

casethatadifferentsetofcodersmightarriveataverydifferentsetofanswers?Features

ofthecoderscapturedinourpost-surveyquestionnairecanbetestedsystematicallyacross

theentiredataset,asnoted.However,wecannottestthepotential impactofadifferent

kindofcodernotincludedinourusualsample.Thisstudythereforefocusesoncomparisons

across different coder types, e.g., partisans, academics, civil society professionals,

businesspeople, cosmopolitans (those speaking foreign languages and with travel or

educationalexperienceabroad),educatedlaycitizens,andlesseducatedlaycitizens.Results

ofthisstudyshouldindicate(a)howfartheconsensusoncodingextends(i.e.,towhattypes

ofcoders),(b)howmuchdifferencethebackgroundofthecodermakes,(c)forwhattypes

of questions itmatters, and (d)which sorts of coders have themost positive view of a

country.Moregenerally,wehopetolearnmoreaboutthesensitivityofV-Demdatatoour

samplingofCountryExperts.

Asecondstudywouldbetofocusoncountrysequencing.Doesitmatterifcoders

have considered other countries prior to coding CountryA? Such a studywould involve

randomizing respondents into two groups. Group 1 is asked to code Country A. Several

weekslater,theyareaskedtocodeahandfulofcountriesincludingCountryA,whichthey

mustre-code.Thecomparisoncasesshouldincludethosethatareinthesameregionaswell

asacountry(preferablyinthesameregion,orwithahistoryofcolonialinvolvementinthe

region) generally regarded as highly democratic. Respondents arenot remindedof their

41

originalcodingsforCountryAandareencouragedtoadjusttheiroriginalcodingiftheyfeel

that a more accurate assessment is possible, in light of their consideration of other

countries.Group2repeatsthisprocedureinreverse.That is,theyfirstcodeahandfulof

relatedcountriesandthenareaskedtocodeCountryA.

Athirdstudywouldbetofocusonquestionordering.TheV-Demquestionnaireis

not randomized for several reasons. First, somequestionsmust be asked in a particular

order(laterquestionsareactivatedorskippeddependingupontheanswers).Second,we

wish tomaintain a logical flowacrossquestions and tomake the flowaspredictable as

possible,sothatinadvertenterrorsareminimized.Finally,wewishtomaintainequivalence

acrosssurveys.However,onemayalsowishtoknowwhethertheorderingofquestionson

thequestionnaireaffectsresponses,andifsohow.Toprobethisquestiononewouldhave

to randomize questions within a survey (but not across surveys), without upsetting

questionsthataredependentuponothers,andwhilemaintainingsomedegreeoflogical

flow.Forexample,wewillreversetheorderofquestionsthatareaskedfirstaboutmenand

nextaboutwomen.

A fourth study could explore the quality of model-based bias adjustment. In

particular,becausecodersfromdifferentcountriesmayunderstandbothquestionwordings

andconceptsindifferentways,twocodersoperatingindifferentcontextsmightratetwo

identical cases differently from one another. A common approach to addressing this

problemistoconstructanchoringvignettes—shorthypotheticaldepictionsofcases—and

thenaskcoders toevaluatevignettes inaddition to realcases,and tousedifferences in

vignette evaluations to correct for inter-personal differences in coder perceptions or

understandingsofconcepts (Kinget.al.2004;King&Wand2007;Hopkins&King2010).

Because the vignettes are fixed, these techniques assume that differences in rater

evaluationsmustrepresentdifferencesinpersonalinterpretation,andthensubtractthese

differences from responses for real cases, ostensibly correcting for respondent

incomparability. Similarly, given sufficient overlap in observed coding across raters, our

latentvariablemodelingtechniquescanusepatternsofinter-coderagreementtoidentify

andcorrectforsystematicdifferencesinraters'perceptionsandconceptualunderstandings.

Inotherwords,differencesinhowexpertsrateidenticalcaseshelptoidentifyinter-expert

variationininterpretationinmuchthesamewaythatvariationinratingsoffixedvignettes

42

does.Wecanvalidatethisfeatureofthemodelbycomparingitsperformancetoavignette-

basedapproachforcontrollingincomparabilityinsurveyresponses.Focusingonasubsetof

indicators, we would recruit country-experts to rate an anchoring vignette, their own

country,andsomecomparisoncountries.Thenwewouldapplybothvignette-basedand

measurement-model based corrections to responses to determine if they produce

comparableresults. Anexperimentalcomponentcanalsoseektodetermine ifvignettes

themselvesaltercoderbehavior.Inparticular,wecouldusepatternsofagreementbetween

raters to determine if treated experts (vignette condition) produce codings that are

systematicallydifferentfromacontrolpopulation(novignettecondition).

43

References

Almond,GabrielA.,SidneyVerba.1963/1989.TheCivicCulture:PoliticalAttitudesandDemocracyinFiveNations.NewburyPark,CA:Sage.

Bernhard,Michael,EitanTzelgov,Dong-JoonJung,MichaelCoppedge,&StaffanI.Lindberg.2015.TheVarietiesofDemocracyCoreCivilSocietyIndex.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPaperSeries,No.12.

Bernhard,Michael,ChristopherReenock,andTimothyNordstrom.2004.“TheLegacyofWesternOverseasColonialismonDemocraticSurvival.”InternationalStudiesQuarterly48(3),225-250.

Bollen,KennethA.,PamelaPaxton.2000.“SubjectiveMeasuresofLiberalDemocracy.”ComparativePoliticalStudies33(1):58–86.

Capoccia,Giovanni,DanielZiblatt.2010.“TheHistoricalTurninDemocratizationStudies:ANewResearchAgendaforEuropeandBeyond.”ComparativePoliticalStudies43(8-9):931-968.

Clinton,JoshuaD.,DavidLewis.2008.“ExpertOpinion,AgencyCharacteristics,andAgencyPreferences.”PoliticalAnalysis16(1):3–20.

Clinton,JoshuaD.,JohnS.Lapinski.2006.“MeasuringLegislativeAccomplishment,1877-1994.”AmericanJournalofPoliticalScience50(1):232–249.

Collier,DavidandJamesMahon(1993).“Conceptual‘Stretching’Revisited:AdaptingCategoriesinComparativeAnalysis.”AmericanPoliticalScienceReview87(4):845-855.

Coppedge,Michael,StaffanLindberg,Svend-ErikSkaaning,andJanTeorell.2015.MeasuringHighLevelDemocraticPrinciplesusingtheV-DemData.UniversityofGothenburg,TheVarietiesofDemocracyInstitute:V-DemWorkingPaperseriesNo.6

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,DavidAltman,MichaelBernhard,M.StevenFish,AdamGlynn,AllenHicken,CarlHenrikKnutsen,KyleMarquardt,KellyMcMann,FarhadMiri,PamelaPaxton,DanielPemstein,JeffreyStaton,EitanTzelgov,Yi-tingWang,andBrigitteZimmerman.2015.V-Dem[Country-Year/Country-Date]Datasetv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,withDavidAltman,MichaelBernhard,M.StevenFish,AdamGlynn,AllenHicken,CarlHenrikKnutsen,KellyMcMann,PamelaPaxton,DanielPemstein,JeffreyStaton,BrigitteZimmerman,FridaAndersson,ValeriyaMechkova,andFarhadMiri.2015.V-DemCodebookv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,KyleMarquardt,ValeriyaMechkova,FarhadMiri,DanielPemstein,JosefinePernes,NataliaStepanova,EitanTzelgov,andYi-tingWang.2015.V-DemMethodologyv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,andVladCiobanu.2015.V-DemCountryCodingUnitsv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,ValeriyaMechkova,JosefinePernes,andNataliaStepanova.2015.V-DemOrganizationandManagementv5.VarietiesofDemocracy(V-Dem)Project.

44

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,andJanTeorell.2015.V-DemComparisonsandContrastswithOtherMeasurementProjects.VarietiesofDemocracy(V-Dem)Project.

Dahl,RobertA.1971.Polyarchy:ParticipationandOpposition.NewHaven:YaleUniversityPress.

Dahl,RobertA.1989.DemocracyanditsCritics.NewHaven:YaleUniversityPress.Epstein,DavidL.;RobertBates;JackGoldstone;IdaKristensen;SharynO’Halloran.2006.

“DemocraticTransitions.”AmericanJournalofPoliticalScience50(3):551-569.Fox,Jean-Paul.2010.BayesianItemResponseModeling:TheoryandApplications.New

York:Springer.Gallie,W.B.1956.“EssentiallyContestedConcepts.”ProceedingsoftheAristotelian

Society56:167–220.Gerring,John,PhilipBond,WilliamBarndt,andCarolaMoreno.2005.“Democracyand

Growth:AHistoricalPerspective.”WorldPolitics57(3):323–364.Goertz,Gary.2006.SocialScienceConcepts:AUser’sGuide.Princeton:Princeton

UniversityPress.Hadenius,AxelandJanTeorell.2005.“CulturalandEconomicPrerequisitesofDemocracy:

ReassessingRecentEvidence.”StudiesinComparativeInternationalDevelopment39(4):87–106.

Held,David.2006.ModelsofDemocracy,3ded.Cambridge:PolityPress.Hopkins,Daniel,andGaryKing.2010.“ImprovingAnchoringVignettes:DesigningSurveys

toCorrectInterpersonalIncomparability.”PublicOpinionQuarterly:1-22.Inglehart,RonaldandWelzel,Christian.2005.Modernization,CulturalChangeand

Democracy:TheHumanDevelopmentSequence.Cambridge:CambridgeUniversityPress.

Isaac,JeffreyC.n.d.“ThinkingAbouttheQualityofDemocracyanditsPromotion.”Unpublishedms.

Jackman,Simon.2004.“WhatDoWeLearnfromGraduateAdmissionsCommittees?AMultipleRater,LatentVariableModel,withIncompleteDiscreteandContinuousIndicators.”PoliticalAnalysis12(4):400–424.

Johnson,ValenE.,JamesH.Albert.1999.OrdinalDataModeling.NewYork:Springer.Junker,Brian1999.SomeStatisticalModelsandComputationalMethodsthatmaybe

UsefulforCognitively-RelevantAssessment.http://www.stat.cmu.edu/~brian/nrc/cfa/documents/final.pdf

King,Gary,ChristopherMurray,JoshuaA.Salomon,andAjayTandon.2004.“EnhancingtheValidityandCross-culturalComparabilityofMeasurementinSurveyResearch.”AmericanPoliticalScienceReview98(1):191–207.

King,Gary,andJonathanWand.2007.ComparingIncomparableSurveyResponses:NewToolsforAnchoringVignettes.PoliticalAnalysis15:46-66.

Knutsen,CarlHenrik.2010.“MeasuringEffectiveDemocracy.”InternationalPoliticalScienceReview31(2):109-128.

45

Knutsen,CarlHenrik,JørgenMøller,andSvend-ErikSkaaning.Forthcoming.GoingHistorical:MeasuringDemocraticnessbeforetheAgeofMassSuffrage.InternationalPoliticalScienceReview.

Lindberg,StaffanI.2015.OrdinalVersionsofV-Dem’sIndices:ForClassification,Description,SequencingAnalysisandOtherPurposes.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.19.

Lord,FredericM.,andMelvinNovick.1968.StatisticalTheoriesofMentalTestScores.Reading,MA:Addison-Wesley.

Mahoney,James,andDietrichRueschemeyer,eds.2003.ComparativeHistoricalAnalysisintheSocialSciences.Cambridge:CambridgeUniversityPress.

Mariano,LouisT.andBrianW.Junker.2007.“CovariatesoftheRatingProcessinHierarchicalModelsforMultipleRatingsofTestItems.”JournaloftheEducationalandBehavioralStatistics32(2):287-314.

McMann,Kelly.2016.“MeasuringSubnationalDemocracy.”VarietiesofDemocracyInstituteWorkingPaper26(March).

McMann,Kelly,DanielPemstein,BrigitteSeim,JanTeorell,andStaffanI.Lindberg.2016.“StrategiesofValidation:AssessingtheVarietiesofDemocracyCorruptionData.”VarietiesofDemocracyInstituteWorkingPaper23(February).

Munck,GerardoL.2009.MeasuringDemocracy:ABridgebetweenScholarshipandPolitics.Baltimore:JohnHopkinsUniversityPress.

Munck,GerardoL.2016.“WhatisDemocracy?AReconceptualizationoftheQualityofDemocracy.”Democratization23(1):1-26.

Nunn,Nathan.2009.“TheImportanceofHistoryforEconomicDevelopment.”AnnualReviewofEconomics1(1):1–28.

Patz,RichardJ.,andBrianW.Junker.1999.“AStraightforwardApproachtoMarkovChainMonteCarloMethodsforItemResponseModels.”JournalofEducationalandBehavioralStatistics24:146-178.

Patz,RichardJ.,BrianW.Junker,MatthewS.Johnson,andLouisT.Mariano.2002.“TheHierarchicalRaterModelforRatedTestItemsanditsApplicationtoLarge-ScaleEducationalAssessmentData.”JournalofEducationalandBehavioralStatistics27(4):341-384.

Pemstein,Daniel,KyleL.Marquardt,EitanTzelgov,Yi-tingWang,JoshuaKrusellandFarhadMiri.2017.“TheV-DemMeasurementModel:LatentVariableAnalysisforCross-NationalandCross-TemporalExpert-CodedData”.UniversityofGothenburg,VarietiesofDemocracyInstitute:WorkingPaperNo.21,2ndedition.

Pemstein,Daniel,StephenMeserve,andJamesMelton.2010.“DemocraticCompromise:ALatentVariableAnalysisofTenMeasuresofRegimeType.”PoliticalAnalysis18(4):426–449.

Pierson,Paul.2004.PoliticsinTime:History,Institutions,andSocialAnalysis.Princeton:PrincetonUniversityPress.

Rose-Ackerman,Susan.1999.CorruptionandGovernment:Causes,Consequences,andReform.Cambridge:CambridgeUniversityPress.

Sartori,Giovanni.1970.“ConceptMisformationinComparativePolitics.”AmericanPoliticalScienceReview64(4):1033-1053.

46

Schedler,Andreas.2012.“JudgmentandMeasurementinPoliticalScience.”PerspectivesonPolitics10:1,21-36.

Shapiro,Ian.2003.TheStateofDemocraticTheory.Princeton:PrincetonUniversityPress.Sigman,RachelandStaffanI.Lindberg.2015.TheIndexofEgalitarianDemocracyandIts

Components:V-Dem'sConceptualizationandMeasurement.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.21

Steinmo,Sven,KathleenThelen,andFrankLongstreth,eds.1992.StructuringPolitics:HistoricalInstitutionalisminComparativeAnalysis.Cambridge:CambridgeUniversityPress.

Teorell,Jan.2011.“OverTime,AcrossSpace:ReflectionsontheProductionandUsageofDemocracyandGovernanceData.”ComparativeDemocratization9:1(February)1,7.

Teorell,Jan,MichaelCoppedge,JohnGerring&StaffanLindberg.n.d.2016"MeasuringElectoralDemocracywithV-DemData:IntroducingaNewPolyarchyIndex.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.23

Teorell,Jan,RachelSigman,andStaffanI.Lindbergn.d.2016.V-DemIndices:RationaleandAggregations.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.22

Teorell,JanandStaffanI.Lindberg.2015.TheStructureoftheExecutiveinAuthoritarianandDemocraticRegimes:RegimeDimensionsacrosstheGlobe,1900-2014.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.5

Thomas,MelissaA.2010.“WhatDotheWorldwideGovernanceIndicatorsMeasure?”EuropeanJournalofDevelopmentResearch22(1):31–54.

Treier,Shawn,andSimonJackman.2008.“DemocracyasaLatentVariable.”AmericanJournalofPoliticalScience52(1):201–217.

Wang,Yi-ting,PatrikLindenfors,AkselSundström,FredrikJansson,andStaffanI.Lindberg.2015.NoDemocraticTransitionWithoutWomen’sRights:AGlobalSequenceAnalysis1900-2012.VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.12.

Wang,Yi-ting,AkselSundström,PamelaPaxton,andStaffanI.Lindberg.2015.”Women’sPoliticalEmpowermentIndex:ANewMeasure.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.18.

Welzel,Christian.2007.“AreLevelsofDemocracyAffectedbyMassAttitudes?TestingAttainmentandSustainmentEffectsonDemocracy.”InternationalPoliticalScienceReview28(4):397–424.

47

APPENDIXA:V-DemIndices,Components,andIndicatorsDemocracyIndicesNames

Mid-LevelDemocracyandGovernanceIndicesNames

Lower-LevelDemocracyandGovernanceIndicesNames

NamesIndicators Tag Uniquenessscore*

Electoraldemocracyindex

v2x_polyarchy

Additivepolyarchyindex

v2x_api

Multiplicativepolyarchyindex

v2x_mpi

Expandedfreedomofexpressionindex

v2x_freexp_thick

Governmentcensorshipeffort-Media

v2mecenefm0,287399531

Harassmentofjournalists v2meharjrn 0,310029642 Mediaself-censorship v2meslfcen 0,299229721 Mediabias v2mebias 0,252931835 Print/broadcastmediacritical v2mecrit 0,235389702 Print/broadcastmediaperspectives v2merange 0,264891589 Freedomofdiscussionformen v2cldiscm 0,234735887 Freedomofdiscussionforwomen v2cldiscw 0,175262473 Freedomofacademicandcultural

expressionv2clacfree

0,328359319 Freedomof

associationindex(thick)

v2x_frassoc_thick

Partyban v2psparban 0,314515483 Barrierstoparties v2psbars 0,252064401 Oppositionpartiesautonomy v2psoppaut 0,257866799 Electionsmultiparty v2elmulpar 0,368900132 CSOentryandexit v2cseeorgs 0,34364043 CSOrepression v2csreprss 0,391819395 Shareof

populationwithsuffrage

v2x_suffr

Percentofpopulationwithsuffrage v2elsuffrage

Cleanelectionsindex

v2xel_frefair

EMBautonomy v2elembaut 0,414401744 EMBcapacity v2elembcap 0,476390647 Electionvoterregistry v2elrgstry 0,479468817 Electionvotebuying v2elvotbuy 0,619782147 Electionothervotingirregularities v2elirreg 0,400394507 Electiongovernmentintimidation v2elintim 0,464657467 Electionotherelectoralviolence v2elpeace 0,67570601

48

Electionfreeandfair v2elfrfair 0,394150747 Electedofficials

index v2x_elecoff

Legislaturebicameral v2lgbicam

Lowerchamberelected v2lgello

Upperchamberelected v2lgelecup

Legislaturedominantchamber v2lgdomchm

HOSselectionbylegislatureinpractice

v2exaphos

HOSappointmentinpractice v2expathhs

HOGselectionbylegislatureinpractice

v2exaphogp

HOGappointmentinpractice v2expathhg

HOSappointscabinetinpractice v2exdfcbhs

HOGappointscabinetinpractice v2exdjcbhg

HOSdismissesministersinpractice v2exdfdmhs

HOGdismissesministersinpractice v2exdfdshg

HOS=HOG? v2exhoshog

Chiefexecutiveappointmentbyupperchamber

v2exapup

Chiefexecutiveappointmentbyupperchamberexplicitapproval

v2exapupap

Liberaldemocracyindex

v2x_libdem

Electoraldemocracyindex

v2x_polyarchy

Liberalcomponentindex

v2x_liberal

Equalitybeforethelawandindividuallibertyindex

v2xcl_rol

Rigorousandimpartialpublicadministration

v2clrspct0,510897097

Transparentlawswithpredictableenforcement

v2cltrnslw0,372337936

Accesstojusticeformen v2clacjstm 0,106424425 Accesstojusticeforwomen v2clacjstw 0,307894313 Propertyrightsformen v2clprptym 0,165920394 Propertyrightsforwomen v2clprptyw 0,385230089 Freedomfromtorture v2cltort 0,380794137 Freedomfrompoliticalkillings v2clkill 0,390809329 Freedomfromforcedlaborformen v2clslavem 0,090370553 Freedomfromforcedlaborfor

womenv2clslavef

0,370180991 Freedomofreligion v2clrelig 0,591125139 Freedomofforeignmovement v2clfmove 0,437467753

49

Freedomofdomesticmovementformen

v2cldmovem0,379460353

Freedomofdomesticmovementforwomen

v2cldmovew0,276608073

Judicialconstraintsontheexecutiveindex

v2x_jucon

Executiverespectsconstitution v2exrescon 0,556533058 Compliancewithjudiciary v2jucomp 0,334738738 Compliancewithhighcourt v2juhccomp 0,354794616 Highcourtindependence v2juhcind 0,455176953 Lowercourtindependence v2juncind 0,434174586 Legislative

constraintsontheexecutiveindex

v2xlg_legcon

Legislaturequestionsofficialsinpractice

v2lgqstexp0,496293669

Executiveoversight v2lgotovst 0,386240561 Legislatureinvestigatesinpractice v2lginvstp 0,25393755 Legislatureoppositionparties v2lgoppart 0,402224961

Deliberativedemocracyindex

v2x_delibdem

Electoraldemocracyindex

v2x_polyarchy

Deliberativecomponentindex

v2xdl_delib

Reasonedjustification v2dlreason 0,013722755 Commongood v2dlcommon 0,018259475 Respectcounterarguments v2dlcountr 0,013655074 Rangeofconsultation v2dlconslt 0,012409352 Engagedsociety v2dlengage 0,012246084

EgalitariandemocracyIndex

v2x_egaldem

Electoraldemocracyindex

v2x_polyarchy

Egalitariancomponentindex

v2x_egal

Equalprotectionindex

v2xeg_eqprotec Socialclassequalityinrespectfor

civillibertiesv2clacjust

0,251630803 Socialgroupequalityinrespectfor

civilliberties v2clsocgrp 0,673137622 Weakercivillibertiespopulation v2clsnlpct 0,819008129

50

Equalaccessindex

v2eg_eqaccess

Powerdistributedbygender v2pepwrgen 0,491661585 Powerdistributedby

socioeconomicpositionv2pepwrses

0,491661585 Powerdistributedbysocialgroup v2pepwrsoc 0,4445505 Equal

distributionofresourcesindex

v2xeg_eqdr Means-testedvs.universalistic v2dlunivl 0,636117818

Encompassingness v2dlencmps 0,693676915 Educationalequality v2peedueq 0,243294135 Healthequality v2pehealth 0,171167797

Participatorydemocracyindex

v2x_partipdem

Electoraldemocracyindex

v2x_polyarchy

Participatorycomponentindex

v2x_partip

Civilsocietyparticipationindex

v2x_cspart

Candidateselection--National/local v2pscnslnl 0,758570546 CSOconsultation v2cscnsult 0,400943859 CSOparticipatoryenvironment v2csprtcpt 0,415201758 CSOwomensparticipation v2csgender 0,673605234 Directpopular

voteindex v2xdd_dd

Initiativespermitted v2ddlexci

Initiativessignatures% v2ddsigpci

Initiativessignature-gatheringtimelimit v2ddsiglci

Initiativessignature-gatheringperiod v2ddsigdci

Initiativeslevel v2ddlevci

Initiativesparticipationthreshold v2ddpartci

Initiativesapprovalthreshold v2ddapprci

Initiativesadministrativethreshold v2ddadmci

Initiativessupermajority v2ddspmci

Occurrenceofcitizen-initiativethisyear v2ddyrci

Referendumspermitted v2ddlexrf

Referendumssignatures% v2ddsigprf

Referendumssignature-gatheringperiod v2ddsigdrf

Referendumsparticipationthreshold v2ddpartrf

Referendumsapprovalthreshold v2ddapprrf

51

Referendumssupermajority v2ddspmrf

Referendumsadministrativethreshold v2ddadmrf

Occurrenceofreferendumthisyear v2ddyrrf

Plebiscitepermitted v2ddlexpl

Plebisciteparticipationthreshold v2ddpartpl

Plebisciteapprovalthreshold v2ddapprpl

Plebiscitesupermajority v2ddspmpl

Plebisciteadministrativethreshold v2ddadmpl

Occurrenceofplebiscitethisyear v2ddyrpl

Constitutionalchangespopularvote v2ddlexor

Obligatoryreferendumparticipationthreshold v2ddpartor

Obligatoryreferendumapprovalthreshold v2ddappor

Obligatoryreferendumsupermajority v2ddspmor

Obligatoryreferendumadministrativethreshold v2ddadmor

Occurrenceofobligatoryreferendumthisyear v2ddyror

Obligatoryreferendumcrediblethreat v2ddthreor

Popularreferencumcrediblethreat v2ddthrerf

Plebiscitecrediblethreat v2ddthrepl

Localgovernmentindex

v2xel_locelec

Localgovernmentelected v2ellocelc

Localofficesrelativepower v2ellocpwr

Localgovernmentexists v2ellocgov

Regionalgovernmentindex

v2xel_regelec

Regionalgovernmentelected v2elsrgel

Regionalofficesrelativepower v2elrgpwr

Regionalgovernmentexists v2elreggov

OtherDemocracyandGovernanceIndices

Names OtherLower-LevelDemocracyandGovernanceIndicesNames

NamesIndicators Tag

Womenpoliticalempowermentindex

v2x_gender

Womencivillibertiesindex

v2x_gencl

Freedomofdomesticmovementforwomen

v2cldmovew0,458950844

52

Freedomfromforcedlaborforwomen

v2clslavef0,49317791

Propertyrightsforwomen v2clprptyw 0,433149229 Accesstojusticeforwomen v2clacjstw 0,362024191 Womencivil

societyparticipationindex

v2x_gencs Freedomofdiscussionforwomen v2cldiscw 0,51935166

CSOwomensparticipation v2csgender 0,247076218 Percent(%)femalejournalists v2mefemjrn 0,512661832 Women

politicalparticipationindex

v2x_genpp Powerdistributedbygender v2pepwrgen

Lowerchamberfemalelegislators v2lgfemleg

Electoralregimeindex

v2x_elecreg

Legislativeorconstituentassemblyelection

v2xel_elecparl

v2eltype v2eltype_0

v2eltype v2eltype_1

v2eltype v2eltype_4

v2eltype v2eltype_5

Legislaturecloseddownoraborted

v2xlg_leginter

Legislaturebicameral v2lgbicam

Presidentialelection

v2xel_elecpres

v2eltype v2eltype_6

v2eltype v2eltype_7

Chiefexecutivenolongerelected

v2x_hosinter

HOS=HOG? v2exhoshog

HOGappointmentinpractice v2expathhg

HOSappointmentinpractice v2expathhs

Presidentialelectionaborted

v2x_hosabort

HOS=HOG? v2exhoshog

HOGappointmentinpractice v2expathhs

HOSappointmentinpractice v2expathhg

Presidentialelection v2xel_elecpres

53

Legislativeorconstituentassemblyelectionaborted

v2x_legabort

Legislaturebicameral v2lgbicam

Legislativeorconstituentassemblyelection

v2xel_elecparl

Executiveregimeindex

v2xex_elecreg

Presidentialelection

v2xel_elecpres

v2eltype v2eltype_6

v2eltype v2eltype_7

Chiefexecutivenolongerelected

v2x_hosinter

HOS=HOG? v2exhoshog

HOGappointmentinpractice v2expathhg

HOSappointmentinpractice v2expathhs

Presidentialelectionaborted

v2x_hosabort

HOS=HOG? v2exhoshog

HOGappointmentinpractice v2expathhs

HOSappointmentinpractice v2expathhg

Presidentialelection v2xel_elecpres

Legislativeelectoralregimeindex

v2xlg_elecreg

Legislativeorconstituentassemblyelection

v2xel_elecparl

v2eltype v2eltype_0

v2eltype v2eltype_1

v2eltype v2eltype_4

v2eltype v2eltype_5

Legislaturecloseddownoraborted

v2xlg_leginter

Legislaturebicameral v2lgbicam

Legislativeorconstituentassemblyelectionaborted

v2x_legabort

Legislaturebicameral v2lgbicam

Legislativeorconstituentassemblyelection

v2xel_elecparl

54

Corruptionindex v2x_corr

Legislaturecorruptactivities v2lgcrrpt

Judicialcorruptiondecision v2jucorrdc

Publicsectorcorruptionindex

v2x_pubcorr

Publicsectorcorruptexchanges v2excrptps 0,287505343 Publicsectortheft v2exthftps 0,186094553 Executive

corruptionindex

v2x_execorr

Executivebriberyandcorruptexchanges

v2exbribe0,339071586

Executiveembezzlementandtheft v2exembez 0,204393329

Electoralcomponentindex

v2x_EDcomp_thick

Freedomofassociationindex(thick)

v2x_frassoc_thick

Partyban v2psparban 0,314515483 Barrierstoparties v2psbars 0,252064401 Oppositionpartiesautonomy v2psoppaut 0,257866799 Electionsmultiparty v2elmulpar 0,368900132 CSOentryandexit v2cseeorgs 0,34364043 CSOrepression v2csreprss 0,391819395 Shareof

populationwithsuffrage

v2x_suffr

Percentofpopulationwithsuffrage v2elsuffrage

Cleanelectionsindex

v2xel_frefair

EMBautonomy v2elembaut 0,414401744 EMBcapacity v2elembcap 0,476390647 Electionvoterregistry v2elrgstry 0,479468817 Electionvotebuying v2elvotbuy 0,619782147 Electionothervotingirregularities v2elirreg 0,400394507 Electiongovernmentintimidation v2elintim 0,464657467 Electionotherelectoralviolence v2elpeace 0,67570601 Electionfreeandfair v2elfrfair 0,394150747 Electedofficials

index v2x_elecoff

Legislaturebicameral v2lgbicam

Lowerchamberelected v2lgello

Upperchamberelected v2lgelecup

Legislaturedominantchamber v2lgdomchm

55

HOSselectionbylegislatureinpractice

v2exaphos

HOSappointmentinpractice v2expathhs

HOGselectionbylegislatureinpractice

v2exaphogp

HOGappointmentinpractice v2expathhg

HOSappointscabinetinpractice v2exdfcbhs

HOGappointscabinetinpractice v2exdjcbhg

HOSdismissesministersinpractice v2exdfdmhs

HOGdismissesministersinpractice v2exdfdshg

HOS=HOG? v2exhoshog

Chiefexecutiveappointmentbyupperchamber

v2exapup

Chiefexecutiveappointmentbyupperchamberexplicitapproval

v2exapupap

Freedomofexpressionindex

v2x_freexp

Governmentcensorshipeffort-Media

v2mecenefm0,287568664

Harassmentofjournalists v2meharjrn 0,309101492 Mediaself-censorship v2meslfcen 0,324996159 Freedomofdiscussionformen v2cldiscm

Freedomofdiscussionforwomen v2cldiscw Freedomofacademicandcultural

expressionv2clacfree

0,299181418

Civillibertiesindex v2x_civlib

Privatecivillibertiesindex

v2x_clpriv

Freedomfromforcedlaborformen v2clslavem 0,090370553 Freedomfromforcedlaborfor

women v2clslavef 0,370180991 Propertyrightsformen v2clprptym 0,165920394 Propertyrightsforwomen v2clprptyw 0,385230089 Freedomofforeignmovement v2clfmove 0,381455761 Freedomofdomesticmovement

formen v2cldmovem 0,422904829 Freedomofdomesticmovement

forwomen v2cldmovew 0,422904829 Freedomofreligion v2clrelig 0,462254159 Religiousorganizationrepression v2csrlgrep 0,486998785 Physical

violenceindex v2x_clphy

Freedomfrompoliticalkillings v2clkill 0,239802867 Freedomfromtorture v2cltort 0,268822617

56

Politicalcivillibertiesindex

v2x_clpol

Governmentcensorshipeffort-Media v2mecenefm 0,32336033

Harassmentofjournalists v2meharjrn 0,336992175 Mediaself-censorship v2meslfcen 0,352861442 Freedomofdiscussionformen v2cldiscm 0,300186907 Freedomofdiscussionforwomen v2cldiscw 0,300186907 Freedomofacademicandcultural

expression v2clacfree 0,327849576 Partyban v2psparban 0,396406162 Barrierstoparties v2psbars 0,3333971 Oppositionpartiesautonomy v2psoppaut 0,341278346 CSOentryandexit v2cseeorgs 0,280345895 CSOrepression v2csreprss 0,291005396

Partyinstitutionalizationindex

v2xps_party

Partyorganizations v2psorgs 0,287962276 Partybranches v2psprbrch 0,286459634 Partylinkages v2psprlnks 0,838590265 Distinctpartyplatforms v2psplats 0,48433842 Legislativepartycohesion v2pscohesv 0,835063838

Corecivilsocietyindex

v2xcs_ccsi

CSOentryandexit v2cseeorgs 0,204224585 CSOrepression v2csreprss 0,327449254 CSOparticipatoryenvironment v2csprtcpt 0,478016631

Alternativesourcesofinformationindex

v2xme_altinf

Mediabias v2mebias 0,290619919 Print/broadcastmediacritical v2mecrit 0,265648013 Print/broadcastmediaperspectives v2merange 0,253035027 Dividedpartycontroloflegislatureindex

v2x_divparctrl

Nationalpartycontrol v2psnatpar Nationalpartycontrolordinal

version v2psnatpar_ord Divisionofpower

index v2x_feduni

57

Localgovernmentexists v2ellocgov

Regionalgovernmentexists v2elreggov

Localgovernmentelected v2ellocelc

Regionalgovernmentelected v2elsrgel

Localofficesrelativepower v2ellocpwr

Regionalofficesrelativepower v2elrgpwr

*Uniquenessisthevariancethatis‘unique’tothevariableandnotsharedwithothervariables.

**BecausetheaccountabilityindicesdonotusestandardBFAs,wedonotreportuniquenessscoreshere.

DetailsregardingmodelfitcanbefoundinthemethodologicalappendixofV-DemWorkingPaperNo.46