CHAPTER 7 Public Opinion Surveys: A New Scale

Bruno Castanho Silva, University of Cologne1 Ioannis Andreadis, Aristotle University of Thessaloniki

Eva Anduiza, Autonomous University of Barcelona

Nebojša Blanuša, University of Zagreb

Yazmin Morlet Corti, National Autonomous University of Mexico

Gisela Delfino, Argentine Catholic University and National Scientific and Technical Research

Council, Argentina

Guillem Rico, Autonomous University of Barcelona

Saskia P. Ruth, University of Zurich, NCCR Democracy

Bram Spruyt, Free University of Brussels

Marco Steenbergen, University of Zurich

Levente Littvay, Central European University

With the ideational turn in populism studies (Mudde and Rovira Kaltwasser 2013),

researchers have started to conceptualize and measure populism as a set of attitudes individuals

hold about politics and society (e.g. Akkerman et al. 2014; Elchardus and Spruyt 2014; Hawkins

et al. 2012; Rooduijn 2014b; Spruyt 2014; Stanley 2011). As proposed in the introduction to this

volume, such attitudes are ordinarily dormant, but may be activated given a favorable context for

populist discourse and its articulation by political actors. The measurement of these attitudes,

however, has been far from uniform, as the review by Van Hauwaert, Schimpf, and Azevedo in

1Contact author: BCS. BCS wrote the paper with substantial support from BS; BCS, IA, LL ran

the analyses; BCS and LL designed the study; IA, EA, NB, YMC, GD, GR, SR, MS collected

data, provided valuable comments and edits, and are listed in alphabetical order; LL led the

project. The authors would like to thank Andreea Nicutar, Daniel Kovarec, Elisa Totino,

Federico Vegetti, Selina Kurer, and Sharon Belli for their help with questionnaire translation and

survey implementation, and Sebastian Jungkunz and Nemanja Stankov for assistance with data

cleaning and writing the codebooks.

the previous chapter shows. The basis of the most commonly scale used today was set in

Hawkins, Riding, and Mudde (2012). It was extended into the popularized six-item version by

Akkerman, Mudde, and Zaslove (2014), and used by Spruyt et al. 2016, and in the chapters by

Andreadis and Ruth; Singer et al.; and Busby et al. in this volume.

However, as Van Hauwaert, Schimpf, and Azevedo have shown in their chapter, there is

room for improvement in scale development. From a survey methodology perspective, the items

fail to identify strong levels of populism and anti-populism and can only discriminate among

moderate forms of it. They are not polarizing enough, as there seems to be a general trend of

agreement: for all countries and items, the item averages are above the scales’ middle point. A

further limitation of the existing measures is that in most scales all items are positively worded –

meaning that more agreement indicates more populism. For this reason it is impossible to

discriminate between actual agreement with the content and acquiescence bias.

Our purpose with this study is to tackle the issue of scale development following

practices common in psychology but that have yet to make their way into political science. We

start with a large number of items, and use various techniques to select the few ones that work

better at capturing populist attitudes. Next we test which items are invariant across countries –

i.e., whether they measure the same thing, the same way, in different countries. Recent research

has shown that several scales, some of which have been around for decades in social sciences,

should not be used for cross-country comparisons because the measure is not invariant across

cultures (Alemán and Woods 2016, Ariely and Davidov 2010, Piurko et al. 2011). Our analyses

result in a short questionnaire of six or nine items which has high cross-cultural validity, and

captures a relatively wide range of information for this construct.

Dimensions of Populism

We start with the definition of populism adopted in this volume: it should be seen as a set

of ideas (whether a discourse, discursive frame, or thin-centered ideology, see Mudde 2004;

Hawkins 2009; and Aslanidis 2016 for further debate), which opposes the good people against an

evil elite, in a Manichaean division of politics where the voice of the good people should prevail

(Mudde 2004; Mudde and Rovira Kaltwasser 2013; Hawkins 2009). From it we derive three

elements, which we call the "core components" of populist attitudes: a) the notion of a good,

homogeneous people as a political actor; b) anti-elitism; and c) the view of politics as a moral

struggle, where one side is clearly good and the other evil – the Manichaean outlook. These are

somewhat different from those proposed by Mudde (2004) and Mudde and Rovira Kaltwasser

(2017): for them, people-centrism and popular sovereignty are distinct constructs. However, as

the items and analyses below show, these two are hardly distinguishable from one another

empirically, reason why we put them together.

These constructs can exist independently from one another. For example, not all

Manichaean discourses are populist. It is possible to see politics as a struggle between good and

evil and not fill these positions with "people" and "elites." Measuring the Manichaean element in

its "pure" form (i.e., not applied to any ideology) enables us to assess its empirical relationship

with other constituting elements of populism. The same applies for having a romanticized view

of common people, or despising political elites.2 Considering that all three are necessary

2Our measurement approach is very close to the elite-level ones conducted by Hawkins and

Castanho Silva in this volume (based on Hawkins 2009), and by Rooduijn, de Lange, and van

der Brug (2014): they also consider that all dimensions of populism must exist simultaneously, in

the same text, for it to be considered populist. For example, texts that are anti-elitist but do not

praise the people are not classified as populists in those scales. Seeing populism as a configural

components of the definition adopted here, we suggest that populism sits at the intersection of

these three broader kinds of discourse, as depicted by the shaded area in

>>> Figure 7.1. AROUND HERE <<<

This conceptual map is of absolute importance in scale development because if populism

indeed sits at the intersection of three kinds of discourse, its measurement should incorporate the

different facets separately. Previous research rarely acknowledges this (for notable exceptions

see Schulz et al. 2017 and Stanley 2011), and most often proposes unidimensional

measurements, eliminating any items that fail to load together with the others. However, if

populism has distinct subcomponents, one should expect that not all items will behave as if they

are measuring a single construct.

Core Constructs of Populism and their Measurement

Proceeding from this conceptual discussion, we now move to identifying measures for

the core components of ideational populism. The first is praise of “the people”, common to

virtually any definition of populism (Ionescu and Gellner 1969; Canovan 1981). Second, anti-

elitist, or anti-establishment ideas, which claim that a powerful and corrupt minority has taken

over politics. And, third, a Manichaean, or good-versus-evil, view of politics (Hawkins 2009), in

that populism only exists when the two sides – people and elites – are seen as in moral

opposition to one another, instead of merely having programmatic disagreements.

concept where all aspects have to be present for it to be characterized is, therefore, at the core of

an ideational understanding.

Praise of the People/People’s values

Praise of "the people" is the first identified characteristic of populism, no matter what

specific definition one uses (see, for instance, Ionescu and Gellner 1969; Canovan 1981). The

definition of who belongs to this people is one of the essential points in distinguishing between

different kinds of populism,3 and changes according to the time and context (Panizza 2005),

making it an "empty signifier" (Laclau 2005). This variously and indeterminately defined people

is glorified as a virtuous entity which embodies ideals of hard-work and honesty (Taggart 2000).

There is a strong sense of the "common" or "ordinary" man, whose values and behavior are

morally superior to those of other groups in society.

Populist discourse not only praises the people and its values, it also understands that a)

the people is a homogeneous entity who b) has an identifiable "general will" which should be the

basis of all politics (Mudde 2004).4 There is no space for disagreements within the people – it is

seen as having a single set of values, preferences, and interests, without room for legitimate

differences (Müller 2016). Populism has a strong emphasis on popular sovereignty and the idea

that the people's voice is not being heard by those in power (Canovan 1999; Mudde and Rovira

Kaltwasser 2013). This results in calls, by populist actors, to "return the power to the people."

To capture this dimension, we have designed or collected 29 survey items from existing

sources. Twelve items focus on the dimension of praising common people, while the other 17 tap

into the idea of the people's general will as the basis of politics, or the homogeneity of the people

(as defined in Schulz 2017). Examples of the former include "generally speaking, average

3We come back to this under the discussion of left- and right- varieties of populism.

4These are two of the three separate dimensions proposed and measured in the scale by Schulz et

al. (2017).

people are hard-working," and "I take pride in being an ordinary person." For the latter, some

examples are "The politicians in Congress need to follow the will of the people," and "The will of

the people should be the highest principle in this country's politics." We have also included

several negative-worded items, such as "There is no such a thing as ‘the will of the people’" and

"The worst politicians are those who come from the common people." The full list of items is in

the Appendix.

Anti-Establishment/Anti-Elite Feelings

Populists always see an elite in opposition to that good people. The elite is a minority of

corrupt forces who have subverted the political system to work for their benefit (Mudde and

Rovira Kaltwasser 2013, 502). It may be found, depending on the context, dominating politics,

the economy, culture, media, and/or the judiciary (Rooduijn 2014, 577). The elite exploits the

people in its pursuit for more power and profit, meaning that liberation from this domination is

needed (Hawkins 2009). Who is categorized as elite (or as the people) varies according to time

and place. Usual elite targets may be governments in general and its officials, the rich,

supranational organizations, international financial bodies, or foreign countries and their leaders.

What is essential is that this elite is represented in domestic politics as well – for example, the

"lackeys of imperialism" in Latin America.

Items were designed to capture the idea of anti-elite feelings without going into detail of

who the establishment5 is – this is done at the sections that try to differentiate left- and right-

wing populisms. Otherwise, ideology would certainly be a confounder. Therefore, it is often

restrained to the political establishment and the idea of elites in abstract. Examples of items

5In this chapter we treat "elite" and "establishment" as equivalents. Within the concept of

populism, they are both referring to the same group.

include "Elected politicians sell out to big business," "Politicians do not want to improve the

lives of ordinary people," and "The government is pretty much run by a few big interests looking

out for themselves." Some of the reverse-worded items are "Politicians are actually interested in

what people like me think" and "Most politicians seek power to serve others."

Manichaean Outlook on Politics

As the description in previous sections already indicates, the division between the people

and the elite is primarily moral (Mudde and Rovira Kaltwasser 2013), with one side being the

monopolist of virtue, and the other of vice. In the existing measures (e.g. Hawkins et al. 2012;

Akkerman et al. 2014) this element was incorporated in the wording of anti-establishment items.

The Manichaean outlook was expressed by statements that accused established politicians of

having betrayed the people. In this way the Manichaean outlook was not considered an essential

element of politics per se but rather of the current state of democracy.6 We follow a different

track. No measurement attempt up to date has tried to gauge Manichaean attitudes of politics

directly and individually (with the partial exception of Hawkins et al. 2012 and Stanley 2011),

even though this is theoretically plausible. Keeping with the idea of measuring these dimensions

separately, we developed questions on whether the respondent perceives politics as a moral

struggle, without referring to people or elite. Our aim is to measure the Manichaean outlook of

politics in its pure form and subsequently assess how this dimension relates to other dimensions

6Indeed, one of the items that presented the Manichaean outlook of politics in essentialist

language (i.e., "Politics is ultimately a battle between ‘good’ and ‘evil’") failed to load on the

populism dimension in the Netherlands (Akkerman et al., 2014). Because there was only one

such item, it is was impossible to determine whether this item was part of a separate latent

dimension that could not be tapped with all other items (which all were applied to populism).

of populist attitudes. Examples of items are "You can tell if a person is good or bad if you know

their politics" or "People I disagree with politically are driven solely by greed." Reverse items

include "The people I disagree with politically are not evil" and "In politics, everyone wants

what they think is the best for the country." It is important to note that because we see this

dimension as a kind of political cosmology or worldview (Hawkins and Rovira Kaltwasser,

2017) the items do not exclusively refer to the differences between "the people" and "the elite."

On the contrary, the first three items refer to differences among the people themselves.

Potential Populism Dimensions

Any attempt would be incomplete if we ignored the plurality of views on what constitutes

populism. We have conducted a literature review to find other constructs which have been

considered part of the concept and, afterwards, used the mailing list of Team Populism7 to ask

for further inputs on what dimensions and questions might be considered. We have settled with

the potential dimensions listed in the following sections, and collected or designed a number of

survey items for each one individually as well.

Strong Leader

Populism is sometimes associated with preference for a strong leader, along the lines of

"delegative democracy" (O’Donnell 1994). Populist leaders, especially in Latin America, have

claimed that being directly elected gives them a popular mandate to override other institutions if

that is necessary for implementing the "general will" (Levitsky and Loxton 2013). For some,

populism would necessarily involve a preference for a personalist authority over institutions of

representative democracy (Pappas 2014). To tap into this idea, we propose items such as "What

7See more at teampopulism.com and in the Preface.

our country needs is a leader who will be admired" or "Our presidents should do what the

people want even when the laws prohibit him from doing it."

Simple, Direct Style

Much work on populism sees it not so much as a discourse, but as a style of dress,

language, and performance (Moffitt 2016; Ostiguy 2017). Populism is said to have a direct and

unmediated connection between leader and followers (Weyland 2001), based on charismatic

appeal (Pappas 2016). To achieve this, the usage of a colloquial language, popular mannerisms

and informal clothing are some of the tools employed by populist leaders who want to "look like

the people" (see, e.g., de la Torre 2013, 37). We suggest a scale that taps into this preference for

a "politician like me," and has negative attitudes towards what may be seen as complex political

dealings or trade-offs. Examples of items in it are "I prefer politicians who tell it how it is," and

"It's important for a political leader to be like the people he or she represents."8

Perception of Crisis

In populist discourse, traditional parties are charged with having created an oligarchical

system where there is no difference among them, thus hollowing the meaning of real democracy

(Mouffe 2005, 64). This makes it necessary that a new, truly popular, kind of party appears

(Mudde 2004, 546), in a call for liberation (Hawkins 2009). Proclaiming that there is a crisis of

representation and in the functioning of democracy may be seen, therefore, as an essential aspect

of populism (Rooduijn 2014; Moffitt 2015). We tap into this dimension with items such as "The

rights of individuals have been systematically curtailed in this country" (Dryzek and Berejikian

1993).

8Both conceptually and empirically (as we see in the analysis) this construct is eerily similar to

the idea of glorifying the people.

Anything-goes attitude

Hawkins (2010, 36) argues that a consequence of populist discourse is an "anything-

goes" attitude, where formal procedures and liberal rights are seen as a hindrance to the

realization of the will of the people. In short, because this general will is the highest principle in

politics, anything and anyone setting limits to it is naturally seen as illegitimate. As such, this

obstacle deserves to be removed, even if by illegal means, in the name of the greater good. This

idea is very close to authoritarianism, or that populism essentially is democratic illiberalism

(Pappas 2014). We have items that tap into a willingness to limit checks and balances

institutions, as well as the rights of some groups that may be seen as enemies of the people

(Seligson 2007). Namely, we use items from the Latin American Public Opinion Project’s

questionnaires9, from several waves, which read, for example, "It is necessary for the progress of

the country that our presidents limit the voice and vote of the opposition parties."

Left and Right Populisms

What makes populism left or right is how "the people" and "the elite" are defined, based

on the ideology with which it is associated in each case. Populists on the left, on one hand, see

this division as one rooted in economic inequality (March 2007), where the people are seen as a

majority of economically and socially disadvantaged groups, oppressed by a minority who,

because of their resources, controls politics. The elite are the rich, financial institutions (both

internal and external), and whoever may be associated with and subordinated to those actors.

Right-wing populism, on the other hand, has a more nationalist understanding of "the people"

which does not emphasize how it is dominated economically, but how the nation and its symbols

9Source: The AmericasBarometer by the Latin American Public Opinion Project (LAPOP),

available at: http://www.lapopsurveys.org.

are threatened by a minority (the "nativism" dimension in Mudde 2007). In economic terms, it

frequently sees the people as middle class, hard-working individuals oppressed by the

government, who want to take their money and give to privileged minorities. The elites are

politicians, local and international, who use their control of the state to harm the national

majority, benefiting themselves and their allies.

To complement the more neutral items from the batteries on core constructs, we have

developed specific statements to reflect right- and left-wing versions of populism, associating

actors to the roles of "people" and "elite" accordingly. For right-wing populism, we include items

such as "People who pay no taxes should have no say in how this country is run," and "In this

country, liberal intellectuals are the real enemy of the people." On the left, examples include

"Big corporations accumulate wealth by exploiting the people" and "The unemployed,

underemployed, marginalized groups – these are the real people who should have more voice in

politics."

Data

Altogether, there are 145 survey items proposed for the ten dimensions outlined above.

For our first two studies, we collected data in eight countries. In the United States, we used an

online survey through Amazon's Mechanical Turk (MTurk), with 234 respondents, in May 2015.

In Argentina, Croatia, Greece, Mexico, Spain, and Switzerland, we relied on samples of

undergraduate and graduate students.10 The sample sizes are 257 in Argentina, 193 in Croatia,

10At the following institutes and dates: Argentine Catholic University (Argentina), June-August

2016, University of Zagreb (Croatia), Fall 2015; Aristotle University Thessaloniki (Greece), June

2015 and February/March 2016; Autonomous University of Barcelona (Spain), Fall 2015,

172 in Spain, 262 in Greece, 163 in Mexico, and 247 in Switzerland. In Belgium, we

complemented a sample with students from the Free University in Brussels with respondents

recruited through social media and university mailing lists. The survey was administered only in

Dutch (N = 153). None of the samples approaches nationally representative, in line with the

general practice of scale development in psychology. The main goal of this exercise is to explore

dimensionality and questionnaire reduction, by getting a first glimpse into what items of the full

list tap into populism constructs, and which ones work across different cultures. Therefore, the

seven countries with student samples, plus the American online sample, offer enough variation to

permit us, first, to reduce the long list into a shorter one with items that load together in the

relevant dimensions, and second, to test which items are invariant across groups.

For our third study, we use a new round of data collected online in nine countries. For the

United States we once again used Amazon’s MTurk, while for the other eight (Brazil, France,

Greece, Ireland, Italy, Mexico, Spain, and the United Kingdom) we used the crowdsourcing

platform CrowdFlower. Because of its international pool of contributors, scholars have begun

using it for scientific surveys. Results of testing show that respondents give answers with similar

levels of quality to other online providers such as MTurk (Peer et al. 2017). Our sample sizes

vary between 200 and 300 respondents for each country, except for a larger American sample of

505; all samples were collected between November 2016 and March 2017.11

National Autonomous University of Mexico, February/March 2016, and University of Zurich

(Switzerland), September 2016.

11The Irish sample was completed with 100 respondents from Qualtrics panels, due to there

being too few CrowdFlower users in this country to reach our target of 250. While we also

collected data through CrowdFlower and Qualtrics for Hungary, we have left it out of the

Despite distortions in CrowdFlower samples relative to national populations, they are still

more diverse than student and other convenience samples. Median ages for countries are around

27-30, with most respondents between 20 and 40. This is a bias towards younger respondents

less pronounced than that observed with students. Education and income distributions are also

more varied, as is geographic location. There are important gender imbalances – one of our

samples is 80% male. However, ideological balance is more closely achieved than with

undergraduate students. In sum, the imbalances in CrowdFlower samples (the largest being

gender) are different from those in student samples (location, age, occupation and ideology), in

relation to their representativeness. Therefore, we do not expect biases to be correlated across the

two.

Scale Development, Step by Step

We proceed with reducing our initial 145 items into a manageable scale in three steps. First, all

of them are included in factor analysis models, to identify the latent dimensions underlying all of

these questions. We seek to find what constructs, of those theorized, can be retrieved from the

data. This step gives us the coherent sub-dimensions to be measured, and a smaller list of items

measuring each one. Second, we perform a test of measurement invariance on the student

samples. This test tells us which items have high cross-cultural validity, and which ones should

be dropped from the scale due to not working well across different cultures. Last, with a new

analyses reported here due to concerns about response quality. While there is a sizeable useful

sample, diagnostics show that many respondents there powered through the questionnaire

without giving faithful answers, and several were able to reach the validation code at the end

without actually taking the survey. Including the Hungarian sample in the analysis, however,

does not change results either for invariance tests or information curves.

round of cross-national data, we validate those findings about the cross-national validity of the

remaining short scale.

Step 1: The Factor Structure of Populism Dimensions

The first study we conduct aims at identifying the underlying dimensions across the 145

survey items. We use exploratory factor analysis (EFA)12 to uncover the number of latent

variables (dimensions, or factors) that can explain the largest amount of variation in the items.

EFA helps us observe whether our proposed items load on the proposed constructs, and which

constructs grant the most explanatory leverage within these data. We apply the model to the

pooled data with student samples and the first Amazon MTurk sample.

We run eleven EFA models, having from 2 to 12 factors. There is no consensus on how

to choose the appropriate number of factors (m) to retain after EFA. We have tried several

existing alternatives, as described in the Online Appendix B. They fail to converge on a single

solution, and we choose to follow an approach of using absolute indicators of model fit. This

works by selecting the solution with the smallest number of factors that gives acceptable fit

(Preacher et al. 2013). We pick the solution with four factors, as the first one in which both

SRMR (Standardized Root Mean Square Residual) and RMSEA (Root Mean Square Error of

Approximation) indicate good fit (meaning they are below 0.05).13 We also follow a substantive

interpretation: inspecting the items in each factor for solutions with a larger m, there are four

stable and clearly identifiable factors, with more or less the same indicators across solutions; the

12With an oblique rotation, since the dimensions are expected to correlate with one another (see

Browne 2001).

13The full table with model fit indicators for all solutions between 2 and 12 factors is in the

Online Appendix B.

others that show up tend to have few indicators or fail to present a meaningful theoretical

construct. The solution with three factors does not discriminate indicators which seem to belong

to different constructs.14

Conceptually, the four factors identified are interpreted as people-centrism, anti-elitism,

Manichaean outlook, and authoritarianism. The first three are core constructs of populism, while

the fourth is a construct whose connection to populism is debated conceptually (Mudde and

Rovira Kaltwasser 2017; Müller 2016) and empirically (see Aguilar and Carlin in this volume).

For each dimension we look at items with an absolute factor loading greater than 0.35. There are

19 for people-centrism, 37 for anti-elitism, 15 for Manichaean outlook, and 20 for

authoritarianism. Table 7.2 presents the factor correlations from the EFA. While people-

centrism and anti-elitism have a moderate correlation with one another (r = 0.28), the other

factor correlations are rather small. This evidence suggests we are indeed capturing different

dimensions that can be seen as separate constructs and should, accordingly, be measured

separately.

Table 7.1 Correlation Matrix after EFA

Praise people Anti-elitism Manichaean outlook Authoritarianism

Praise people 1.00

14Table 7.A1, in the Online Appendix A, includes all items that have a factor loading above 0.3

in each of the constructs. Further, it indicates which items load into single dimensions on a

Mokken Scale Analysis (Mokken 1971; Van Der Ark 2012). As we can see, there is broad

agreement regarding the composition of dimensions between MSA and EFA for anti-elitism and

on the items with higher loadings in authoritarianism and Manichaean outlook. MSA produces a

larger number of dimensions for the people-centrism dimension than EFA.

Anti-elitism .278 1.00

Manichaean Outook .073 .095 1.00

Authoritarianism .034 .006 .108 1.00

Step 2 – First Test of Measurement Invariance

Given our large number of items, we could expect that EFA would return several ones for

each possible dimension. While EFA completes the dimensionality reduction, questionnaire

reduction continues with our first test of measurement invariance. This test tells us which items,

for each dimension, should be retained as they work in similar ways across different countries.

After this part, our goal of creating a more manageable scale, with few statements per dimension,

can be obtained.

The most common way of testing the equivalence of measurement instruments is with

multiple group Confirmatory Factor Analysis (MGCFA, Jöreskog 1971). In this strategy, three

CFA models are fit to the data: the first where all estimated parameters are allowed to vary

across groups (configural model), the second where factor loadings are constrained to be the

same across groups (metric invariance), and the third where loadings and intercepts are held

constant (scalar invariance). In other words, the configural one is equivalent to fitting separate

CFA models in each country. In the metric invariance model, the factor loading of each single

indicator on its latent variable (say, the loading of "Quite a few of the people running the

government are crooked" on Anti-elitism) is forced to be the same in all countries. If the

indicator measures that dimension similarly across countries, the factor loadings in the configural

model would be similar anyway, and forcing equality is not expected to make the model worse.

This evaluation is done with model fit information: if the more constrained models do not fit

significantly worse than the less constrained, it means there is measurement invariance.

Substantively, what the second model means is that an increase in one unit of the latent construct

has the same impact on the observed indicators across all groups. The third model, scalar

invariance, tests not only whether variation in the latent construct means the same for

respondents in all groups, but also that respondents with the same level of the latent construct

would give the same answer in all groups.15

Multigroup CFA has been criticized for being too strict (see, for instance, a comparison

with alternatives in Davidov et al. 2015). In cross-national surveys, items rarely work perfectly

the same way in all countries, and the invariant models tend to be rejected. Alternative

approaches that allow for slight noninvariance have been proposed. In this step we use the

alignment method (Asparouhov and Muthén 2014). It modifies the configural model to align

factors and intercepts, similar to a rotation in EFA, taking into account actual factor means and

factor variances. Not only is this method more practical than MGCFA, and not demanding exact

invariance, it also lists which indicators are (non)invariant in each group. MGCFA is used in our

last validation test, in step 3.

For our model, we select all constructs we retain from EFA, and include twelve items

with the highest absolute factor loadings as indicators of the predicted latent variables in the

CFA alignment model.16 We strive to include at least three negative-worded statements in each

15Metric invariance is usually accepted as sufficient, because for regression purposes it is enough

that a change of one unit in the indicator means the same across groups.

16We limit to twelve per scale (eleven for Manichaean outlook), leading to 47, so that the model

is identified. With this number, the model has 147 free parameters, and our limit is 152: the

number of observations in the smallest sample (153, Flanders), minus 1.

dimension, even if these would not be on the list of highest absolute loadings, because of their

value in questionnaire design. Table 7.2 shows information for the 5-7 best working items in

each dimension at this stage. Items are chosen based on invariance, average loadings, and

distributional characteristics, and we sought to include at least one negative-worded item in each

category. It is possible to find at least four items with invariant factor loadings across all eight

samples for each construct, and a few even have invariant intercepts across all countries. Those

that violate invariance do so in only one or two samples. Moreover, we observe items with both

higher and lower than average means, indicating it is possible to get indicators that people

disagree with. Factor loadings are high for most of the indicators, with a few exceptions in the

people-centrism dimension.

Broadly, items have face-validity in capturing the proposed concepts. A few points must

be noticed at this stage: first, we observe a clear and distinctive Manichaean outlook element in

populist attitudes. This is the first time such a construct has been tested, and it contributes to our

empirical and conceptual understanding of how populist attitudes are structured. Second,

exploratory factor analysis results show one cannot clearly distinguish between three theoretical

constructs: "praise of the people," "homogeneous people/general will" and "simple, direct style."

The six items in the people-centrism part of Table 7.2 are an amalgam with contributions from

all three theorized constructs, amounting to a coherent idea of glorifying common people in

politics. This is contrary to previous findings by Schulz et al. (2017), who distinguish between

people homogeneity and popular sovereignty. While scholars go through great lengths to

theoretically distinguish between these concepts, we find that individuals make less fine-grained

distinctions when thinking about politics.

Third, the anti-elitism dimension has items not only from the original scale proposed for

it, but also from those for left- and right-wing populism. Item Ant4 in Table 7.3 was originally

drafted for the left-wing populism scale, as it clearly frames corporations as the elite against

whom to fight. Ant2, on the other hand, is a reverse-worded item from the right-wing populism

scale. Nevertheless, both load together with more generic anti-establishment statements. The

fourth point to highlight is that noninvariance is most egregious in two samples: Croatia (four

loadings and five intercepts) and Switzerland (one loading and five intercepts). That might be

caused by specificities of these student samples, or by different characteristics of populism in

both countries in relation to the rest. For this reason it is essential to conduct an independent

validation as we do in the next step.

Table 7.2 Invariance Test with the CFA Alignment Method

Order Item NI

Load. NI

Int. Mean

int. Mean

load.

People-centrism

Ppl1. Politicians should always listen closely to the problems of the people. – GR 6.29 0.50

Ppl2. Politicians don't have to spend time among ordinary people to do a good job.* – – 2.87 -0.82

Ppl3. The will of the people should be the highest principle in this country's politics. – – 5.15 0.90

Ppl4. In a democracy, the will of the majority should prevail. – – 5.46 0.57

Ppl5. It's important for a political leader to be like the people he or she represents. – – 4.86 0.92

Ppl6. I prefer politicians who tell it how it is. – HR 6.00 0.50

Anti-elitism

Ant1. The government is pretty much run by a few big interests looking out for themselves. – – 5.23 1.18

Ant2. Government officials use their power to try to improve people's lives.* HR,

CH MX 3.66 -0.92

Ant3. Quite a few of the people running the government are crooked. – HR,

CH 5.48 1.10

Ant4. Big corporations accumulate wealth by exploiting the people. HR – 5.11 1.22

Ant5. Politicians are not really interested in what people like me think. – ES,

CH 4.76 1.21

Ant6. Politicians are actually interested in what people like me think.* – BE,

CH 3.00 -1.01

Ant7. The government is currently run for the benefit of all the people.* HR HR,

AR 2.70 -0.87

Manichaean outlook

Man1. You can tell if a person is good or bad if you know their politics. – – 3.69 0.81

Man2. I would never stop talking to a friend because of their political opinions.* – BE,

CH 5.37 -0.88

Man3. The people I disagree with politically are just misinformed. – HR 2.73 0.92

Man4. Politics is a struggle between good and evil. – US,

ES,

BE,

CH

3.94 0.60

Man5. The difference between me and those who support other parties is that I care about what's

good for everyone. HR,

MX HR 4.13 0.80

Authoritarianism

Aut1. People who only say bad things about [country] should not be allowed to conduct even

peaceful demonstrations. (any4) – – 1.85 1.12

Aut2. People who only say bad things about this country have the same right as anyone else to

appear on television to make speeches.* (any8) – CH 5.81 -1.20

Aut3. People who only say bad things about our form of government, not just the current

administration but the system of government, should not have the right to vote. (any3) – AR 1.84 0.94

Aut4. People who only say bad things about [country] should have the same right as anyone else

to conduct peaceful demonstrations.* (any5) – – 5.95 -1.16

Aut5. People who only say bad things about this country should not be allowed to appear on

television to make speeches. (any7) – – 2.06 1.20

Step 3 – Cross-National Validation

In our final step, the items in Table 7.2 were fielded in new surveys in nine countries. To

that we added three more negative-worded items for the Manichaean outlook battery since the

best performing items there are all (but one) positively worded. Moreover, we do not include

authoritarianism; this is a well defined distinct construct with over a half a century of scale

development and its relation to populism is conceptually controversial. While the EFA identified

an authoritarianism dimension from all those items, and several were invariant across student

samples, it is distinct from the construct we are trying to measure. For this reason, items were

selected only for people-centrism, anti-elitism, and Manichaean outlook.

The model tested is a multigroup CFA with three latent variables, and three indicators for

each latent variable, as depicted in Figure 7.2. The results in it are those for the model with factor

loadings constrained to be the same across countries (the loadings invariant model). Model fit is

acceptable, with RMSEA and SRMR below 0.07, and TLI above 0.90. CFI is a bit below the

recommended minimum of 0.95, and the chi-square test is significant (however, this test is

known to be sensitive to large samples, as noted in Kline 2016). We have included a method

factor to take into account the fact that some items are worded negatively, and because research

shows that individuals respond differently to positive- and negative-worded statements

(DiStefano and Motl 2006).17 Factor loadings reported for individual items are unstandardized.18

Items work well for both the anti-elitism and people-centrism scales, with high absolute

loadings. For Manichaean outlook, the first two work better, while the last ("The people I

disagree with politically are just misinformed") has a somewhat lower loading. This is an

interesting finding because in terms of stereotype content (Fiske et al. 2002) the first two items

17Four models were fit, following the suggestions by DiStefano and Motl (2006): model (a),

reported, with a method-factor for positive-worded items; model (b) with a method factor for

negative-worded items; model (c) with correlations among residuals of all positive-worded

items, and model (d) with correlations among residuals of all negative-worded items. In the main

text we present model (a), which has the best model fit. Fit indices for the others are: (b): Chi-

square = 760.487, df = 254, p < .001, RMSEA = .085 (90% CI: .078-.091), SRMR = .088, TLI =

.812, CFI = 853. Model (c): Chi-square = 357.662, df = 129, p < .001, RMSEA = .080 (90% CI:

.071-.089), SRMR = .054, TLI = .838, CFI = .935. Model (d): Chi-square = 703.797, df = 237, p

< .001, RMSEA = .084 (90% CI: .077-.091), SRMR = .086, TLI = .814, CFI = .864.

18Items measured in Likert scales from 1 (Strongly disagree) to 7 (Strongly agree).

differ from the third. Specifically, while the first two refer to "warmth" (i.e., people’s intentions),

the items that attribute political disagreement to misinformation taps into a "competence" related

claim. It is clear that the Manichaean view on politics is basically a view which refers to people’s

intentions, so it is perhaps not surprising to find that exactly the items that refer to that element

turn out to work best. From the top three suggestions for each dimension in Table 7.2, Man2 had

a low factor loading and decreased model fit substantively in this test.19 We replaced it with

another option of a negative worded item that had been present in the original questionnaire:

"The people I disagree with politically are not evil." The final, nine-item scale tested is:

Box 1. Final Item Suggestions

People-centrism:

Ppl1. Politicians should always listen closely to the problems of the people.

Ppl2. Politicians don't have to spend time among ordinary people to do a good job.*

PPl3. The will of the people should be the highest principle in this country's politics.

Anti-elitism:

Ant1. The government is pretty much run by a few big interests looking out for

themselves.

Ant2. Government officials use their power to try to improve people's lives.*

Ant3. Quite a few of the people running the government are crooked.

Manichaean outlook:

Man1. You can tell if a person is good or bad if you know their politics.

19This item has a very positively skewed distribution in all samples, meaning that it was invariant

on step 2, since most respondents agree with it in all countries, but does not provide much

information since answers are clustered on the upper end.

Man2. The people I disagree with politically are not evil.*

Man3. The people I disagree with politically are just misinformed.

>>> FIGURE 7.2 AROUND HERE <<<

A measurement invariance test with multigroup CFA shows that this model does not fit

significantly worse than the configural model, i.e., one where factor loadings are allowed to vary

across countries (meaning, in practice, one CFA is fit in each country), as seen in Table 7.3.

These indicators capture the latent variables across all nine countries included in this sample in a

similar manner. In other words, this survey instrument has a high degree of cross-national

validity, at least for the countries tested. Note that the countries were selected to represent

several regions, with variation in left and right-wing populism and varying degrees of populism

overall. Model comparison is done with a chi-square test of model difference. It works the

following way: each model has a chi-square statistic indicating fit, where lower values indicate

better fit. If we compare the difference in chi-squares between two nested models and there is a

significant difference, it means that the second (more restricted) has a significantly worse fit than

the first. If the model with constrained loadings is not significantly worse, the next step is

constraining intercepts to be the same across groups, and once again testing whether the chi-

square difference is significant. Measurement invariance, therefore, is achieved when we do not

observe significant differences between models (i.e., p > 0.05). As a comparison, we have also

tested the invariance of the six-item scale proposed by Akkerman et al. (2014) in the same data,

which is currently the most used for measuring populist attitudes. Results in the lower part of

Table 7.3 show that factor loadings are not invariant across countries: the model with constrained

loadings fits significantly worse than the configural (p = 0.02).

Table 7.3: Multigroup CFA Invariance Test

Model Chi-square Chi-sq. Diff. df p

New scale

Configural 440.58

Loadings 599.12 102.20 88 .14

Intercepts 912.57 322.17 40 <.001

Akkerman et al.

(2014)

Configural 230.76

Loadings 297.15 59.935 40 .02

Intercepts 553.11 253.283 40 <.001

While these results may seem a bit dark for the Akkerman et al. (2014) scale, we

emphasize that a multigroup CFA is a conservative test of invariance. A p-value of .02 does not

indicate a strongly non-invariant instrument. This means that researchers who have already

collected data with the scale suggested by Akkerman et al. (2014) should conduct the appropriate

measurement invariance tests on their own data before analyzing substantive results, but it does

not a priori invalidate any analyses using their scales, including those in this book. However, if

one plans to include a battery in cross-national surveys with multiple countries, non-invariance in

the Akkerman et al. 2014 instrument would likely become an issue.20

20The problematic item in this battery, for invariance, is “I would rather be represented by a

citizen than by a specialized politician”. If the factor loading for only this indicator is allowed to

vary across groups, the constrained model does not fit significantly worse than the configural

(p=.18). Doing the same for each of the other indicators still result in models with significantly

worse fit.

Scale and Item Information

Following the analyses by Van Hauwaert, Schimpf, and Azevedo in the previous chapter,

we conclude the assessment of this scale with a model to test the amount of information it

contains, and how much of the latent construct it is able to capture. We have pooled the data for

all nine countries and run a graded ratings scale model (Muraki 1992) to identify, first, the

amount of information of this scale along the construct, and second the amount of information

that each item contributes. This analysis was done in two ways: first, each dimension separately;

and second, with the two top items of each construct pooled to form a short, unidimensional

scale for populism.

>>> FIGURES 7.3 – 7.5 AROUND HERE<<<

Figures 7.3-7.5 have the information curves for each dimension on the left-hand side of

each panel (solid lines). As can be seen, each set of items captures a different area of their

respective dimensions, with the first two mostly on the negative side (on the [-3:1] interval)

while the last (Manichaean outlook) performs better on individuals higher on this trait (those in

the [-1:3] range). At the extremes the dashed error curves become higher, indicating that the

measure is less apt to capture individuals positioned on those levels of the construct. Item

information curves show that for each scale there is one most informative item (Ant3, Ppl1, and

Man1), and that the negative-worded items (Ant2, Ppl2, Man2) contribute by capturing

information further away from the center in all three, even if it is not the highest level of

information.

Information curves also suggest that the two positive-worded items in each scale capture

a similar range of their constructs. Therefore, we test an aggregate short version of our scale with

six items – the first positive and the negative-worded for each dimension (Ant1, Ant2, Ppl1, Pp2,

Man1, Man2) – loading on a single construct.21 The information curve for this short scale is in

Figure 7.6, with a comparison to the information curve obtained with the Akkerman et al. (2014)

six-item scale, which Van Hauwaert, Schimpf and Azevedo found to have a broad range in the

previous chapter. Here we see that the short version of the new scale captures a somewhat

broader range than Akkerman et al. (2014), primarily on the low end of the scales, but both still

fall short when discriminating extremely high levels of populism.

>>> FIGURE 7.6 AROUND HERE <<<

Conclusion

The first goal of this project is to suggest a psychometrically validated scale to measure

populism as an attitude. We start with the core dimensions that compose the concept, develop

batteries of questions to tap into them, and conduct exploratory analyses to identify those

dimensions which can be found among the public. Confirming the ideational theory, we find

three stable constructs that are clearly part of populism among our original 145: people-centrism,

anti-elitism, and Manichaean outlook. We use two rounds of cross-national validation in order to

reduce the original number of items for each into a short scale, and reach a final battery that can

be used either with nine or six statements. Confirmatory factor analysis and Item Response

Theory models show that the scale captures a broad range of the construct and has high cross-

national validity.

21Model fit: Chi-square = 53.873, df = 3, p < .001, RMSEA = .079, SRMR = .03, CFI = .959,

applied on the pooled data (n = 2708). The lowest standardized loading is .331, for Ppl2. All

residuals of negative-worded items are correlated to one another, as are those of positive-worded

ones, to correct for method bias.

This new scale has several advantages over existing alternatives: first, it has been

developed with the concern for cross-cultural validity from its inception. Few, if any, of the

existing scales have had their invariance tested to know whether they do work in a similar way to

capture the phenomenon of populism in different countries. Until such an exercise is done, they

should not be used to compare levels of populism, or even correlates, across large numbers of

countries. The scale proposed here is a reliable instrument across seventeen samples from

thirteen different countries, including both convenience (student) samples and more diverse,

online ones. Moreover, in this process we have also mitigated potential translator effects for the

languages we tested. Researchers who wish to apply this battery have a pool of languages into

which these items have already been translated and in which their validity has been tested,

available in the Online Appendix C. While it is not possible to assure that the scale would be

invariant in other countries than those included here, we sought for large regional coverage, as

well as examples of cases with distinct kinds of populist parties (including no successful

populists at all), to maximize the possibility that the scale would work well in countries we did

not include.

A second advantage of this scale is dividing populism into its subcomponents and

measuring each one separately. While some other scales have done that (e.g. Schulz et al. 2017,

Stanley 2011, Oliver and Rahn 2016), ours is the first to depart inductively from a range of

potential conceptual constructs and narrow the scale down into those that appear to be the most

stable on the data, as well as conceptually sound following an ideational definition. Researchers

now have the flexibility of investigating not only correlates of populist attitudes writ large, but

how each one of its subcomponents might be related to a different set of social and psychological

characteristics. Additionally, our analysis highlighted the importance of authoritarianism in

relation to the other facets of populist attitudes. The structure of relationships between these

concepts warrant additional work. And we encourage scholars to work with these dimensions in

a flexible and theory-based way when they do their own research.

A concern can be raised regarding the data we used. Student samples are much more

homogeneous than national populations, and so it is natural that sometimes findings from studies

with them do not generalize. We attempted to minimize this problem by relying on a different

kind of convenience sample for the validation exercise: those recruited through CrowdFlower.

Nevertheless, they are still not representative of any population, and formed by individuals who

chose to take the survey. As we argue earlier, the imbalances in CrowdFlower samples are

different from those in student samples. Therefore, biases should not be correlated between the

two. For instance, if an item works very well only among well-educated young people, it would

perform poorly in the CrowdFlower samples, which are more diverse in this respect. Our

samples, therefore, offer enough variation that these shortcomings are minimized. Moreover,

much of the causes behind measurement non-invariance include translation effects, and terms or

concepts that do not make sense in different contexts. These can be captured with cross-national

samples even if they are not diverse. Finally, from a practical perspective, including the full

batteries into representative cross-national surveys would have been prohibitively expensive –

even the shorter questionnaires in CrowdFlower were 10 minutes surveys. For the scale

reduction exercise, we must rely on sub-optimal samples. However, now that we arrived at a

short version, this can be included in surveys with representative samples, and their properties

then reassessed.

From a theoretical perspective, we have two main findings. First, that a general

Manichaean outlook of politics, dissociated from people and elites, is a component of populist

attitudes. This had not been tested before, and in general only incorporated into how populists

frame elites and the people. Second, we fail to statistically differentiate between praising

common people and the idea of popular sovereignty in politics. These two are treated as

conceptually distinct (e.g. in Mudde and Rovira Kaltwasser 2017), but at the attitudinal level are

too close to be distinguished from one another. These findings call for a theoretical reevaluation

of these dimensions' status within the concept of populism, at lease when seen in its

psychological dimension.

Further, while the measurement divides populism into its components, we offer a short

version of the scale. It has six items that can be used to measure a single underlying dimension,

for those researchers who have stricter limitations on the amount of survey space they can use.

Another contribution is offering negative-worded items in each of the dimensions, making sure

that we are indeed measuring populist attitudes and not acquiescence bias. This is a shortcoming

in several scales (an exception is Stanley 2011), and explains at least in part why populist

attitudes are found to be so widespread almost everywhere they are measured.

Finally, another shortcoming of existing scales found by Van Hauwaert, Schimpf, and

Azevedo was that they fail to capture the full breadth of populist attitudes: all scales either

discriminate only moderate populism/non-populism, or work better on one end (full populists or

full non-populists). The best performing one in this regard was found to be that proposed by

Akkerman, Mudde, and Zaslove (2014). The scale proposed here captures a somewhat broader

range of the concept, offering information about full non-populists, and moderates on both sides

of zero. Its advantage, however, lies in the cross-national validity of its application.

The benefit from this effort is not only the development of a more refined,

psychometrically tested scale of populist attitudes. The division into dimensions also allows

researchers to 1) use the dimensions that best fit their definition when doing future studies,

creating more precise measurements; 2) study the relations between dimensions of populism that

were previously unexplored or only hypothesized, and 3) study the impact of each dimension of

populism individually over other outcomes of interest. It opens the possibility of analyzing the

varieties of populist discourse and attitudes, and how different aspects of populism might predict,

or be predicted by, other attitudes and behavior. Moreover, the high cross-national validity will

be an essential tool for the blossoming comparative research on populism across countries and

regions. Researchers in several areas have much to gain from this improved measurement.

References

Agger, Robert. E., Marshall. N. Goldstein, and Stanley. A Pearl. (1961). “Political Cynicism:

Measurement and Meaning.” The Journal of Politics 23(3): 477–506.

Akaike, Hirotogu. (1973). “Information Theory and an Extension of the Maximum Likelihood

Principle.” In Second international Symposium on Information Theory, edited by Petrov,

Boris N. and Frigyes Csaki, 267–281. Akademia Kiado, Budapest.

Akkerman, Agnes, Cas Mudde, and Andrej Zaslove. (2014). “How Populist Are the People?

Measuring Populist Attitudes in Voters.” Comparative Political Studies 47(9): 1324-

1353.

Alemán, José, and Dwayne Woods. (2016). “Value Orientations From the World Values Survey:

How Comparable Are They Cross-Nationally?” Comparative Political Studies 49(8):

1039–1067.

Andreadis, Ioannis, and Yannis Stavrakakis. (2017) “European Populist Parties in Government:

How Well are Voters Represented? Evidence from Greece.” Swiss Political Science

Review, online first. doi: 10.1111/spsr.12255.

Ariely, Gal, and Eldad Davidov. (2010). “Can we Rate Public Support for Democracy in a

Comparable Way? Cross-National Equivalence of Democratic Attitudes in the World

Value Survey.” Social Indicators Research 104(2): 271–286.

Aslanidis, Paris. (2016). “Is Populism an Ideology? A Refutation and a New Perspective.”

Political Studies 64(15): 88–104.

Asparouhov, Tihomir, and Bengt Muthén. (2014). “Multiple-group Factor Analysis Alignment.”

Structural Equation Modeling: A Multidisciplinary Journal 21(4): 495–508.

Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. (2012). “Evaluating Online Labor

Markets for Experimental Research: Amazon.com’s Mechanical Turk.” Political Analysis

20(3): 351–368.

Bozdogan, Hamparsum. (2000). “Akaike’s Information Criterion and Recent Developments in

Information Complexity.” Journal of Mathematical Psychology 44(1): 62–91.

Browne, Michael W. (2001). “An Overview of Analytic Rotation in Exploratory Factor

Analysis.” Multivariate Behavioral Research 36(1): 111–150.

Cacioppo, John T., and Richard E. Petty. (1982). “The Need for Cognition.” Journal of

Personality and Social Psychology 42(1): 116–131.

Canovan, Margaret. (1981). Populism. Junction Books, London.

Canovan, Margaret. (1999). “Trust the People! Populism and the Two Faces of Democracy.”

Political Studies 47: 2–16.

Cattell, Raymond. B. (1966). The Scree Test for the Number of Factors. Multivariate Behavioral

Research 1(2): 245–276.

Davidov, Eldad, Jan Cieciuch, Bart Meuleman, Peter Schmidt, René Algesheimer, and Mirjam

Hausherr. (2015). “The Comparability of Measurements of Attitudes toward Immigration

in the European Social Survey: Exact versus Approximate Measurement Equivalence.”

Public Opinion Quarterly 79(SI): 244–266.

de la Torre, Carlos. (2013). “Technocratic Populism in Ecuador.” Journal of Democracy 24(3):

33–46.

Dryzek, John S. and Joffrey Berejikian. (1993). “Reconstructive Democratic Theory.” The

American Political Science Review 87(1): 48–60.

Elchardus, Marc and Bram Spruyt. (2016). “Populism, Persistent Republicanism and Declinism:

An Empirical Analysis of Populism as a Thin Ideology.” Government & Opposition 51:

111–33.

Fiske, Susan. T., Amy J. Cuddy, Peter Glick, and Jun Xu. (2002). “A Model of (Often Mixed)

Stereotype Content: Competence and Warmth Respectively Follow From Perceived

Status and Competition.” Journal of Personality and Social Psychology 82 (6): 878-902.

Funke, Friedrich. (2005). “The Dimensionality of Right-Wing Authoritarianism: Lessons from

the Dilemma between Theory and Measurement.” Political Psychology 26(2): 195-218.

Hawkins, Kirk. (2009). “Is Chavez Populist?: Measuring Populist Discourse in Comparative

Perspective.” Comparative Political Studies 42(8): 1040–1067.

Hawkins, Kirk. (2010). Venezuela’s Chavismo and Populism in Comparative Perspective.

Cambridge University Press, Cambridge.

Hawkins, Kirk A., Scott Riding, and Cas Mudde. 2012. “Measuring Populist Attitudes.” Working

Paper Series on Political Concepts, ECPR Committee on Concepts and Methods.

Hawkins, Kirk A., and Cristóbal Rovira Kaltwasser. (2017). The Ideational Approach to

Populism. Latin American Research Review 52(4): 513–528.

Hibbing, John R. and Elizabeth Theiss-Morse. (2002). Stealth Democracy: American’s Beliefs

About How Government Should Work. Cambridge University Press, New York.

Horn, John L. (1965). A Rationale and Test for the Number of Factors in Factor Analysis.

Psychometrika 30(2): 179–185.

Ionescu, Ghita and Ernest Gellner, editors. (1969). Populism: Its Meanings and National

Characteristics. Weidenfeld and Nicholson, London.

Jöreskog, Karl G. (1971). “Simultaneous Factor Analysis in Several Populations.”

Psychometrika 36: 409–26.

Kline, Rex B. (2016). Principles and Practice of Structural Equation Modeling. The Guilford

Press, New York, 4th. edition.

Laclau, Ernesto. (2005). On Populist Reason. Verso, London.

Levitsky, Steven. and James Loxton. (2013). “Populism and Competitive Authoritarianism in the

Andes.” Democratization 20(1): 107–136.

March, Luke. (2007). “From Vanguard of the Proletariat to Vox Populi: Left-Populism as a

’Shadow’ of Contemporary Socialism.” SAIS Review 27(1): 63–77.

Moffitt, Benjamin. (2015). “How to Perform Crisis: A Model for Understanding the Key Role of

Crisis in Contemporary Populism.” Government and Opposition 50(2): 189–217.

Moffitt, Benjamin. (2016). The Global Rise of Populism. Stanford: Stanford University Press.

Mokken, Rob J. (1971). A Theory and Procedure of Scale Analysis. The Hague: Mouton & Co.

Mouffe, Chantal. (2005). “The ‘End of Politics’ and the Challenge of Right-Wing Populism.” In

Populism and the Mirror of Democracy, edited by F. Panizza. London: Verso.

Mudde, Cas. (2004). “The Populist Zeitgeist.” Government and Opposition 39(4): 542-563.

Mudde, Cas. (2007). Populist Radical Right Parties in Europe. Cambridge: Cambridge

University Press.

Mudde, Cas, and Cristóbal Rovira Kaltwasser. (2013). “Populism.” In Oxford Handbook of

Political Ideologies, edited by M. Freeden, L. T. Sargent, and M. Stears, 493-512.

Oxford: Oxford University Press.

Mudde, Cas, and Cristóbal Rovira Kaltwasser. (2017). Populism: A Very Short Introduction.

Oxford University Press, New York.

Müller, Jan-Werner. (2016). What Is Populism?. Philadelphia: University of Pennsylvania Press.

Muraki, Eiji. (1992). “A Generalized Partial Credit Model: Application of an EM Algorithm.”,

Applied Psychological Measurement 16(2): 159-176.

O’Donnell, Guillermo. (1994). “Delegative Democracy.” Journal of Democracy 5(1): 55–69.

Ostiguy, Pierre. (2017). “Populism: A Socio-cultural Approach.” In The Oxford Handbook of

Populism, edited by C. Rovira Kaltwasser, P. A. Taggart, P. Ochoa Espejo, and P.

Ostiguy. Oxford: Oxford University Press.

Panizza, Francisco. (2005). Populism and the Mirror of Democracy. London: Verso.

Pappas, Takis. S. (2014). “Populist Democracies: Post-authoritarian Greece and Post-communist

Hungary.” Government and Opposition 49(1): 1–23.

Pappas, Takis. S. (2016). “Are Populist Leaders ‘Charismatic’? The Evidence from Europe”.

Constellations 23(3): 378-390.

Peer, Eyal, Sonam Samat, Laura Brandimarte, and Alessandro Acquisti. (2017). “Beyond the

Turk: Alternative Platforms for Crowdsourcing Behavioral Research.” Journal of

Experimental Social Psychology 70: 153-163.

Piurko, Yuval, Shalom H. Schwartz, and Eidad Davidov. (2011). “Basic Personal Values and the

Meaning of Left-Right Political Orientations in 20 Countries.” Political Psychology

32(4): 537–561.

Preacher, Kristopher J., Guangjian Zhang, Cheongtag Kim, and Gerhard Mels. (2013).

“Choosing the Optimal Number of Factors in Exploratory Factor Analysis: A Model

Selection Perspective.” Multivariate Behavioral Research 48(1): 28–56.

Rooduijn, Matthijs. (2014). “The Nucleus of Populism: In Search of the Lowest Common

Denominator.” Government and Opposition 49(4): 573–99.

Rooduijn, Matthijs. (2014b). “Vox Populismus: A Populist Radical Right Attitude among the

Public?” Nations and Nationalism 20(1): 80–92.

Rooduijn, Matthijs, Sarah L. de Lange, and Wouter van der Brug. (2014). “A Populist Zeitgeist?

Programmatic Contagion by Populist Parties in Western Europe.” Party Politics 20(4):

563-575.

Rydgren, Jens. (2004). “Explaining the Emergence of Radical Right-Wing Populist Parties: The

Case of Denmark.” West European Politics, 27(3):474–502.

Satorra, Alberto. and Peter M. Bentler. (2001). “A Scaled Difference Chi-square Test Statistic

for Moment Structure Analysis.” Psychometrika 66: 507-514.

Schulz, Anne, Philipp Müller, Christian Schemer, Dominique S. Wirz, Martin Wettstein, and

Werner Wirth. (2017). Measuring Populist Attitudes on Three Dimensions. International

Journal of Public Opinion Research. Online first. doi: .

Schwarz, Gideon. (1978). “Estimating the Dimension of a Model.” The Annals of Statistics, 6(2):

461–464.

Seligson, Michael A. (2007). “The Democracy Barometers: The Rise of Populism and the Left in

Latin America.” Journal of Democracy 18(3): 81-95.

Simons, Daniel J. and Christopher F. Chabris. (2012). “Common (Mis)Beliefs About Memory: a

Replication and Comparison of Telephone and Mechanical Turk Survey Methods.” PloS

one 7(12): 1–5.

Spruyt, Bram. (2014). “An Asymmetric Group Relation? An Investigation into Public

Perceptions of Education-based Groups and the Support for Populism.” Acta Politica, 49:

123–143.

Spruyt, Bram, Gil Keppens, and Filip Van Droogenbroeck. (2016). “Who Supports Populism and

What Attracts People to It?” Political Research Quarterly 69(2): 335-346.

Stanley, Ben. (2011). “Populism, Nationalism, or National Populism? An Analysis of Slovak

Voting Behaviour at the 2010 Parliamentary Election.” Communist and Post-Communist

Studies 44(4): 257–270.

Stavrakakis, Yannis, Ioannis Andreadis, and Giorgos Katsambekis. (2016). A New Populism

Index at Work: Identifying Populist Candidates and Parties in the Contemporary Greek

Context. European Politics and Society, advance view,

http://dx.doi.org/10.1080/23745118.2016.1261434.

Taggart, Paul A. (2000). Populism. Buckingham: Open University Press.

Van Der Ark, L. Andries. (2012). New Developments in Mokken Scale Analysis in R. Journal of

Statistical Software, 48(5).

Van Hauwaert, Steven, Christian Schimpf, and Flavio Azevedo. (2016). “Measuring Populism as

a Demand-side Phenomenon: A Psychometric Evaluation of Populist Attitudes across

Europe.” Team Populism January Conference: The Causes of Populism, Jan 28–30,

Provo (UT).

Weyland, Kurt. (2001). “Clarifying a Contested Concept: Populism in the Study of Latin

American Politics.” Comparative Politics 34(1): 1–2