Transitive in Our Preferences, But Transitive in Different Ways:An Analysis of Choice Variability
Daniel R. CavagnaroCalifornia State University, Fullerton
Clintin P. Davis-StoberUniversity of Missouri
Numerous empirical studies have examined the question of whether transitivity ofpreference is a viable axiom of human decision making, but they arrive at differentconclusions depending on how they model choice variability. To bring some consis-tency to these seemingly conflicting results from the literature, this article movesbeyond the binary question of whether or not transitivity holds, asking instead: In whatway does transitivity hold (or not hold) stochastically, and how robust is (in)transitivepreference at the individual level? To answer these questions, we reanalyze data from7 past experiments, using Bayesian model selection to place the major models ofstochastic (in)transitivity in direct competition, and also carry out a new experimentexamining transitivity under direct time pressure constraints. We find that a majority ofindividuals satisfy transitivity, but according to different stochastic specifications (i.e.,models of choice variability), and that individuals are largely stable in their transitivity“types” across decision making environments. Thus, transitivity of preference, as wellas the particular type of individual choice variability associated with it, appear to berobust properties at the individual level.
Keywords: transitivity, preference reversal, Bayesian model selection, linear ordering polytope
Few axioms of decision making have gar-nered as much attention in the behavioral sci-ences literature as transitivity of preference. LetC be a nonempty set of choice alternatives with|C | � 3, and let a, b, c be any three distinctelements of C. A decision maker (DM) satisfiesthe axiom of transitivity if, and only if, for anytriple a, b, c, if the DM prefers a to b and prefersb to c then the DM also prefers a to c. Transi-tivity has been cast as a normative property ofdecision making via arguments such as themoney pump (e.g., Anand, 1993).1 In addition,transitivity of preference is a necessary assump-
tion for nearly all theories of utility (see Luce,2000, for a discussion).
Despite its centrality in modeling choice be-havior, there exists a fundamental gap betweenthe axiom’s definition and a researcher’s abilityto empirically test it against a DM’s observedchoices. The axiom of transitivity is definedalgebraically, hence deterministically, whilechoice data are intrinsically probabilistic. At-tributed to either a lapse of attention, a mistake,or a “change of mind,” DMs rarely make iden-tical choices across repeated presentations ofthe same set of choice alternatives (e.g., Hey,2005). This choice variability requires a re-searcher to translate the axiom of transitivityinto a testable statement involving choice prob-abilities, or similar probabilistic components
1 Briefly, the money pump argument states that a DMwith intransitive preferences would be driven out of themarket by clever traders who could sequentially trade itemswithin the DM’s intransitive cycle, for example, trading afor b, b for c, and then c for a again. In this sequence, theintransitive DM ends up with the same item he started with,but poorer because of the transaction costs.
Daniel R. Cavagnaro, Mihaylo College of Business andEconomics, California State University, Fullerton; ClintinP. Davis-Stober, Department of Psychological Sciences,University of Missouri.
Correspondence concerning this article should be ad-dressed to Daniel R. Cavagnaro, Department of InformationSystems and Decision Sciences, Mihaylo College of Busi-ness and Economics, California State University Fullerton,Fullerton, CA 92834-6848. E-mail: [email protected]
(see Wilcox, 2008, for a summary). Within theliterature, there are three primary stochasticmodels of transitivity that accomplish this, ei-ther derived from basic principles or as neces-sary assumptions for various decision makingmodels. They are weak stochastic transitivity(WST; Davidson & Marschak, 1959), moderatestochastic transitivity (MST; Grether & Plott,1979), and strong stochastic transitivity (SST;Tversky & Russo, 1969). However, there is nouniversal agreement on which stochastic model“best” represents the transitivity of preferenceaxiom; either from the perspective of whichstochastic model is the most appropriate opera-tionalization of the algebraic axiom (Regenwet-ter, Dana, & Davis-Stober, 2011) or which sto-chastic model best describes DMs’ actualchoices (Myung et al., 2005; Regenwetter et al.,2010).
A host of previous studies have claimed toshow that even the most lenient specification oftransitivity, WST, is violated by participants inlaboratory experiments (see Mellers & Biagini,1994, for a review). However, Regenwetter,Dana, and Davis-Stober (2011) provided a thor-ough critique of many past empirical tests oftransitivity, highlighting several conceptualchallenges associated with different stochastictreatments of transitivity. In light of their argu-ments, they endorsed the use of a mixture modelof transitive preference (MMTP) as a probabi-listic specification of the axiom. Regenwetter etal. reanalyzed a large number of prior data setsfrom studies designed to elicit intransitivechoice patterns, along with their own newlycollected data, and found that the MMTP waswell supported. Regenwetter et al. thus con-cluded that the axiom of transitivity holds, over-turning the conclusions of numerous prior stud-ies (e.g., Montgomery, 1977; Ranyard, 1977;Tversky, 1969).
One shortcoming of Regenwetter et al.’sstudy is that, while their advanced statisticalmethods avoided many of the pitfalls in prioranalyses, they could still only evaluate thegoodness-of-fit of their mixture model and wereunable to directly compare its descriptive accu-racy to other stochastic models of transitivity(Davis-Stober, 2009). It is therefore possiblethat other stochastic models may account forthese data as well as, or better than, MMTP.Should this be true, the empirical conclusions ofRegenwetter et al. would necessarily be drawn
into question. Such a conclusion would alsoraise the substantive questions of whether DMsare better fit by “mixture models” or by fixedpreference with random error models, or bysome combination thereof (Davis-Stober &Brown, 2011; Loomes et al., 2002).
Fortunately, there are Bayesian methodol-ogies that are well suited to nonnested modelcomparison (Lee & Wagenmakers, 2014,chapter 7). An important difference betweenthe classical and Bayesian statistical method-ologies is that the latter can quantify andcompare the evidence in favor of each model,rather than simply setting up each model as anull hypothesis and then rejecting or failing-to-reject them. For example, Myung et al.(2005) developed a Bayesian model selectionframework based upon the Bayesian p valueand the Deviance Information Criterion(DIC), and used it to evaluate WST, MST,and SST using the well-studied Tversky(1969) data. While none of the three modelswere well supported for six out of eight sub-jects, in agreement with Tversky’s originalanalysis, they found strong support for SSTfrom the other two subjects (see also a similarreanalysis by Karabatsos, 2006).
Myung et al. did not consider MMTP intheir analysis, nor did they analyze other datasets from the literature. On the other hand,while Regenwetter et al. (2010) and Regen-wetter, Dana, and Davis-Stober (2011) foundthat neither the Tversky data nor their owndata were inconsistent with MMTP or WST,they did not do a statistical model comparisonof MMTP and WST, nor did they considerMST and SST. This raises the question ofwhether WST, MST, or SST could provide asuperior explanation of these data sets com-pared with MMTP.
In this article, we place the four major modelsof stochastic transitivity, WST, MST, SST, andMMTP, in direct competition with one anotherby reanalyzing data from seven experiments infive prior studies from the literature. We rean-alyze, using Bayesian model selection, the datafrom Tversky (1969), three experimental repli-cations of Tversky (1969) run by Regenwetter,Dana, and Davis-Stober (2011), two experi-ments from Ranyard (1977), and experimentsfrom Montgomery (1977) and Tsai and Böck-enholt (2006). The statistical analysis we carryout to compare the different models of transi-
103TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
tivity in these datasets is based on the orderconstrained methodology of Klugkist and Hoi-jtink (2007), which yields the Bayes factor foreach model pair (Kass & Raftery, 1995).2 Theseanalyses add to the literature by evaluating notjust the goodness-of-fit of these models of tran-sitivity, but also comparing them head-to-headto determine which model(s), if any, best rep-resent a DM’s choice behavior.
Like Regenwetter, Dana, and Davis-Stober(2011), we find that violations of transitivity arefairly rare and occur primarily in the studies thatactively sought out participants with a propen-sity toward intransitive behavior. In support ofRegenwetter, Dana, and Davis-Stober (2011),we find that MMTP generally provides a betterexplanation of individual data than does WST.However, we find substantial heterogeneityamong which models best fit individual DMs.Our results show that the preferred model forthe largest number of participants is SST, incontrast to prior studies using classical statisti-cal methods (e.g., Mellers & Biagini, 1994;Rieskamp et al., 2006). We also conduct a meta-analysis using a latent Dirichlet allocationmodel to estimate the distribution of best-fittingmodels of (in)transitivity across studies. At thislevel, we also find substantial heterogeneityamong the best-fitting models. Therefore, weconclude that most individuals are transitive,but how they are transitive can vary dramati-cally from individual to individual. Because ofthese individual differences, we conclude that a“one-size-fits-all” approach to modeling choicevariability is unlikely to describe human choicebehavior adequately.
To examine the robustness of our findingsfrom this reanalysis, we conducted a partialreplication of the Regenwetter et al. study withthe additional manipulation of a “time pressure”condition. The gamble stimuli we use in thisstudy were designed to induce intransitive pref-erence based upon a lexicographic semiorderstrategy, similar to Tversky (1969). Prior liter-ature has suggested that DMs are more likely toapply lexicographic strategies when the time tomake a decision is greatly limited (Rieskamp &Hoffrage, 1999, 2008). We investigatedwhether this additional manipulation could in-duce individuals to violate transitivity. While ahandful of participants were induced to makeintransitive choice under this additional condi-tion, most participants were classified as transi-
tive according to one of the four models. Re-markably, while we again found substantialvariability in the best-fitting models of transi-tivity across individuals, most participants wereconsistently best described by the same modelof transitivity (e.g., SST) for both timed anduntimed conditions. These results strongly sup-port the conclusion that DMs are generally tran-sitive in their preferences, albeit according todifferent stochastic models, and that individualsare largely stable in their transitivity “type”across different environments.
The rest of this article is organized as fol-lows. We first introduce basic assumptions andmodeling definitions and define the four proba-bilistic specifications of transitivity that we con-sider. We then describe the Bayesian modelselection procedure and present the results ofour reanalysis for each prior dataset. Next, wedescribe the method and results of the newempirical study as well as a meta-analysisacross studies. We conclude with a general dis-cussion of issues related to this work.
Preliminary Definitions and ModelingAssumptions
In a typical 2-alternative forced choice(2AFC) experiment, participants are presentedwith pairs of items and asked to choose one itemfrom each pair. Pairs of items are denoted (a, b),for all (unordered) pairs (a, b) � C. In a com-plete experiment, each participant is presentedwith each possible pair of items at least once.We write Nab for the number of times that aparticipant is presented with item pair (a, b). Aparticipant’s responses can be summarized bythe vector n � �nab�a,b�C,a�b where nab is thenumber of times that a participant chooses itema from the pair (a, b). Hence, Nab � nab is thenumber of times that the participant chose itemb from the pair (a, b).
2 The DIC and Bayes factor both account for modelcomplexity so as to avoid preferring overly flexible models(overfitting). However, the Bayes factor also provides areadily interpretable metric of evidence based on the like-lihood of each model, whereas the DIC only provides onlyordinal information on the suitability of the various modelsunder consideration. We also computed the DICs for eachmodel using the method proposed by Myung et al. (2005)and found general agreement between the two statistics.Therefore, we only present the results of the Bayes factoranalysis.
104 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
In the present study, we consider models inwhich a decision maker is assumed to chooseitem a over item b with some fixed probability,denoted Pab. We refer to Pab as the binarychoice probability of a being chosen over b. Inclassical analyses of such models, repeatedchoices on the same stimulus pair, say (a, b), areassumed to be independent Bernoulli trials withprobability of success Pab, with success definedas choosing item a from the pair (a, b) (e.g.,Regenwetter et al., 2010; Regenwetter, Dana, &Davis-Stober, 2011). In addition, the binomialrandom variables themselves are assumed to beindependent of each other. Together, these as-sumptions are commonly referred to as iid sam-pling. Under the iid sampling assumption, thelikelihood function for a set of responses n takesthe following, product-of-binomials form:
f(P |n) � �a,b�C,a�b
�Nab
nab�Pab
nab(1 � Pab)Nab�nab,
(1)
where P � �Pab�a,b�C,a�b, and 0 � Pab � 1, forall pairs (a, b) � C.
For the Bayesian analyses carried out in thepresent study, Equation 1 can be derived fromweaker assumptions than those required for aclassical analysis. In particular, the assumptionthat repeated choices on the same stimulus areindependent may be replaced with the weakerassumption that they are exchangeable (Ber-nardo, 1996; Lindley & Phillips, 1976). Theassumption of exchangeability means that be-fore the experiment begins we believe that thelikelihood of a being chosen on the ith presen-tation of (a, b) is equal to the likelihood of abeing chosen on the jth presentation of (a, b),for any (a, b) and any i, j � 0. Note thatexchangeability does not imply statistical inde-pendence because the conditional probability ofa being chosen on presentation I � 1 of (a, b),given the choices on the first I presentations of(a, b), will change through Bayesian updating.
It is important to note that these assumptions,like all modeling assumptions, are almost cer-tainly wrong and only an approximation of amore complex reality. However, the assumptionof exchangeability (or independence) is usefulinsofar as adopting it allows for a parsimoniousexplanation of the data, and it is at least reason-able for the experiments we consider in our
analyses given the design measures that weretaken to mitigate memory and order effects.Nevertheless, we will test the validity of theexchangeability assumption to the extent that itis possible given the condition of each dataset(e.g., some datasets do not include the originalchoice sequences, which are required for testsof independence/exchangeability). Instead oftesting exchangeability directly, we will testindependence between repeated choices on thesame stimulus pair using a method proposed bySmith and Batchelder (2008). Since indepen-dence implies exchangeability, data that passthis test of independence would also pass a testof exchangeability.3 Other modeling ap-proaches that do not assume that choices areindependent or exchangeable have been pro-posed in the literature, but these models arenecessarily more complex in order to captureadditional structure in the data. Such methodsare discussed further in the general discussion.
Stochastic Models of Transitivity
We now define the four probabilistic specifi-cations of transitivity under consideration:WST, MST, SST, and MMTP. Each specifica-tion is defined by a set of restrictions on thechoice probabilities for each choice pair. Theserestrictions can be represented geometrically assubsets of the unit hypercube of dimensiond �
k�k � 1�2 , where k � |C |. Because each binary
choice probability Pab is required to lie in theinterval [0, 1], the set of all distinct vectors Pof binary choice probabilities is [0, 1]d. Eachprobabilistic specification of transitivity thus re-stricts P to some subset � � �0, 1�d of choiceprobabilities that are consistent with that spec-ification of the axiom. After defining the subset
3 Testing statistical independence between choices ondifferent stimulus pairs has been the subject of much debatein the recent literature. A test proposed by Birnbaum (2012)showed that the iid assumption may not be justified in thedata from Regenwetter, Dana, and Davis-Stober (2011), butthis test has been criticized by Cha et al. (2013) for, amongother things, frequently, falsely rejecting iid sampling whenit actually holds. Moreover, the test may detect violations ofiid that do not significantly affect the substantive conclu-sions of the iid-based model (Regenwetter, Dana, Davis-Stober, & Guo, 2011). Indeed, while these research groupshave debated the validity of the iid assumption, neithergroup is claiming that iid violations in existing data have ledto alternative substantive conclusions regarding transitivity.
105TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
corresponding to each specification, we willcast each specification as a Bayesian model byplacing a prior distribution over the associatedsubset and pairing it with the likelihood func-tion in Equation 1.
WST
A DM satisfies WST if, and only if, for any threedistinct choice alternatives, a, b, c, if the probabilitiesof choosing a over b and b over c are at least one
half, the probability of choosing a over c is also atleast one half. Thus, �WST is defined as the solutionspace to the following set of implications,
Pab � .5 Pbc � .5 ) Pac � .5,
∀a, b, c � C, (2)
where “” denotes conjunction.Figure 1a displays �WST for the case when
k � 3. Geometrically, WST forms a union of
Figure 1. a–d depict the binary choice probabilities that satisfy the constraints of the fourprobabilistic specifications of transitivity, for three choice alternatives, and correspond to theregions of the unit cube defined by weak stochastic transitivity (WST), moderate stochastictransitivity (MST), strong stochastic transitivity (SST), and mixture model of transitivepreference (MMTP), respectively. For more than three choice alternatives, the regions arehigher-dimensional. The volume of each region, for 3-, 4-, and 5-choice alternatives isdisplayed in the lower left-hand corner of each figure. The relationships between the regionscan be described visually by the nature of how each one “removes” the intransitive cornersfrom the unit cube. For example, at the intransitive vertex, Pab � 1, Pbc � 1, Pac � 0 (topfront corner), WST removes a single half-unit cube, while MST and SST remove progres-sively larger chunks of the space, leaving less of the space consistent with those specifications.In this depiction, MMTP bears some similarity to MST; however, MMTP is convex whileMST is not.
106 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
half-unit cubes (see also Iverson & Falmagne,1985). WST is implied by a large class of nor-mally distributed random utility models, such asmany Thurstonian models (Halff, 1976), as wellas by decision field theory (Busemeyer &Townsend, 1993; Roe et al., 2001).
MST
MST presents a more restrictive probabilisticinterpretation of the transitivity axiom com-pared to WST. A DM satisfies MST if, and onlyif, for any choice triple a, b, c, if the DMchooses a over b with probability greater than orequal to one half, and chooses b over c withprobability greater than or equal to one half,then he or she also chooses a over c with prob-ability greater than or equal to the minimum ofPab and Pbc. Thus, �MST is defined as the solu-tion space to the following set of implications
Pab � .5 Pbc � .5 ) Pac � min{Pab, Pbc},
∀a, b, c � C. (3)
The geometric representation of MST in theunit hyper-cube is a single nonconvex polyhe-dron. Figure 1b displays �MST in the unit hyper-cube when k � 3. MST is an implication of theprobabilistic model of multiattribute choice,Elimination-by-Aspects (Tversky, 1972).
SST
SST is a yet more restrictive take on weakand moderate stochastic transitivity. A DM sat-isfies SST if, and only if, for any choice triple a,b, c, if the DM chooses a over b with probabilitygreater than or equal to one half, and chooses bover c with probability greater than or equal toone half, then he or she must choose a over cwith probability greater than or equal to themaximum of Pab and Pbc. Thus, �SST is definedas the solution space to the following set ofimplications,
Pab � .5 Pbc � .5 ) Pac � max{Pab, Pbc},
∀a, b, c � C. (4)
Similar to MST, SST can be described as asingle nonconvex polyhedron in the appropriate
probability space. Geometrically, SST is aproper subset of both moderate- and weak sto-chastic transitivity. Figure 1c shows a plot of�SST when k � 3.
SST is equivalent to the properties of inde-pendence, that is, ∀a, b, c, d � C, Pab �Pcb ⇔ Pad � Pcd, and substitutability, that is,∀a, b, c � C, Pac � Pbc ⇔ Pab � 1
2 (Tversky &Russo, 1969). There have been numerous quan-titative tests of SST (and independence), gener-ally rejecting this restrictive model of transitivepreference (Busemeyer, 1985; Mellers & Bi-agini, 1994). See also Rieskamp et al. (2006) fora review of related principles and theoreticalaccounts.
MMTP
MMTP assumes that each DM makes choicesaccording to a probability distribution over acollection of transitive preference states. Unlikethe preceding specifications, which were de-fined in terms of probabilistic deviations from afixed preference state, MMTP allows a DM tovary his or her preferences across time points.This type of model is also known as a “mixture”or “random preference” model (Loomes & Sug-den, 1995).
Formally, let C be a collection of choice al-ternatives and write T for the set of all com-plete, asymmetric, transitive binary relations onC. MMTP is defined by a discrete probability distri-bution � over T. A vector P � �Pab�a,b�C,a�b ofchoice probabilities satisfies MMTP if and only ifthere exists a discrete probability distribution � overT such that
Pab � �T�T (a,b)�T �
(T)
for all a, b � C, where �(T) is the probabilitythat a person is in the transitive state of prefer-ence T. Under two-alternative forced choice, thefollowing equations are a minimal descriptionof �MMTP for up to 5 alternatives:
Pab � Pbc � Pac � 1 ∀(a, b, c) � C, (5)
Pab � 0 ∀(a, b) � C, (6)
Pab � Pba � 1 ∀(a, b) � C. (7)
107TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
The inequalities defined by (5) are known asthe triangle inequalities. The other two equa-tions are implied by the 2AFC setup. This list ofinequalities completely characterizes �MMTP forup to five choice objects. Figure 1d shows a plotof �MMTP when k � 3.4
Statistical Methodology
The goal in comparing these four probabilis-tic specifications of transitivity is to identify themodel that best captures the underlying regular-ities in a set of choice data. However, a model’sability to simply fit a given set of empirical datamay have more to do with the model’s com-plexity than with its capturing something mean-ingful about the data-generating process. Forexample, a maximally complex model of binarychoice would be one that places no restrictionson the allowable choice probabilities. Such amodel could automatically fit any set of dataperfectly but, because it is so flexible, it is likelyto overfit the data and thus generalize poorly tofuture, unseen data generated from the sameunderlying mental process (Myung, 2000;Myung & Pitt, 2002). Therefore, in choosingamong probabilistic specifications of transitiv-ity, we must consider not only how well thespecification fits the observed data, but also howcomplex it is. Essentially, we should favor themost restrictive model that provides an ade-quate fit to the data.
The challenge of selecting among models inthis way, by trading of goodness-of-fit and com-plexity, has been considered at length in thestatistics literature, and many different measure-ment criteria have been proposed. Among themost principled of these criteria is the Bayesfactor (Kass & Raftery, 1995). The Bayes factoris defined as the ratio of the marginal likeli-hoods of two models, derived from Bayesianupdating, and provides a direct and naturallyinterpretable metric for model selection. ABayes factor of 10, for example, means that thedata are 10 times more likely to have occurredunder one model than the other. Bayes factorscan also be interpreted within an established“magnitude” scale (Jeffreys, 1961). WhileBayes factors are very difficult to compute ingeneral, Klugkist and Hoijtink (2007) have pre-sented a specialized procedure for computingthem for inequality constrained models, such asthose constructed by Myung et al. to represent
decision making axioms (see also Wagenmak-ers et al., 2010, for a related approach).
The first step in the procedure is to casteach probabilistic specification as a Bayesian,inequality constrained model. This is detailedin the next subsection. Following that, wepresent the details of our implementation ofthe rest of the procedure.
Bayesian Model Specification
Bayesian model selection requires that thefour probabilistic specifications of transitivitybe cast as Bayesian models (with a likelihoodfunction and a prior) that instantiate the restric-tions of each specification on binary choiceprobabilities. So far, we have defined restric-tions on binary choice probabilities, but speci-fying models requires defining priors and a like-lihood function to manifest those restrictions.This can be accomplished following the ap-proach of Myung et al. (2005), in which priorsare defined with support over just those choiceprobabilities that are consistent with each spec-ification. Full models of transitivity follow nat-urally by combining each order-constrainedprior with the likelihood function in Equation(1). Thus, it is the prior, not the likelihoodfunction, that instantiates the restrictions im-posed by each probabilistic specification, andwhich thereby distinguishes the different mod-els of transitivity from one another (Vanpaemel,2010).
Formally, if t is any probabilistic specifica-tion of transitivity and �t � �0, 1�d is the subsetof binary choice probabilities consistent with t,we construct the Bayesian model Mt with theprior distribution
�(P | Mt) � ct if P � �t,0 otherwise
where ct is a positive constant such that���P Mt� dP � 1. In other words, as in Myunget al. (2005), this defines a uniform distributionover choice probabilities consistent with the
4 Once the number of choice objects exceeds five, theabove equations provide only a partial characterization. Acomplete enumeration of a minimal set of defining inequal-ities for this specification as a function of |C | is currently anunsolved problem (Fiorini, 2001).
108 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
model. The value of ct is simply the reciprocalof the volume of �t, which depends on thenumber of choice alternatives and can be esti-mated via Monte Carlo simulation. Thevolumes of �WST, �MST, �SST and �MMTP for3, 4, and 5 choice alternatives are given inFigure 1.5
In addition to the four models of transitivity,we also define a baseline model, denoted M1,with no restrictions on choice probabilities. It isdefined by ��PM1� � 1, P � �0, 1�d; that is, auniform prior over the entire space of possiblechoice probabilities. We will refer to this modelas the “encompassing” model, as the four mod-els of transitivity are, by definition, nestedwithin it. Because the encompassing model al-lows, but does not assume transitivity, it willserve as a benchmark against which to comparethe models that do assume some form of tran-sitivity. Essentially, M1 represents the hypothe-sis that violations of transitivity are possible.Strong evidence in favor of M1 over any othermodel indicates that the restrictions imposed bythat model are not supported by the data. Thus,evidence in support of M1 over a model oftransitivity constitutes evidence against thatparticular specification of transitivity.
Klugkist and Hoijtink (2007) Procedure
The procedure of Klugkist and Hoijtink(2007) takes advantage of the fact that all of themodels are nested in M1 by computing theBayes factor of each model relative to M1. For-mally, the Bayes factor for M1 over Mt, denotedBFt1, is defined as the ratio of the two marginallikelihoods,
BFt1 �p(n |Mt)
p(n |M1)�
�Pr(n |P)�(P |Mt)dP
�Pr(n |P)�(P |M1)dP. (8)
While BFt1 is defined with regard to the en-compassing model, a Bayes factor for anymodel pair can be constructed by taking theratio of the BFt1 values for the two models ofinterest. For example, the Bayes factor for M2
over M3 is the ratioBF21
BF31.
As described more fully in Klugkist and Hoi-jtink (2007), Equation (8) can be further sim-plified. In particular, the Bayes factor under this
inequality-constrained framework can be de-scribed as the ratio of two proportions: the pro-portion of the encompassing prior in agreementwith the constraints of Mt and the proportion ofthe encompassing posterior distribution inagreement with the constraints of Mt. This sim-plification gives,
BFt1 �ct
dt, (9)
where 1ct
is the proportion of the encompassingprior in agreement with the constraints of Mtand 1
dtis the proportion of the encompassing
posterior distribution in agreement with theconstraints of Mt. Given that all of our stochas-tic models of transitivity are full-dimensional inthe unit hypercube, [0, 1]d, the proportion 1
ctis
simply the volume of the parameter space sat-isfied by Mt. For WST, this volume calculationcan be done analytically (Iverson & Falmagne,1985). For the remaining models of stochastictransitivity, we estimated the appropriate vol-umes via a rejection sampling Monte Carlo al-gorithm using a uniform distribution. We cal-culated the 1
dtterms using standard Monte Carlo
sampling methods (Gelman et al., 2004).Given that we are specifying a prior distribu-
tion, one could ask how sensitive the choice ofprior is to the resulting Bayes factors. Klugkistand Hoijtink (2007) demonstrate that this meth-odology is relatively robust to the choice ofprior, especially when the models of interest aredefined solely in terms of inequality constraints(as opposed to some constraints being equalityor “about” equality constraints). See Klugkistand Hoijtink (2007) for a full discussion.
Reanalysis of Prior Studies
In this section, we present the results of ourreanalysis across all considered datasets. A fulldescription of each dataset can be found in theonline supplement. In general, the datasets are
5 Regarding the overlap of the four models, SST is nestedwithin MST which is nested within WST. For 3-5 choicealternatives, MMTP contains both SST and MST as nestedsubsets. Thus, only MMTP and WST have a non-nestedrelationship. Regenwetter et al. (2010) calculated the sharedvolume of MMTP and WST to be .6251, .1991, and .0288for 3, 4, and 5 choice alternatives, respectively.
109TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
all the results of replications of the Tversky(1969) experiment, aside from minor proceduralvariations.6 Each experiment collected choicedata on every possible pairwise comparisonover a set of either four or five total choicealternatives (e.g., gambles). To estimate indi-vidual-level preferences, each pair was pre-sented multiple times to each individual (sepa-rated by decoy stimuli). Thus, the data consistof choice proportions on each possible pair, foreach participant. Across all five studies, therewere seven such experiments: one each fromTversky (1969); Montgomery (1977); Ranyard(1977), and Tsai and Böckenholt (2006), andthree from Regenwetter, Dana, and Davis-Stober (2011). The latter three are referred to as‘Regenwetter Cash I,’ ‘Regenwetter Cash II,’and ‘Regenwetter Noncash,’ following the no-tation of the original study. Across these sevenexperiments, there are 81 individual-level vec-tors of choice proportions.
Cha et al. (2013) carried out a statistical testof the independence assumption on the datasetsfrom Regenwetter, Dana, and Davis-Stober(2011) using the test of Smith and Batchelder(2008) and showed that the independence as-sumption was justified in those datasets. Thedatasets from Tversky (1969); Montgomery(1977); Ranyard (1977), and Tsai and Böcken-holt (2006) do not contain the sequential infor-mation that is required to carry out a test of theindependence assumption. However, similar tothe Regenwetter, Dana, and Davis-Stober(2011) study, all of these experiments were car-ried out with design measures in place to limitmemory and order effects, which make the in-dependence assumption reasonable.
As a preliminary assessment of the descrip-tive adequacy of each model, we counted howmany of these vectors of choice proportionssatisfy all of the inequality constraints for eachmodel. Treating the observed choice propor-tions as point estimates of the choice probabil-ities, we say that a model has a “perfect fit” tothe data if the observed choice proportions sat-isfy all of the model’s inequality constraints.Table 1 reports the number of “perfect fits” ofeach model in each data set. Across all studies,about half of all participants had a perfect fit toMMTP or WST (43 and 41 out of 81, respec-tively), and only 11 out of 81 had a perfect fit toSST.
It is not surprising that MMTP and WST havemore perfect fits than either MST or SST, be-cause the former have far less restrictive in-equality constraints (i.e., they occupy largersubsets of the unit-hypercube, so they are morecomplex). However, the numbers of perfect fitsare strikingly large given the relatively smallvolume each model occupies relative to theencompassing model. For example, for fivechoice alternatives, SST occupies less thanthree thousandths of one percent of the space(0.8% for four alternatives), meaning that a DMmaking random choices in the experimentwould have less than a 0.00003 chance of pro-ducing data that perfectly satisfy the constraintsof SST. Yet, 11 of the 81 vectors of choiceproportions fall perfectly within it. By compar-ison, 41 of the 81 vectors of choice proportionsfall within the region consistent with WST,which occupies about 12% of the space for 5alternatives (37% for 4 alternatives). Figure 1gives the volume of each region for 3, 4 and 5choice alternatives.7 All of the studies under
6 One important variation in the Tversky (1969), Mont-gomery (1977), and Ranyard (1977) experiments is thatparticipants were prescreened with preliminary testing ses-sions to select those most likely to exhibit intransitivepreference patterns.
7 Volumes were estimated by drawing 10 million samplesuniformly from the hypercube and counting the proportionof them that were consistent with each model. For example,just 238 out of 10 million sampled choice proportionssatisfied the constraints of SST, resulting in a volume of0.0000238.
Table 1Number of Perfect Fits of Each Model inEach Experiment
Note. MMTP � mixture model of transitive preference;WST � weak stochastic transitivity; MST � moderatestochastic transitivity; SST � strong stochastic transitivity.N is the number of participants in each experiment. M1provides a perfect fit to all participants in all studies, byvirtue of its high complexity, so it is not included.
110 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
consideration had 5 choice alternatives, exceptfor Tsai and Böckenholt (2006), which had 4.
Simply counting choice proportions in thisway speaks only in a very rudimentary way tothe goodness-of-fit of each model. It is not arigorous statistical test of each model becausethe data are assumed to contain binomial sam-pling variation. That is, each observed choiceproportion is modeled as a noisy realizationof a probability representing the participant’strue preference. Because of this sampling vari-ation, it is possible for the observed choiceproportion to be outside of the constraints of amodel even when the underlying probability isactually inside the constraints of the model.This is especially likely when the underlyingprobability is near the boundary of the model.Therefore, to properly account for sampling/choice variability as well as the relative com-plexity of each model, we analyze each datasetunder the Bayesian methodology described ear-lier. Bayes factors were computed using theprocedure of Klugkist and Hoijtink (2007). Foreach model, ct was computed based on10,000,000 samples from the encompassingprior and dt was computed based on 100,000samples from the encompassing posterior. Thevalue of ct for each model was shown in Figure1. Individual-level Bayes factors are reported inthe online supplement.
To interpret the Bayes factor results, we usethe rule-of-thumb cutoff for “substantial” evi-dence according to Jeffreys (1961): BFt1 10�1 ⁄ 2 � 0.316 meaning substantial evidence infavor of the null hypothesis, and BFt1 �101 ⁄ 2 � 3.16 meaning substantial evidenceagainst the null hypothesis. We will say that amodel of transitivity “fails” if its Bayes factor isless than 0.316, for in that case M1, which servesas the null hypothesis because it assumes neithertransitivity nor intransitivity, is at least 3.16 timesmore likely to have generated the data. We willalso say that a model of transitivity is “best” if ithas a Bayes factor of at least 3.16 over M1 and ithas the highest Bayes factor among the models oftransitivity. If the most likely model of transitivityhas a Bayes factor between 0.316 and 3.16 thenthe analysis is inconclusive: none of the modelsare substantially more likely than M1, but neitheris M1 substantially more likely than all of themodels of transitivity, so we say that none of themodels are best.
Using the conventions described above forinterpreting the Bayes factor, we can then ad-dress the following three questions: whichmodel of transitivity fails least often, how oftendo all of the models of transitivity fail, andwhich model of transitivity is best the mostoften? To answer the first question, Table 2reports the number of participants for whomeach model of transitivity fails in each experi-ment. The results show that none of the modelsof transitivity can explain all of the data becauseeach model of transitivity fails for at least someparticipants, across experiments. It is not sur-prising to find that failures were much morefrequent in the Tversky, Montgomery, and Ran-yard studies, in which participants were pre-screened to identify those with a propensity tobehave intransitively. In the studies in whichparticipants were not prescreened, all of themodels of transitivity failed far less frequently.
The next two questions are answered in Table3, which reports the number of times each modelwas best according to the Bayes factor analysis.The first column of this table gives the number ofparticipants in each experiment for whom all ofthe models of transitivity failed (i.e., M1 was best).The table shows that this happened just 16 timesoverall, with 13 of those 16 coming from theexperiments in which participants were pre-screened. That is not to say that these participantsdefinitely had intransitive preferences, only that amodel that does not assume transitivity is substan-tially more likely to have generated the data thanone that does. Among the remaining 65 partici-
Table 2Number of Times in Each Experiment That ThereWas Substantial Evidence Against Each Model ofTransitivity (Bayes Factor Less Than 10�1/2)
Note. MMTP � mixture model of transitive preference;WST � weak stochastic transitivity; MST � moderatestochastic transitivity; SST � strong stochastic transitivity.N is the number of participants in each experiment.
111TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
pants, 57 have Bayes factors of at least 3.16 for atleast one model of transitivity over M1, but there isvery little consensus as far as which model oftransitivity is best. SST is best the most often, butstill just for 25 participants. MMTP, the modelfavored by Regenwetter, Dana, and Davis-Stober(2011), outperformed all other models only 12times, while MST and WST were favored 17times and 3 times, respectively. The fact that WSTwas favored just 3 times suggests that, while WSTmay fit the data well, it is seldom the preferredmodel of transitivity because of its high complex-ity. On the other hand, SST may not always fit thedata well, but when it does, it is the preferredmodel because of its very low complexity.
In summary, while transitivity appears to holdfor many participants, there is a substantial minor-ity for whom it does not. However, the vast ma-jority of violations came from studies that eitherselectively reported data on participants whoseemed to violate transitivity or actively soughtout participants with a propensity toward intran-sitive behavior (i.e., Tversky, Montgomery, andRanyard; see full descriptions in the online sup-plement). This suggests that the transitivity axiomis a reasonable modeling assumption, although itmay not accurately describe the preferences ofevery individual. More importantly, no one modelof transitivity dominated the others across sub-jects. This suggests that different individuals ex-press choice variability in substantively differentways. This is not a minor point, as any theory ofpreferential choice requires a probabilistic compo-nent for modeling choice variability. These resultsstrongly suggest that even if we were to ignore
intransitive individuals when developing theory,we would still be left with heterogeneity in choicevariability. This raises the question of how stableindividuals are with regard to choice variability.Said differently, if the choices of an individual arebest-described by a particular model of stochastictransitivity (e.g., MMTP), would this same modelthen best describe this individual’s choices in adifferent context or occasion? To answer thisquestion and evaluate the robustness of the resultsfrom our reanalysis, we carried out a partial rep-lication of the Regenwetter, Dana, and Davis-Stober (2011) study with additional within-subjectmanipulations of two experimental conditions:stimulus set and time pressure.
Method
We recruited 30 students from the Universityof Missouri to participate in our study, with oneparticipant failing to complete all trials; datafrom this participant were omitted from theanalysis. This number of participants is halfagain as large as that of the Regenwetter, Dana,and Davis-Stober (2011) study, and is morethan triple the number of participants in theTversky (1969) study. Participants were com-pensated as described below. This experimentfollowed a fully crossed, two-by-two, within-subjects design with two stimulus conditionsand two timing conditions (four conditions inall). Each experimental condition consisted of120 presentations of gamble pairs (i.e., stimuli)in a two-alternative forced choice framework(i.e., 120 trials). The two stimulus conditions
Table 3Number of Times in Each Experiment That Each Model Was Best According to the Bayes Factor (HighestBayes Factor Among Models Under Consideration and at Least Substantial Evidence in Support)
Note. MMTP � mixture model of transitive preference; WST � weak stochastic transitivity; MST � moderate stochastictransitivity; SST � strong stochastic transitivity. The “None” column counts the number of times that all of the Bayesfactors were between 10�1/2 and 101/2, i.e., no substantial support for any of the models of transitivity, nor in support ofthe encompassing model. Rows sum to N, the number of participants.
112 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
used different sets of gambles to comprise thegamble pairs (defined below). Each set con-sisted of five distinct gambles, hence 10 possi-ble gamble pairs in each condition. Upon pre-sentation of each gamble pair on the computerscreen, participants indicated which gamblethey preferred by pressing a button on the key-board. Similar to the Regenwetter, Dana, andDavis-Stober (2011) study, we used two sets ofgamble stimuli displayed in “pie” format (seeRegenwetter, Dana, & Davis-Stober, 2011, for adescription). For each timing condition (timedand nontimed, described in more detail below),participants were presented 12 repetitions ofeach gamble pair, counterbalanced so that anygiven gamble appeared on the left- or right-handside of the screen an equal number of times.Hence, per timing condition, each participantmade 10�2�12 � 240 choices, for a total of 480choices across conditions. The order of gamblepresentation was randomly determined withineach timing condition. Both timing conditionswere completed in a single experimental sessionand the order in which a participant completedthe two conditions was randomly determined. Inaddition to being paid $10 for participating,each participant was given additional compen-sation by randomly selecting one of their chosengambles to be played out for real money.
The two gamble sets that were used in theexperiment are as follows. The five gambles in Set1 were: �25.43, 7
24�, �24.16, 824�, �22.89, 9
24�, �21.62, 1024�,
and �20.35, 1124�, where (X, p) denotes a binary gam-
ble with probabilities p of winning X dollars and1 � p of winning 0 dollars. These gambles weregenerated by updating the dollar values of thegamble stimuli from Tversky’s (1969) experi-ment, to adjust for inflation (2009 dollars using theConsumer Price Index). The gambles in Set 2were generated with identical probabilities tothose in Set 1, but with larger variances in thepayoffs. Specifically, the five gambles in Set 2were: �31.99, 7
24�, �27.03, 824�, �22.89, 9
24�, �19.32, 1024�,
�16.19, 1124�. In Set 2, a participant choosing accord-
ing to expected value would always choose thegamble with the greater payoff value, whereas inSet 1, the same participant would always choosethe gamble with a greater probability of winning(Tversky, 1969). Within each timing condition,we randomly intermixed gamble pairs from thetwo gamble sets.
Time Pressure Manipulation
Prior research has demonstrated that theamount of time that a DM has to make a deci-sion can affect how he or she searches forinformation (Ben Zur & Breznitz, 1981; Böck-enholt & Kroeger, 1993; Payne et al., 1988).More recent work has suggested that directlylimiting the amount of time available to a DMcan alter the strategies that he or she uses whenmaking decisions. For example, Rieskamp andHoffrage (1999, 2008) found that DMs are morelikely to use lexicographic strategies when mak-ing choices under various types of time pressuremanipulation. Given that the stimuli used in Tver-sky’s (1969) experiment were designed to induceintransitive preferences arising from a lexico-graphic semiorder structure, we hypothesized thatan additional time pressure manipulation may leadto more frequent violations of transitivity. To in-vestigate this hypothesis, we tested participantsunder two conditions: “timed” and “nontimed.” Inthe nontimed condition, similar to previous stud-ies, participants were allowed as much time asneeded to make their choices. Under the timedcondition, participants were given only 4 s torespond. If a participant did not respond within theallotted time, the computer flashed a message in-dicating that they had run out of time and theexperiment advanced to the next trial. To maintainthe integrity of the incentive structure (i.e., ran-domly selecting one of the participant’s preferredgambles to be played out for money at the end ofthe experiment), a “preferred” gamble was se-lected at random in such trials. However, for thepurposes of data analysis, these trials were simplyomitted. The average response time for the timedcondition, across participants, was 1.572 s (me-dian � 1.430). The average response time for thenontimed conditions, across participants, was2.431 s (median � 1.801). The average rate inwhich participants failed to respond within theallotted time, across participants, was 0.0069, thatis, just over one half of 1%.
Model Selection Analysis
Before starting the main analysis, we tested theindependence assumption on the new data usingthe method proposed by Smith and Batchelder(2008). We refer the reader to the original paperfor details of the test procedure (see also Cha et al.,2013). The number of significant violations of
113TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
independence according to this test was consistentwith what would be expected from type-1 error, sowe conclude that the independence assumption isreasonable. Detailed results of the test are given inthe online supplement.
Having justified the independence assumptionfor these data, we analyzed them using the Bayesfactor method described earlier. The results,shown in Table 4, indicate that our three mainfindings regarding past studies persist in the newexperiment: (1) no model can adequately explainevery participant’s choice data, (2) WST fails theleast often but is seldom the preferred model, and(3) SST is the preferred model the most often.
Somewhat surprisingly, the timing manipula-tion in the experiment did not seem to change howfrequently each model provided the best descrip-tion. It was hypothesized that the timed conditionwould cause participants to make decisions ac-cording to a fast heuristic like a lexicographicsemiorder, which would result in more intransitivepreference patterns, but that hypothesis is not sup-ported by the data. In fact, slightly fewer choice
patterns are best described by M1 in the timedcondition as compared to the nontimed condition.This result further highlights the “robustness” oftransitivity of preference.
Consistency Across Timing Conditions andGamble Sets
Because the experimental conditions weremanipulated within-participant, we are able toinvestigate whether participants’ choices tend tobe best described by the same model acrosstiming conditions and gamble sets. To that end,we constructed contingency tables relatingmodel classification across experimental condi-tions. First, to investigate consistency acrosstiming conditions, the top panel of Table 5relates model classification in the timed condi-tion to model classification in the nontimedcondition. In order to make the table less sparse,the counts are aggregated across gamble sets. Inaddition, MMTP, WST, and MST are collapsedinto a single category, as they were the threeleast-frequently occurring classifications, yield-ing a 3 � 3 table. A �2 test of independenceindicated a significant and relatively strong as-sociation between model classifications acrosstiming conditions, �2(4, N � 58) � 25.4111,p .001, Cramer’s V � 0.468, which wasconfirmed by Fisher’s exact test (p .001).
Next, to investigate consistency across gam-ble sets, the bottom panel of Table 5 relatesmodel classifications across the two gamble sets. Asbefore, MMTP, WST, and MST are collapsed into asingle category to make the table less sparse, andthis time the counts are aggregated across tim-ing conditions. A �2 test of independence indi-cated a moderate association between modelclassifications in the two gamble sets, �2(4, N �58) � 16.3464, p � .0026, Cramer’s V � 0.375,which was also confirmed by Fisher’s exact test(p � .0024).
Taken together, these results speak to therobustness of the model classifications. Al-though there were not enough observations toassess the stability of MMTP, WST, and MSTseparately across conditions, these analysesclearly show that some participants consistentlymake choices that are best described by SST,other participants consistently make choicesthat are best described by some weaker modelof transitivity (either MMTP, WST, or MST),while still other participants seem to consis-
Table 4Number of Participants in the New Experiment forWhom Each Model of Transitivity Failed (TopPanel) and Was Best (Bottom Panel) According tothe Bayes Factor
Note. MMTP � mixture model of transitive preference;WST � weak stochastic transitivity; MST � moderatestochastic transitivity; SST � strong stochastic transitivity.The number in each condition indicates the gamble set (e.g.,Timed 1 is the timed condition with Set 1). The “None”column counts the number of times that the model selectionstatistic was inconclusive (all of the Bayes factors between10�1/2 and 101/2). Rows sum to N, the number of partici-pants.
114 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
tently make choices that violate all of the mod-els of transitivity that we have considered. Thissuggests that these models capture stable prop-erties of choice behavior.
Hierarchical Analysis
The analyses we have reported so far pertainedto each participant in each experiment separately.They were intended to identify the model that isthe best explanation of the data. While these anal-yses successfully identified a best model on aper-participant basis, the results were not unani-mous; different participants were explained bestwith different models. Nevertheless, all but ahandful of participants were best described by oneof the models of transitivity rather than the en-compassing model. Therefore, it seems that peo-ple are generally transitive in their preferences, buttransitive in different ways.
To bring these separate analyses together andsee what they can tell us about transitivity as awhole, we also conducted a meta-analytic as-
sessment of the collection of experiments, usinga hierarchical Bayesian mixture model. Unlikethe analyses that have been reported so far,which were based on counts of the best andworst performing models, this approach willfully utilize the continuous measurements ofmodel performance provided by the Bayes fac-tor. The key idea in this approach is that, ratherthan assuming exactly one model is correct andthe others incorrect, we assume all of the mod-els are useful but that some may be more likelyto explain the behavior of more participants thatothers. This type of assessment has been used touncover distributions of strategies within cog-nitive toolbox models (Scheibehenne,Rieskamp, & Wagenmakers, 2013), and in theanalysis of recognition memory models (Denniset al., 2008). The approach can also be viewedas an application of Latent Dirichlet Allocation(LDA), which is commonly used in machinelearning and natural language processing to dis-cover the distribution of abstract “topics” thatoccur in a collection of documents (Blei et al.,2003). In this case, instead of discovering adistribution of topics in a document, we wish todiscover the distribution of models of transitiv-ity in a population.
The LDA approach requires that each per-son makes choices according to a fixedmodel, which is drawn from a latent categor-ical distribution over the five models we haveconsidered: M1, MMTP, WST, MST, SST.Each person’s choices are then assumed to begenerated from their model, via the prior andlikelihood functions defined in the previoussection. This links the participants’ choices tothe distribution of models, so that the lattercan be estimated based on the former. Esti-mation of the distribution of models is accom-plished by assuming a Dirichlet hyper-priorover possible categorical distributions, whichis then updated based on the observed choicesof each participant. For clarity, we will writeM1, . . . , M5 for the five models under consid-eration. We can then write � (1, . . . ,5) forthe parameters of the categorical distributionover the five models and � � (�1, . . . , �5) forthe concentration parameters of a Dirichlet dis-tribution over . We will define the prior dis-tribution on to be Dirichlet with �1 � �2 � .. . � �5 � 1 (i.e., a uniform distribution overpossible categorical distributions). This prior
Table 5Contingency Table of Model Classifications in theTimed and Nontimed Conditions, AggregatedAcross Gamble Sets (Top Panel) and in the TwoGamble Sets, Aggregated Across Timing Conditions(Bottom Panel)
Timing conditions
Timed
Nontimed
M1 SST Other
M1 10 1 1SST 3 17 6Other 5 6 9
�2(4, n � 58) � 25.4111, p .001Cramer’s V � 0.468
Gamble sets
Set 2
Set 1 M1 SST Other
M1 9 2 6SST 4 14 9Other 0 7 7
�2(4, N � 58) � 16.3464, p � .0026Cramer’s V � 0.375
Note. SST � strong stochastic transitivity. The “other”category includes weak stochastic transitivity (WST), mod-erate stochastic transitivity (MST), and mixture model oftransitive preference (MMTP).
115TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
will be updated sequentially upon observationof each participant’s vector of choices as
p(� |nj) � i�1
5p(� |Mj � Mi)p(Mj � Mi |nj),
where nj denotes the vector of choices and Mjdenotes the “true” model for participant j, andthe sum is taken over the five models underconsideration. This expression allows for aneasy approximation of the posterior distributionfrom the statistics we have already computed. Inparticular, p(Mj � Mi | nj) is the posterior prob-ability of model Mi given observed choices nj.This probability is derived easily for each Mifrom the Bayes factors that were obtained pre-viously. Furthermore, since the Dirichlet prioris conjugate to categorical data, p( | Mj � Mi,�) is also Dirichlet with the ith concentrationparameter increased by 1. It follows that | nj isa convex sum of Dirichlet distributions,weighted by, p(Mj � Mi | nj). To avoid combi-natorial explosion upon subsequent updating,this convex sum is approximated with a singleDirichlet distribution with the same mean andvariances. Following this sequential updatingprocedure, each concentration parameter �i inthe posterior distribution is computed by addingthe posterior probability of the correspondingmodel Mi across participants. That is, �i | D �
1 � j p�Mj � Minj�, where D denotes thecollection of choice vectors of all participants.
We first fit the LDA model to the data fromeach study separately, counting the three ex-periments from Regenwetter, Dana, and Da-vis-Stober (2011) as one study (which we willrefer to as “RDS”) and counting the fourconditions from our own experiment as onestudy (which we will refer to as “Current”).These results are given in the first 6 rows ofTable 6. The first three rows of the table givethe estimated (modal) distribution of the mod-els based on the studies of Tversky, Ranyard,and Montgomery, respectively. Even thoughthese studies prescreened participants and se-lectively reported data that seemed likely toviolate transitivity, the results of our hierar-chical analysis show that MMTP best de-scribes an estimated 34% of the populationfor which Tversky’s (1969) participants arerepresentative. Of course, it is also importantto note that the between-subjects sample size
is extremely small in these three studies, sothis result may be highly sensitive to theassumed prior over categorical distributions.
The next three rows of Table 6 give theestimated distributions of the models basedon the studies of Tsai and Bockenholt, RDS,and Current, respectively. The differencesamong these estimates may be attributed todifferences between the experimental designsin the respective studies (e.g., Tsai & Bock-enholt had 4 gamble stimuli with 120 repeti-tions per paired comparison, while RDS andCurrent had 5 gamble stimuli and 10 –20 rep-etitions per paired comparison). Since each ofthese studies has a relatively small between-subjects sample size, it may be more useful toexamine the fit of the LDA model to thecombined data from all three. We did justthat, and the resulting distribution (mode ofthe posterior) is shown in the last row ofTable 6, labeled “Overall.” In this distribu-tion, none of the probabilities are lower than0.14 (WST), and none are higher than 0.31(SST), indicating that each of the models wehave considered is likely to be useful fordescribing the choice behavior of a nontrivialproportion of participants.
Table 6Distribution of Models of (In)Transitivity asEstimated by the Posterior Mode of the LatentDirichlet Allocation (LDA) Model for Each Study(or Group of Studies)
Note. MMTP � mixture model of transitive preference;WST � weak stochastic transitivity; MST � moderatestochastic transitivity; SST � strong stochastic transitivity.“RDS” refers to the combined results from the three exper-iments in the Regenwetter, Dana, and Davis-Stober (2011).“Current” refers to the combined results of the four condi-tions from the new experiment reported in this paper. Thedata from the Tversky, Montgomery, and Ranyard studiesare omitted from the “Overall” analysis because those stud-ies either prescreened participants or selectively reporteddata.
116 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Discussion
Summary
We analyzed the adequacy of four majormodels of transitivity across five previousexperimental, within-subjects studies, as wella new within-subjects study. Our Bayesianstatistical methodology allowed us to evaluateand compare the relative merits of differentmodels of transitivity. Thus, we were able todetermine precisely which models best ac-count for each participant’s observed choicebehavior, properly balancing both goodness-of-fit and model complexity. We concludethat individuals are generally transitive intheir preferences, in agreement with recentstudies on the topic (e.g., Birnbaum, 2011;Regenwetter, Dana, & Davis-Stober, 2011).However, in contrast to these studies, we ar-rive at a more nuanced position. We find thatwhile most individuals are transitive, theyexhibit choice variability in different ways.Across the six studies, using our hierarchicalmethodology, we found substantial heteroge-neity in which model of transitivity best ac-counted for individuals’ choices. In otherwords, some individuals are best consideredas “changing their minds” among multipledecision states (i.e., MMTP), while other in-dividuals are better described by having asingle, transitive decision state, occasionallymaking random errors when choosing amongdifferent choice alternatives (i.e., WST, MST,or SST). Among the latter group, about halfare best described by SST, meaning that theirchoice probabilities seem to satisfy sub-stitutability and independence (Luce, 1977),while the other half are better described bymodels that do not assume these properties(i.e., WST or MST). Our new study demon-strated that transitive individuals were stilllikely to express transitive choice patternseven under time pressure conditions. This re-sult further strengthens the interpretation oftransitivity as an invariant of decision makingbehavior. This is not to say that such individ-uals would always be transitive, only thattransitive preference, and the particular typeof individual choice variability associatedwith it, appears to be a robust property at theindividual level.
Model Specification
The models of WST, MST, and SST that wedefined here differ in complexity from thosedefined by Myung et al. (2005). One key dif-ference between these studies is that the modelsconsidered by Myung et al. were specified con-ditionally on particular transitive orderings ofthe five alternatives in Tversky’s (1969) exper-iment, resulting in models that were containedwithin a single half-unit cube within the 10-dimensional hypercube. In contrast, the modelswe considered here were not conditional on aparticular ordering, resulting in models that canbe characterized as unions of subsets of half-unit cubes, where the unions are taken over thepossible transitive orderings. As a result, ourmodels of WST, MST, and SST are consider-ably more complex than those considered byMyung et al. (2005). Their reasoning in select-ing a single ranking was that the models shouldbe tailored to reflect the way Tversky antici-pated the data would come out, given that Tver-sky’s experiment was engineered, through pre-screening of participants and strategic design ofchoice alternatives, to elicit a particular patternof choice responses. While the resulting modelsmay provide a simpler explanation of Tversky’sdata, the apparent simplicity is built out of ad-ditional assumptions beyond just transitivity. Inthe present work, our intention is to assess onlythe axiom of transitivity, and therefore our mod-els are built to minimally capture each probabi-listic specification. This makes the models ap-plicable to a wider range of data sets, includingexperiments that are not strategically engi-neered to elicit particular preference patterns.Despite these differences in the model specifi-cation, and the fact that we used the Bayesfactor rather than the DIC, our results agreewith those of Myung et al. (2005) in that theencompassing model is favored over WST,MST, and SST for six of the eight participantsin Tversky’s experiment.
The preceding argument applied only toWST, MST, and SST, but similar modificationscould be made to MMTP. Of course, a mixturemodel cannot be constructed from just one rank-ing, but if only a small subset of rankings areconsidered plausible then a polytope could beformed from the convex hull of those rankingsin the hypercube. Such a model would be sim-pler than the full version of MMTP, and hence
117TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
may be preferred according to Bayesian modelselection metrics. One challenge in this ap-proach, however, would be identifying the in-equality constraints that define the resultingpolytopes, which is not a trivial problem. An-other challenge would be to justify which rank-ings should be included in the plausible set.
Violations of Transitivity
While a majority of the participants we ana-lyzed were best described by a model of tran-sitivity, a distinct minority were best describedby the encompassing model. By our meta-analysis, we estimated that around 16% of par-ticipants are best described by this model. How-ever, the encompassing model does notinstantiate any theory of stochastic choice; itmerely serves as null model against which toassess the adequacy of models of transitivity.Our results then raise the question: Which theo-ry(ies) of stochastic choice would best describethis minority of participants for whom modelsof transitivity are inadequate? Such a theorywould need to allow intransitive preferencestates, yet still account for choice variability.
One possibility would be a theory based onlexicographic semiorders, which are consistentwith both transitive and intransitive preferencestates. Davis-Stober (2012) recently developeda mixture model of stochastic choice basedupon lexicographic semiorders and future workcould investigate the descriptive adequacy ofthis type of “intransitive” model. However, asthe model allows indifference in addition tostrict preferences, this would require new ex-periments carried out under a ternary choiceframework as opposed to two-alternative forcedchoice. Such experiments have been carried outby Davis-Stober et al. (2013), who find thatabout 20% of participants are better describedby a lexicographic semiorder mixture modelthan a mixture model of weak orders. It isimportant to note that our focus in the currentstudy was to assess the viability of transitivityas a basic modeling assumption. Our goal wasnot to assess the descriptive accuracy of anyparticular decision theory, transitive or intransi-tive. In this way, our approach is quite general.For example, choices from the 16% of partici-pants best described by the null model wouldnot be well fit by any transitive decision theory.We leave it future work to explore the underly-
ing algebraic preference structure of these par-ticipants.
Parsimonious Alternative Models
Although the encompassing model was re-jected for a majority of participants, meaningthat their choices were better described by amodel that assumes transitivity than one thatdoes not, our analyses do not rule out othernontransitive models. The encompassingmodel, being fully unconstrained, is the leastparsimonious model in our analysis. BecauseBayesian model selection rewards parsimony,this raises the question of whether a more con-strained, nontransitive model might have faredbetter in our analysis and actually been the mostpreferred model in some cases. It may indeed bepossible to construct such a sufficiently parsi-monious, nontransitive model ad hoc, by intel-ligently carving up the model space in just theright way, or by using an informative prior onthe unconstrained space that puts relatively lessweight on “intransitive” regions. However,given that the choice proportions of most par-ticipants perfectly satisfied the constraints of atleast one model of transitivity (Table 1), anysuch nontransitive model could be trumped by afurther constrained transitive model constructedby dropping the nontransitive choice probabili-ties. A more promising candidate would be aparsimonious model that is consistent with bothtransitive and intransitive preference states, likethe aforementioned lexicographic semiordermodel. This model lies outside the strict-linear-ordered preference framework of the other mod-els considered in this study, so we leave itsanalysis for future work (see, e.g., Davis-Stoberet al., 2013).
On a related note, although we found strongsupport for SST in many cases (Tables 3, 4),SST had relatively few perfect fits in the data(Table 1) so it follows that its support waslargely based on it being the most parsimoniousof the models we considered. Therefore, it ispossible that an even more parsimonious modelthat violates SST, such as Decision Field The-ory (Busemeyer & Townsend, 1993), or a con-trast-weighting model (Mellers & Biagini,1994), or a stochastic difference model(González-Vallejo, 2002), could be preferredover SST in Bayesian model selection. Our ex-periment did not include manipulations specif-
118 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
ically designed to produce violations of SST, sowe should be cautious with our conclusionsabout its robustness. Berkowitsch et al. (2013)recently showed that models satisfying SST arefavored with consumer preference choice de-signs that do not manipulate context effects, butmodels violating SST are preferred in designsthat include choice options producing contexteffects. In fact there is quite strong empiricalevidence for these violations across the litera-ture (Rieskamp et al., 2006). Future workshould further examine the robustness of SSTand competing models in a Bayesian modelselection framework.
All of the models we considered in this studyare defined at the level of marginal choice prob-abilities. Even MMTP, although motivated bythe idea of a dynamic process in which a linearorder is drawn at random from some distribu-tion each time a choice is made, is definedformally by the marginal choice probabilitiesthat are consistent with such a process. Becausethe models are defined at the level of marginalchoice probabilities, we tested them by estimat-ing marginal choice probabilities via marginalchoice proportions. Because this approach as-sumes that choices are independent (or at leastexchangeable), it cannot capture any additionalstructure (such as correlation or nonstationarity)that may be present in the data generating pro-cess.
Although this approach to modeling stochas-tic choice is standard in the field, its use comeswith the caveat that if the data-generating pro-cess does have additional structure (e.g., corre-lation or nonstationarity) then the marginalchoice proportions do not map uniquely back tothe choice probabilities that generated them.8For example, Birnbaum (2011) showed (withhypothetical data) that choice proportions con-sistent with a mixture model could be generatedfrom a nonstationary process in which the actualchoice probabilities are never consistent withthe mixture model. However, such extreme con-clusions have not been drawn from any set ofhuman data, to our knowledge, and a violationof iid sampling does not necessarily mean thatthe estimated choice proportions are not repre-sentative of the actual process that generated the
data. Future work should be done to develop atest for whether a deviation from iid is severeenough to invalidate the substantive conclu-sions of a model based on based on binarychoice probabilities.
Even with countermeasures such as decoystimuli and randomized presentation in place tolimit memory and order effects, the assumptionof iid is so strong that there will almost certainlybe deviations from it. The important questionthen is whether these deviations are largeenough to warrant modeling them explicitly.There are other promising approaches to mod-eling stochastic choice that are capable of cap-turing such additional structure in the data, suchas the “true and error” approach of Birnbaumand Gutierrez (2007). The unit of analysis inthis approach is the pattern of responses withineach ‘block’ of choices. A ‘block’ is defined asa particular grouping of trials, typically a singlepresentation of all possible gamble pairs. In thetrue and error model, decision makers are as-sumed to have a “true” preference state thatremains fixed within each block of choices, butmay change between blocks, and to make ran-dom “errors” that generate choice variability.Choices within each block are assumed to beconditionally independent (conditional on thetrue preference state in the given block), ratherthan iid, which allows for dependencies be-tween binary choice probabilities when margin-alizing over response patterns. True-and-errormodels have been shown to provide an excellentfit to empirical data (Birnbaum, 2011; Schmidt& Stolpe, 2011), but the approach has beencriticized for merely shifting the locus of the iidassumption from the choices to the errors, andfor being highly sensitive to the choice of whichsets of trials constitute blocks (Regenwetter etal., 2010; Regenwetter, Dana, Davis-Stober, &Guo, 2011). Future effort should be devoted tocomparing these approaches, in order to ascer-tain how much complexity can and should beextracted from binary choice data.
While the approaches based on response pat-terns are promising and worthy of further inves-tigation, in this study we focused on binary
8 It is also worth noting that, in MMTP, even if the iidassumption holds perfectly, the exact mixture of preferencestates that marginalize to the estimated binary choice prob-abilities cannot be identified uniquely.
119TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
choice probabilities for several reasons. First,the most general and well-studied stochasticdefinitions of transitivity are defined in terms ofbinary choice probabilities, for example, WST,MST, SST, and so forth. Second, binary choiceprobabilities are easily extended to differentstochastic operationalizations of choice vari-ability such as mixture modeling, error models,and so forth, (Davis-Stober & Brown, 2011).Third, this was the most reasonable way to lookat the data from past experiments that do notexplicitly include the blocking structure re-quired to facilitate an analysis of response pat-terns. Fourth, marginal choice probabilities arereadily applicable to many decision making do-mains and constitute a general approach to mod-eling choice, while other approaches that focusupon choice patterns often rely upon specificexperimental designs, such as blocking (e.g.,Birnbaum, 2011). Finally, and most impor-tantly, despite the theoretical differences be-tween modeling choice patterns and choice pro-portions, recent analyses using either techniquehave agreed in their basic conclusions abouttransitivity: significant violations of transitivityare rare.
Conclusion
Heretofore, the majority of studies investigat-ing transitivity of preference have largely fo-cused on the binary question of whether or notthe axiom holds. We find that transitivity holdsfor many, but not all individuals. Thus, a “one-size-fits all” approach to modeling choice vari-ability will be unlikely to describe humanchoice behavior sufficiently. In this way, ourfindings are in strong agreement with previousstudies examining choice variability by Loomeset al. (2002) and Hey (2001, 2005). Futuretheory development should be sensitive to indi-vidual differences with regard to both stochasticvariability as well as the algebraic structure ofpreference.
References
Anand, P. (1993). The philosophy of intransitivepreference. The Economic Journal, 103, 337–346.
Ben Zur, H., & Breznitz, S. J. (1981). The effect oftime pressure on risky choice behavior. Acta Psy-chologica, 47, 89–104.
Berkowitsch, N. A., Scheibehenne, B., & Rieskamp,J. (2013). Rigorously testing multialternative deci-
sion field theory against random utility models.Journal of Experimental Psychology: General.Advance online publication.
Bernardo, J. (1996). The concept of exchangeabilityand its applications. Far East Journal of Mathe-matical Sciences, 4, 111–122.
Birnbaum, M. H. (2011). Testing mixture models oftransitive preference: Comments on Regenwetter,Dana, and Davis-Stober (2011). Psychological Re-view, 118, 675–683.
Birnbaum, M. (2012). A statistical test of indepen-dence in choice data with small samples. Judgmentand Decision Making, 7, 97–109.
Birnbaum, M. H., & Gutierrez, R. J. (2007). Testingfor intransitivity of preferences predicted by a lex-icographic semi-order. Organizational Behaviorand Human Decision Processes, 104, 96–112.
Birnbaum, M. H., & Schmidt, U. (2008). An exper-imental investigation of violations of transitivity inchoice under uncertainty. Journal of Risk and Un-certainty, 37, 77–91.
Blei, D., Ng, A., & Jordan, M. (2003). Latent dirich-let allocation. The Journal of Machine LearningResearch, 3, 993–1022.
Böckenholt, U., & Kroeger, K. (1993). The effect oftime pressure in multiattribute binary choice tasks.In O. Svenson & A. J. Maule (Eds.), Time pressureand stress in human judgment and decision making(pp. 195–214). New York, NY: Plenum Press.
Busemeyer, J. (1985). Decision making under uncer-tainty: Simple scalability, fixed sample, and se-quential sampling models. Journal of Experimen-tal Psychology: Learning, Memory, andCognition, 11, 538–564.
Busemeyer, J., & Townsend, J. (1993). Decision fieldtheory: A dynamic-cognitive approach to decisionmaking in an uncertain environment. Psychologi-cal Review, 100, 432–459.
Cha, Y. S., Choi, M., Guo, Y., Regenwetter, M., &Zwilling, C. (2013). Reply: Birnbaum’s (2012).statistical tests of independence have unknowntype-I error rates and do not replicate within par-ticipant. Judgment and Decision Making, 8, 55–73.
Davidson, D., & Marschak, J. (1959). Experimentaltests of a stochastic decision theory. Measurement:Definitions and theories, pp. 233–69.
Davis-Stober, C. (2009). Analysis of multinomialmodels under inequality constraints: Applicationsto measurement theory. Journal of MathematicalPsychology, 53, 1–13.
Davis-Stober, C. P. (2012). A lexicographic semi-order polytope and probabilistic representations ofchoice. Journal of Mathematical Psychology, 56,86–94.
Davis-Stober, C., & Brown, N. (2011). A shift instrategy or “error”? strategy classification overmultiple stochastic specifications. Judgment andDecision Making, 6, 800–813.
120 CAVAGNARO AND DAVIS-STOBER
This
docu
men
tis
copy
right
edby
the
Am
eric
anPs
ycho
logi
calA
ssoc
iatio
nor
one
ofits
allie
dpu
blis
hers
.Th
isar
ticle
isin
tend
edso
lely
for
the
pers
onal
use
ofth
ein
divi
dual
user
and
isno
tto
bedi
ssem
inat
edbr
oadl
y.
Davis-Stober, C. P., Brown, N., & Cavagnaro, D. R.(2013). Evaluating decision makers’ preferencesvia lexicographic semiorders. Retrieved from http://www.icmconference.org.uk/index.php/icmc/ICMC2013/paper/viewFile/674/199
Dennis, S., Lee, M. D., & Kinnell, A. (2008). Bayes-ian analysis of recognition memory: The case ofthe list-length effect. Journal of Memory and Lan-guage, 59, 361–376.
Fiorini, S. (2001). Determining the automorphismgroup of the linear ordering polytope. DiscreteApplied Mathematics, 112, 121–128.
Gelman, A., Carlin, J., Stern, H., & Rubin, D. (2004).Bayesian data analysis. Boca Raton, FL: CRCPress.
González-Vallejo, C. (2002). Making trade-offs: Aprobabilistic and context-sensitive model of choicebehavior. Psychological Review, 109, 137–155.
Grether, D., & Plott, C. (1979). Economic theory ofchoice and the preference reversal phenomenon.The American Economic Review, 69, 623–638.
Halff, H. (1976). Choice theories for differentiallycomparable alternatives. Journal of MathematicalPsychology, 14, 244–246.
Hey, J. (2001). Does repetition improve consistency?Experimental Economics, 4, 5–54.
Hey, J. (2005). Why we should not be silent aboutnoise. Experimental Economics, 8, 325–345.
Iverson, G., & Falmagne, J. (1985). Statistical issuesin measurement. Mathematical Social Sciences,10, 131–153.
Jeffreys, H. (1961). Theory of probability, 3rd edi-tion. Oxford, UK: Oxford University Press.
Karabatsos, G. (2006). Bayesian nonparametricmodel selection and model testing. Journal ofMathematical Psychology, 50, 123–148.
Kass, R., & Raftery, A. (1995). Bayes factors. Jour-nal of the American Statistical Association, 90,773–795.
Klugkist, I., & Hoijtink, H. (2007). The Bayes factorfor inequality and about equality constrained mod-els. Computational Statistics & Data Analysis, 51,6367–6379.
Lee, M., & Wagenmakers, E. (2014). Bayesian cog-nitive modeling: A practical course. Cambridge,UK: Cambridge University Press.
Lindley, D., & Phillips, L. (1976). Inference for abernoulli process (a Bayesian view). The AmericanStatistician, 30, 112–119.
Loomes, G., Moffatt, P., & Sugden, R. (2002). Amicroeconometric test of alternative stochastic the-ories of risky choice. Journal of Risk and Uncer-tainty, 24, 103–130.
Loomes, G., & Sugden, R. (1995). Incorporating astochastic element into decision theories. Euro-pean Economic Review, 39, 641–648.
Luce, R. D. (1977). The choice axiom after twentyyears. Journal of Mathematical Psychology, 15,215–233.
Luce, R. (2000). Utility of gains and losses: Mea-surement-theoretical, and experimental ap-proaches (Vol. 13). Mahwah, NJ: Lawrence Erl-baum Associates, Inc.
Montgomery, H. (1977). A study of intransitive pref-erences using a think aloud procedure. In H.Jungerman & G. de Zeeuw (Eds.), Decision mak-ing and change in human affairs (pp. 347–362).Dordrecht, the Netherlands: Reidel.
Myung, I. (2000). The importance of complexity inmodel selection. Journal of Mathematical Psy-chology, 44, 190–204.
Myung, I. J., & Pitt, M. A. (2002). Mathematicalmodeling. In H. W. Pashler & J. Wixted (Eds.),Stevens’ handbook of experimental psychology:Vol. 4. Methodology in experimental psychology(3rd ed., pp. 429–460). New York: Wiley.
Myung, J., Karabatsos, G., & Iverson, G. (2005). ABayesian approach to testing decision making ax-ioms. Journal of Mathematical Psychology, 49,205–225.
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988).Adaptive strategy selection in decision making.Journal of Experimental Psychology: Learning,Memory, and Cognition, 14, 534–552.
Ranyard, R. (1977). Risky decisions which violatetransitivity and double cancellation. Acta Psycho-logica, 41, 449–459.
Regenwetter, M., Dana, J., & Davis-Stober, C.(2010). Testing transitivity of preferences on two-alternative forced choice data. Frontiers in Psy-chology, 1, 148.
Regenwetter, M., Dana, J., & Davis-Stober, C.(2011). Transitivity of preferences. PsychologicalReview, 118, 42–56.
Regenwetter, M., Dana, J., Davis-Stober, C. P., &Guo, Y. (2011). Parsimonious testing of transitiveor intransitive preferences: Reply to Birnbaum(2011). Psychological Review, 118, 684–688.
Rieskamp, J., Busemeyer, J., & Mellers, B. (2006).Extending the bounds of rationality: Evidence andtheories of preferential choice. Journal of Eco-nomic Literature, 44, 631–661.
Rieskamp, J., & Hoffrage, U. (1999). When do peo-ple use simple heuristics, and how can we tell? InG. Gigerenzer & P. Todd (Eds.), Simple heuristicsthat make us smart. Transitivity of preferences.Oxford, United Kingdom: Oxford UniversityPress.
Rieskamp, J., & Hoffrage, U. (2008). Inferences un-der time pressure: How opportunity costs affect
121TRANSITIVE IN OUR PREFERENCES BUT DIFFERENT WAYS
Roe, R., Busemeyer, J., & Townsend, J. (2001).Multialternative decision field theory: A dynamicconnectionist model of decision making. Psycho-logical Review, 108, 370–392.
Schmidt, U., & Stolpe, M. (2011). Transitivity inhealth utility measurement: An experimental anal-ysis. Health Economics Review, 1, 1–4.
Smith, J. B., & Batchelder, W. H. (2008). Assessingindividual differences in categorical data. Psycho-nomic Bulletin & Review, 15, 713–731.
Tsai, R., & Böckenholt, U. (2006). Modelling intran-sitive preferences: A random-effects approach.Journal of Mathematical Psychology, 50, 1–14.
Tversky, A. (1969). Intransitivity of preferences.Psychological Review, 76, 31–48.
Tversky, A. (1972). Elimination by aspects: A theoryof choice. Psychological Review, 79, 281.
Tversky, A., & Russo, J. R. (1969). Substitutabilityand similarity in binary choices. Journal of Math-ematical Psychology, 6, 1–12.
Vanpaemel, W. (2010). Prior sensitivity in theorytesting: An apologia for the Bayes factor. Journalof Mathematical Psychology, 54, 491–498.
Wagenmakers, E., Lodewyckx, T., Kuriyal, H., &Grasman, R. (2010). Bayesian hypothesis testingfor psychologists: A tutorial on the savage-dickeymethod. Cognitive Psychology, 60, 158–189.
Wilcox, N. (2008). Stochastic models for binary dis-crete choice under risk: A critical primer andeconometric comparison. Research in Experimen-tal Economics, 12, 197–292.
Received November 23, 2013Revision received February 17, 2014
Accepted February 19, 2014 �
E-Mail Notification of Your Latest Issue Online!
Would you like to know when the next issue of your favorite APA journal will beavailable online? This service is now available to you. Sign up at http://notify.apa.org/ andyou will be notified by e-mail when issues of interest to you become available!
